PERCEPTIONS OF DECISION SUPPORT SYSTEM SUCCESS: LESSONS FROM THE NORTH CENTRAL REGION By Laura Young A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Community Sustainability–Master of Science 2020 ABSTRACT PERCEPTIONS OF DECISION SUPPORT SYSTEM SUCCESS: LESSONS FROM THE NORTH CENTRAL REGION By Laura Young Decision support systems (DSSs) have been utilized extensively to enhance problem- framing, structure decisions around complex natural resource issues, increase learning, and enrich group collaboration. However, these tools are rarely evaluated for their actual impact on decision-making. Challenges commonly encountered by developers include low rates of adoption, limited end-user participation in DSS development, and issues with securing long-term maintenance. This research explores the extent to which four DSSs have been successfully deployed to facilitate decision-making and support agricultural conservation within the North Central Region of the U.S. The Agricultural Conservation Planning Framework, the Daily Erosion Project, the Great Lakes Watershed Management System, and the Runoff Risk Advisory Forecast have been utilized in multiple states to improve land management and conservation. Conducting interviews with both developers and end-users of these systems, I demonstrated that these DSSs are used to improve conservation planning, enable performance- based conservation, prioritize outreach activities, and enhance farmer engagement. However, adoption has been mixed. Some DSSs are embedded within routine planning practices and others indicate few users. Lack of consistent funding and the potential to lose key personnel that either possess the technical knowledge or social influence to maintain and promote the system threaten DSS sustainability. Despite challenges, the process of developing and deploying DSSs provided value to stakeholders, fostering collaboration among diverse sets of organizations. Copyright by LAURA YOUNG 2020 ACKNOWLEDGEMENTS There are many people that supported me along the way in completing this work. To my family and friends near and far, thank you for your encouragement over the last three years. I would like to extend my gratitude to my advisor, Dr. Steven Gray for supporting and guiding me through my graduate degree. You’ve helped me grow as a researcher, and I will carry forward what I’ve learned about mentorship from you to support others in my career. To my committee members Dr. Doug Bessette and Ruth Kline-Robach, thank you for your valuable insights that enriched my inquiry, and stepping up to help when needed. To the CSUS community, and especially Marsha Edington and Dr. Gail Vanderstoep, thank you for your assistance throughout this process. I would also like to thank the Institute of Water Research at Michigan State University for supporting this work and inspiring my research. To everyone that was interviewed for this study, thank you for sharing your stories and experiences that have illuminated so much about the nature of decision support systems. This material is based upon work supported by the U.S. Geological Survey under Grant No. G16AP00063. iv TABLE OF CONTENTS LIST OF TABLES ......................................................................................................................... vi LIST OF FIGURES ...................................................................................................................... vii KEY TO ABBREVIATIONS ...................................................................................................... viii 1. INTRODUCTION ...................................................................................................................... 1 1.1. Study Objectives .................................................................................................................. 4 1.2. Study Area ............................................................................................................................ 5 2. METHODS ................................................................................................................................. 7 2.1. Case Study Selection ............................................................................................................ 7 2.2. Data Collection ................................................................................................................... 12 2.3. Data analysis ...................................................................................................................... 14 3. RESULTS ................................................................................................................................. 16 3.1. Purpose, Audience and Uses .............................................................................................. 16 3.2. Outreach and Education ..................................................................................................... 23 3.3. Adoption ............................................................................................................................. 27 3.4. User Satisfaction ................................................................................................................ 32 3.5. Broader Outcomes .............................................................................................................. 35 3.6. Sustainability ...................................................................................................................... 40 4. DISCUSSION ........................................................................................................................... 45 4.1 Outcomes ............................................................................................................................. 45 4.2 Challenges ........................................................................................................................... 46 5. CONCLUSION ......................................................................................................................... 51 APPENDICES .............................................................................................................................. 52 APPENDIX A: Agricultural DSS in the North Central Region ................................................ 53 APPENDIX B: Participant Consent Form ................................................................................ 55 APPENDIX C: Interview Guide – Developers ......................................................................... 57 APPENDIX D: Interview Guide – End-Users .......................................................................... 60 APPENDIX E: Recruitment Email ........................................................................................... 63 APPENDIX F: Participant Thank You Letter ........................................................................... 65 APPENDIX G: Codebook ......................................................................................................... 67 REFERENCES ............................................................................................................................. 78 v LIST OF TABLES Table 1. Overview of Case Study DSS ........................................................................................... 8 Table 2. Participant Affiliations .................................................................................................... 13 Table 3. DSS Roles of Development Team, End Users and Stakeholders ................................... 18 Table 4. Agricultural Decision Support Systems Available in North Central Region ................. 54 Table 5. Codebook ........................................................................................................................ 68 vi LIST OF FIGURES Figure 1. Example ACPF Output Data. .......................................................................................... 9 Figure 2. DEP Online Mapping Interface. .................................................................................... 10 Figure 3. GLWMS Interface and Example Output. ...................................................................... 11 Figure 4. RRAF Mapping Interface Example from Wisconsin. ................................................... 12 vii KEY TO ABBREVIATIONS ACPF: Agricultural Conservation Planning Framework ARS: Agricultural Research Service BMP: Best Management Practice DEP: Daily Erosion Project DSS: Decision Support System EMS: Environmental Modeling and Software EPA: Environmental Protection Agency GIS: Geographic Information System GLRI: Great Lakes Restoration Initiative GLWMS: Great Lakes Watershed Management System HUC: Hydrologic Unit Code HIT: High Impact Targeting IS: Information System LIDAR: Light Detection and Ranging L-THIA: Long-Term Hydrologic Impact Analysis NCRWN: North Central Region Water Network NGO: Nongovernmental Organization NRCS: Natural Resources Conservation Service NWS: National Weather Service RRAF: Runoff Risk Advisory Forecast SARE: Sustainable Agriculture Research & Education STEPL: Spreadsheet Tool for Estimating Pollutant Loads viii SWAT: Soil Water Assessment Tool TNC: The Nature Conservancy USACE: U.S. Army Corps of Engineers WEPP: Water Erosion Prediction Project ix 1. INTRODUCTION Environmental modeling and software (EMS) programs are often employed to enhance understanding of complex environmental problems by researchers, decision-makers, and natural resource managers. In particular, decision support systems (DSSs) have been developed in numerous contexts to bolster decision-making around natural resources at the individual, community or policy levels. DSSs are commonly described as computer-based tools that support the decision-making process when a given problem or solution is ill-defined (McIntosh et al., 2011; Lautenbach et al., 2009; Le Bars & Le Grusse, 2008). Other researchers emphasize the role DSSs play in eliciting values, beliefs and objectives (Portoghense et al., 2013; Bessette et al., 2019), facilitating end-user learning and shared understanding (Jakku & Thorburn, 2010; Reiter et al., 2017), increasing transparency (McIntosh et al., 2011; Fassio et al., 2005) and enriching group collaboration (Newman et al., 2017). DSSs have been applied in a number of environmental contexts including watershed management (Zhang, Chen, & Yao, 2015), groundwater management (Le Page et al., 2012), natural hazards including flooding and wildfires (Kochilakis et al., 2016), agricultural adaptations to climate change (Wenkel et al., 2013; Prokopy et al., 2017) and siting for renewable wind energy (Gorsevski et al., 2013). Environmental DSS development has benefitted from technological advancements over the last two decades in GIS (Tayyebi et al., 2016), web- based computing and mobile accessibility (Car et al., 2012), and real-time data integration (Easton et al., 2017). Despite an abundance of environmental DSSs, relatively few are evaluated to assess the extent to which a given system: (1) is adopted by end-users (Newman et al., 2017; McIntosh et al., 2011), (2) effectively informs a decision-making process (Matthews et al., 1 2011), and (3) contributes to broader social, economic or ecological outcomes (Matthews et al., 2011). Multiple researchers have attempted to identify factors that are attributable to the success of EMS products to improve evaluation efforts (Diez and McIntosh, 2009; McIntosh et al., 2011; Merritt et al., 2017; Newman et al., 2017; Gibson et al., 2017). For example, Diez and McIntosh (2009) evaluated over 250 factors that influence the usefulness of information systems (IS). In their extensive review, they captured multiple definitions of success for these tools, ranging from those focused on use and user satisfaction (Finlay and Forghani, 1998) to improved organizational effectiveness and efficiency (Irani, 2002; Olugbode et al., 2007). The researchers identified predictors of success across three phases of the IS life cycle: pre-implementation, implementation and post-implementation. User participation was the best predictor for user satisfaction and success in the pre-implementation phase, while computer experience, user support and top management support were among the best predictors of adoption during the implementation phase. In the post-implementation phase, user satisfaction was the best predictor of IS success. Later work by Merritt et al. (2017) categorized 33 success factors for EMS into the following groups: project management, project actors, stakeholder engagement, model development, model evaluation, expectation management, contextual factors, and model use. Their characterization of modeling and DSS success extended beyond adoption, encompassing other benefits such as social learning and expanded professional networks. After applying their success framework to 15 water resource models and DSSs, stakeholder interaction, trust and communication were among the top factors identified by participants as essential for a successful EMS project. 2 The importance of stakeholder engagement in the success and adoption of DSSs has been suggested by DSS developers (Sandink et al., 2016; Rossi et al., 2014; McIntosh et al., 2011). However, participatory approaches to DSS development may be underutilized. A recent review of 77 DSSs designed to address natural hazards found that stakeholder participation in DSS development was reported for only 13 DSSs (Newman et al., 2017). Even when developers employ an elaborate stakeholder engagement process, it may not be sufficient to guarantee DSS success. For example, Reiter et al. (2017) conducted an evaluation of a climate adaptation DSS that was developed using a highly participatory process. The researchers found that this was not enough to overcome organizational issues including the need for ongoing training and the lack of an organizational mandate to incorporate the DSS into common workflows. Despite extensive interaction with stakeholders, over 70% of end-users stopped using the DSS after implementation. Limited adoption of a DSS is a common challenge for DSS developers (McIntosh et al., 2011; Gibson et al., 2017). Effectively engaging end-users and sustaining the operation of DSS in the face of technology advancements, maintenance and rising costs are other issues that hinder DSS success (McIntosh et al., 2011). There are few reports in the literature where environmental DSS developers have overcome challenges of adoption and inadequate evaluation. Rossi et al. (2014) present a well- adopted DSS (vine.net®) designed to assist on-farm decision-making for vineyard managers, which supported nearly 55,000 logins in a six-month period. Researchers surveyed 21 end-users about their use and satisfaction of the respective DSS. Of note, 93% of survey respondents used vine.net® at least once a day and 100% of these users indicated that the system helped them make better decisions. 3 Measuring success solely by adoption of the DSS does not capture other benefits that can be realized throughout the DSS lifecycle. Examining DSS success through the lens of boundary objects may offer additional insights. Boundary objects, first described in Star and Griesemer (1989), allow multiple stakeholders and users to adapt the object to meet their specific needs through interpretative flexibility. In Jakku and Thorburn (2010), researchers developed an irrigation DSS using a participatory process. The DSS served as a boundary object between researchers and farmers, facilitating co-learning related to irrigation water management. While the DSS was not adopted by end-users (Thorburn et al. 2011), those involved in the development process gained a better understanding of optimal water management. Framing the DSS as a boundary object that enables shared learning provided greater understanding of system success than adoption alone. While recent contributions examining success indicators for DSSs are promising, issues remain. As noted in Reiter et al. (2017), many of these attempts to analyze success factors are heavily focused on researchers’ perspectives. For example, in Merritt et al. (2017), only modelers and DSS developers were surveyed about their perspectives on the success of their own EMS products rather than end-users. When end-users have participated in DSS evaluation, the sample size is often small (Gibson et al., 2017; Inman et al., 2011) or focused on short-term influences of DSSs rather than long-term outcomes (Newman et al. 2017). 1.1. Study Objectives This study seeks to address research gaps associated with DSS development, implementation and evaluation by exploring the extent to which four DSSs have been successfully deployed to facilitate decision-making and support agricultural conservation within 4 the North Central Region of the United States. This study incorporates perspectives of both developers and end-users to shed light on DSS successes and challenges. The study objectives are to: 1. Characterize the purpose, target audiences, and uses of the DSSs 2. Examine the extent to which the DSSs have been successfully deployed and adopted 3. Explore factors that contribute to the long-term sustainability of the DSSs 1.2. Study Area The North Central Region, consisting of 12 states in the Midwest (IA, IL, IN, KS, MI, MN, MO, ND, NE, OH, SD, WI), is a suitable region to explore DSS implementation as it relates to agricultural conservation and water resource impacts. The region is highly agricultural, with over 336 million acres of farmland in 2019, comprising approximately 37.5% of farms in the U.S. (NASS, 2020). The region contains thousands of miles of rivers and streams, including tributaries that drain into the Gulf of Mexico and four of the Great Lakes. Agricultural runoff is a primary driver of harmful algae blooms in Lake Erie (Watson et al., 2016; Smith et al. 2015) and hypoxic conditions in the Gulf of Mexico (Rabalais et al., 2019; Bianchi et al., 2010). Millions of federal dollars are invested each year in voluntary conservation programs to prevent agricultural nonpoint source pollution. For example, the Natural Resources Conservation Service (NRCS) administers the Environmental Quality Incentives Program (EQIP), which provides financial and technical assistance for farmers and landowners to implement conservation practices. The Mississippi River Basin Healthy Watersheds Initiative delivers conservation to farmers through EQIP to address nonpoint source pollution issues and has funded $307 million in conservation contracts between 2010-2019 (NRCS, 2019). In select 5 watersheds within the Great Lakes, the U.S. Environmental Protection Agency (EPA) through the Great Lakes Restoration Initiative (GLRI) allocated $79.3 million for conservation practice implementation between 2010-2016 (Gold, 2020). DSSs have the ability to assist government agencies, conservation professionals, and farmers in addressing nonpoint source pollutants and executing effective conservation planning and delivery programs. 6 2.1. Case Study Selection 2. METHODS I used a two-pronged approach to identify possible DSSs for inclusion in the study. First, I conducted a search of peer-reviewed articles using the Web of Science database and Google Scholar, querying for terms including: decision support system, decision-making, water, agriculture, water quality, conservation and state names. Given past experience promoting DSSs, I anticipated that some systems used by practitioners may not appear in peer-reviewed publications. Therefore, I also reviewed gray literature and used my professional experience and contacts to scope out potential DSSs. For example, I reviewed resources listed by the North Central Region Water Network (NRCWN) and Sustainable Agriculture Research & Education (SARE) – North Central Region to identify possible DSSs. Both NCRWN and SARE are affiliated with the land-grant Extension programs in the North Central states. I used five criteria to select the final cases. First, I utilized the definition from Rizzoli & Young (1997) to define what constitutes a DSS: “software systems that integrate models, or databases or other decision aids, and package them in a way that decision makers can use”. Second, the system needed to still exist. Some papers published over ten years ago discussed DSSs that may have been suitable for this project, but had little evidence that they were still active. Third, the DSS must have evidence of use by individuals and organizations outside the development team. Given that DSS have low rates of adoption, I was not concerned with the extent of users, but wanted to ensure there would be enough individuals to interview about their experience with a given system. Fourth, I wanted to include a diverse set of DSSs, considering both the conservation and water quality issue that the system was intended to address, as well as including systems with different technologies and features. Fifth, I selected DSSs that are 7 available in multiple states to explore funding pathways and other aspects that may be associated with system expansion and longevity. The four systems selected for this study include the Agricultural Conservation Planning Framework (ACPF), the Daily Erosion Project (DEP), the Great Lakes Watershed Management System (GLWMS), and the Runoff Risk Advisory Forecast (RRAF). Table 1 provides an overview of each system and its purpose, audience, features, scale, and available locations. Additional DSSs that were identified in my initial inventory and by research participants are detailed in Appendix A. Table 1. Overview of Case Study DSS DSS Location Scale Purpose Audience Features HUC12 BMP siting; watershed planning Conservation Staff, Government, NGOs Researchers Toolbox within ArcGIS; Outputs available online for select watersheds Government, Farmers, NGOs, Researchers Internet browser Internet browser; Batch evaluation; Report generation Internet browser; text messaging ACPF DEP IA, IL, IN, MD, MI, MN, OH, SC, WI IA, KS, MN, MS, NE Estimate soil HUC12 loss from water erosion Quantify GLWMS MI, NY, OH, WI Field- scale environmental Conservation Staff, benefits of BMPs Farmers, NGOs RRAF MI, MN, OH, WI 6 mi2 grid Reduce runoff risk of nutrient applications Farmers, Manure Haulers, Nutrient Applicators, Extension 8 Agricultural Conservation Planning Framework The ACPF was developed by researchers at the Agricultural Research Service (ARS), part of the U.S. Department of Agriculture. The DSS is a toolbox within ArcGIS that helps conservation planners identify suitable locations for structural BMPs on the landscape. ARS was a partner on a Conservation Innovation Grant submitted by the Environmental Defense Fund in 2011, where the ACPF’s underlying algorithms were initially developed and applied with the conservation pyramid (Tomer et al. 2013). This work led to the creation of the ACPF (Tomer et al., 2015). Users must set up the ACPF within ArcGIS to generate outputs on a watershed basis. In some watersheds, ACPF outputs are available online. NRCS is a predominant funder of the system. The system was piloted in several states to improve the system’s accuracy in recommending BMP locations in a variety of landscapes. Figure 1. Example ACPF Output Data. 9 Daily Erosion Project The DEP (Gelder et al., 2018) was developed by researchers at the Iowa Water Center at Iowa State University to estimate soil erosion in real time using rainfall data spatially distributed across a region. The initial modeling work started around 2000. The base model for the system is the Water Erosion Prediction Project (WEPP) model. Around 2010, the developers started looking at using LIDAR to estimate soil cover and residue, which was incorporated into their modeling work. The model outputs are displayed in an online mapping interface and updated daily. The underlying databases for DEP are the same as the ACPF. The DEP has expanded into other states as interested entities learned about and expressed interest in having the information available in their state. The DEP is one tool that the Minnesota Board of Water and Soil Resources is utilizing to better understand annual soil loss throughout the state. Figure 2. DEP Online Mapping Interface. 10 Great Lakes Watershed Management System The GLWMS (Fales et al., 2016) was developed at the Institute of Water Research (IWR) at Michigan State University and quantifies environmental impacts from BMPs using multiple models including the HIT, L-THIA, STEPL and SWAT models. The underlying soil erosion and sediment models were initially developed in the early 2000s in collaboration with the U.S. Army Corps of Engineers (USACE). In a follow up project funded by the USACE, IWR and Purdue University made their respective pollutant load models available online. The first version of the GLWMS was developed when developers merged the USACE work with an online, dynamic sediment calculator designed for the Michigan Chapter of The Nature Conservancy (TNC). TNC has since been a major champion of the system. Through support from TNC projects, additional functionality has been added including batch field evaluations, report generation, and incorporating modeling for estimating groundwater recharge and wind erosion. IWR has funded additional expansions through federal programs like the GLRI. Figure 3. GLWMS Interface and Example Output. 11 Runoff Risk Advisory Forecast The RRAF (Goering, 2013) helps farmers reduce runoff risk of nutrient applications. The model underlying the RRAF was developed by the National Weather Service (NWS). The DSS is available in four states, with each state funding the development of its own online interface. The project started around 2005 in Wisconsin after manure runoff was linked to fish kills and well contamination. The state legislature approached a state agency about building an online tool to discourage manure applications when runoff was likely, which led to the creation of the first RRAF system. The lead modeler manipulated an existing NWS model that does real-time soil moisture and runoff modeling and applied it in this new capacity for the pilot DSS. In 2014, EPA wanted to expand the tool to states within GLRI priority watersheds for Focus Area 3, which started the version 2 development cycle. GLRI is supporting the third RRAF version, based on the NWS National Water Model, which is finer scale and covers the entire lower 48 states. Figure 4. RRAF Mapping Interface Example from Wisconsin. 12 2.2. Data Collection Study population and sampling The study population consisted of developers and end-users for DSSs deployed in the North Central Region to address agricultural conservation and water quality issues. The sample was derived from developers and end-users associated with the ACPF, DEP, GLWMS and RRAF. For this project, developers encompass programmers, modelers, lead researchers and project managers that played a critical role in either the conception, modeling, programming or maintenance for the DSS. End-users comprise a diverse array of governmental, nongovernmental and private sector entities, including government agency employees, field conservation staff, watershed coordinators, and farmers. See Table 2 for participant affiliations. Table 2. Participant Affiliations Participant Organization Type Conservation district Extension Farmer Federal agency NGO State agency University (non-Extension) Total Number of Participants 4 3 1 3 4 4 5 24 I used purposive sampling strategies to identify participants. I identified developers using key informant sampling and end-users through snowball sampling. I initially asked developers if they could provide me with contacts for end-users, and also asked end-users to refer me to additional users if possible. All participants were recruited via email. I conducted 24 interviews between May and July 2020. Seven participants reflected on their experiences with ACPF, three on DEP, six on GLWMS, and eight on RRAF. Three individuals contacted about the study 13 declined to participate and four did not respond. Interviews were conducted over Zoom software or by phone and typically lasted about 30 minutes but ranged from 15 minutes to one hour. Instrumentation All interviews were semi-structured. I developed tailored interview guides for developers and end-users. The developer interview guide consisted of 17 questions and the end-user instrument consisted of 19 questions. The tailored guides asked questions related to the purpose, use and promotion of a given DSS; the interaction between end-users and developers, if any; the level of confidence in outputs generated by a system; challenges related to a participant’s experience with a DSS; and outcomes derived from the systems. While I asked a standard set of questions of participants, I probed on emerging concepts that deviated from the guide. The interview questions were derived from key concepts in the literature related to decision support system success. I reviewed the 33 success factors for EMS products identified by Merritt et al. (2017). I also incorporated additional factors from other synthesis papers, including DSS influence on decision-making (McIntosh et al., 2011), user support and training and institutional constraints (Diez & McIntosh, 2009). 2.3. Data analysis I transcribed data using Sonix.ai software and corrected transcripts for errors. These data were imported into NVIVO 12 software for coding and display analysis. I used thematic coding during preliminary analysis to identify important concepts and themes that emerged from the data and developed a codebook. To complete preliminary analysis, I extracted all the data for each code by research subject to develop summary statements that characterize what the data 14 portray in relation to the research objective. I also wrote memos and created displays and matrices to facilitate analysis. 15 3. RESULTS The results are organized around six core concepts that examine the ways in which the four DSSs have been successfully deployed to facilitate decision-making and support agricultural conservation: 1. Purpose, Audience and Uses: What is the DSS intended to do, who uses it, and how? 2. Outreach and Education: How is the DSS promoted to end-users? How are end-users supported in their use of the tool? 3. Adoption: What is the nature and extent of use? 4. User Satisfaction: Is the DSS perceived to be useful and usable? Are users confident in the predictions or recommendations made by the DSS? 5. Broader Outcomes: In what ways is the DSS supporting conservation delivery, improved water quality, enhanced viability of agriculture, or other outcomes? 6. Sustainability: What factors contribute to or limit the sustainability of the DSS? 3.1. Purpose, Audience and Uses Purpose The ACPF, DEP and GLWMS are all intended to identify areas in need of conservation treatment but do so in slightly different ways. For example, ACPF evaluates landscape characteristics that are favorable for certain structural BMPs and produces maps with recommended locations. The system can also assess the level of implementation needed for a given practice. In contrast, the GLWMS has high risk maps for soil loss and sediment loading to streams that help identify areas for tillage BMPs. In addition, it quantifies the environmental benefits of select BMPs at the field level. In contrast to the other systems, the RRAF focuses on 16 nutrient management, specifically the application of manure and nutrients on farm fields. The system predicts the potential for runoff based on forecasted precipitation, soil moisture conditions and other factors, helping nutrient applicators plan applications when risk is low. I compared responses between end-users and developers in relation to the purpose and use of a system. I did not find a single case where the stated purpose and intended use by developers differed drastically from how the end-users perceive and use the system. However, I did find instances where end-users found additional value or uses for a given DSS beyond what was stated by the developer. For example, some users talked about DSS outputs helping with farmer engagement, which developers did not discuss. Audiences and Uses Across the four DSSs, there is a complex web of individuals and organizations that use or support the systems in varying ways to advance agricultural conservation. It is important to understand the nature of roles related to DSS development, project management, maintenance, funding, ownership, promotion, training, and user support in addition to examining the intended users of the systems. I found that many participants serve multifaceted roles in relation to the support and use of the system. For example, an “owner” may not necessarily be the “developer” or “modeler” that initially conceptualized and built the DSS. The “developer” may be the sole “promoter” and “user support” for the system, whereas other DSSs have multiple entities serving in these roles. Furthermore, distinction needs to be made between a “user” who is able to run and generate outputs from a DSS versus the “user” who actually uses the information to inform their work and decision-making processes. Table 3 highlights the various roles that I documented in the interviews related to the various systems. 17 Table 3. DSS Roles of Development Team, End Users and Stakeholders Role Description Funder Owner Funds the development, improvement, expansion, promotion, training and/or evaluation of a DSS. Often ambiguous, but tends to be associated with the organization that hosts the interface. In some cases, the owners are considered the original researchers and modelers that conceptualized the DSS. Project Manager Obtains funding, coordinates staff and partners, and serves other administrative functions related to development, improvement, expansion, promotion, training and/or evaluation of a DSS. May be distributed across partners. Researcher Publishes in peer reviewed literature to advance knowledge and “promote” work. Modeler Develops model; validates model, sometimes in collaboration with other researchers Programmer Programs and updates the user interface Promoter Promotes DSS to intended users and/or to potential collaborators, partners, funders Trainer Trains users how to use the DSS User Support Assists users in answering questions, addressing technical issues Evaluator Collects user feedback on the DSS Intermediary User User Generates DSS outputs for use by others Utilizes DSS outputs to inform work and decision-making processes Government Agencies Government agencies at the federal and/or state level have invested funds to support the development and promotion of all four DSSs. In multiple cases, the agencies are not direct users but serve project management and promotion roles. All four systems are utilized to advance 18 and/or track progress toward state nutrient reduction strategies. One system is being used in multiple states for 319 watershed grant programs. Nongovernmental Organizations (NGOs) NGOs utilize the DSSs to support watershed planning, conservation delivery, and in limited cases, for developing policy briefs. For one DSS, two NGOs use the system to track progress and report the environmental benefits of their projects to public and private funders. They also train field-level staff who use the DSS on a regular basis for farmer recruitment. The system supports decisions at multiple scales, as described by this manager: It affects the decisions we make in our job in terms of where we are targeting and ultimately implementing conservation practices. The [DSS] has been the default prioritization tool. I mean, pick the scale you want to work at. If we're talking about where do we just launch an entire new project? Honestly, we're using the [DSS] to find out where are those hotspots, those impacted watersheds, where would be a good place to go work next. So in a project, then we're working within the target area – well, where are the priority actual farms, where are the priority fields within that farm, even? So, at every step of the way, whether it's project conception and development, we're using it. Intermediary Users There are intermediary roles for individuals that promote a DSS and/or generate its outputs for intended end-users that utilize the outputs to make a decision. For example, field- level staff may lack both the technical expertise and the time needed to generate DSS outputs. External parties, whether housed at a government agency, NGO, or consulting firm generate the outputs and provide shapefiles and/or paper maps to the field-level staff that then use the information for conservation purposes. This type of arrangement was typically associated with DSSs that required a higher level of GIS experience and competency. Intermediary users also comprise those that serve in an outreach capacity and play an essential role in promoting the system (e.g., Extension educators). While they are not the intended end-users, the DSS is helping them provide additional technical assistance to farmers 19 and landowners in implementing best practices. Below an Extension educator shares their reflections: We've got two or three groups of farmers in [state]. We've got the group that is really concerned about nutrient movement, and they probably were doing the right thing long before we started talking about the problem with Lake Erie or the Gulf of Mexico. We've got another group that kind of needs a push to make them think about those problems…If you can somehow get the middle group aware…and the bottom group who doesn't care— if you can put an easy tool in their hands, they're more likely to use it to actually help with the problem. That's why I try and find valuable tools. Field-Level Staff Conservation technicians, watershed coordinators, and other local field staff utilize DSSs to plan, prioritize and implement conservation on the ground. These tools help identify areas in need of conservation treatment at the field-scale based on landscape characteristics such as soils, slope, and proximity to waterways. For one DSS, field-level staff utilize the system’s “hotspot” sediment maps to identify potential locations for conservation treatment. Another DSS has been helpful in watershed planning, especially for organizations that lack capacity to develop formal watershed plans, as this user explains: It can be used in lieu of full-on watershed planning, which I think is just incredibly valuable for smaller groups…It's this paradox. If you don't have enough money to have paid staff, you can't even achieve the minimum level of organization necessary to apply for a grant…And for a group like us to get, you know, a quarter of a million dollars to write a watershed plan, it's probably not going to happen. But we can use the [DSS], which is this free and accessible tool to show people where we might have these practices and then we can take other things into consideration when making prioritization decisions. Field-level staff also use DSSs to prioritize outreach efforts when promoting conservation programs and recruiting farmers. For example, they will use output maps to determine locations to host farmer meetings or identify addresses for sending targeted mailings to producers. Whether these targeted efforts lead to more farmer participation is an open question. One user, who didn’t explicitly quantify return on investment from targeted mailings, felt that while they 20 were a “slight improvement” over blanket mailings, they would not “solve all the outreach challenges”. Another user found targeted mailings to be a complete failure: It's just not been a successful method of recruitment…So it's showing us all of those concentrated areas that are susceptible for soil erosion. And we spent a lot of money putting postcards and letters in those farmers’ mailboxes, letting them know, 'Hey, you're in a high priority area. This is gonna be a great program for you.' We spent like, I don't know, $5,000 on this mailing. It generated zero leads. For one user, targeted outreach was successful when comparing map outputs with her local knowledge of willing landowners: We use that data to really host some local meetings to try to generate interest in certain practices. And then it's a good tool for us locally because we know a lot of these landowners. We know what they're willing to do, what they're not willing to do. So it gives you a good starting point for identifying good locations for the practices and then seeing whether you've got a willing landowner to go with it. In some cases, participants use the outputs to engage farmers in a more comprehensive conversation about their land and potential conservation treatments, as described by this user: It's just a visual aid that brings it full circle to the producer so we can talk about sediment leaving their field. We can talk about how cover crops or no till is really going to help. But when they look at it, they can see the numbers. They can see their problem area showing up on the program as far as where they see them in the field. I think it just helps kind of complete the conversation. Of the four systems, the GLWMS is the only system that can quantify environmental benefits of conservation practices, such as the amount of phosphorus or sediment loading reduced. This system is being used in innovative pay-for-performance conservation programs, that pay farmers not on a per-acre basis for a practice, but rather on the pollutant reduction they will achieve for implementing a practice. Field-level staff run batch evaluations to determine high priority fields for the program that will net the highest payment for the farmer. At this time, research is underway that couples the other DSSs with models that can assess potential environmental benefits. 21 Farmers Farmers who apply manure and nutrients are a primary audience for the RRAF, in addition to manure haulers and other nutrient applicators. The system helps applicators plan when they’ll spread manure based off runoff risk potential. There’s an open question whether farmers should be considered end-users of the other systems. This was a sentiment shared both by developers and users. One DSS has been designed for use by farmers but ultimately is used by conservation staff, as this developer describes: Initially we wanted to design something that would be useful and usable by farmers. Again, most of the work was focused on trying to support decisions in agricultural management…So we're trying to make it as simple and usable to the widest possible audience. But that said…it's mainly conservation district technicians who've utilized it. For another DSS, a manager questions the accessibility of the system for farmers: We say it's accessible to people, but by that we mean it's free. That doesn't necessarily mean it's technologically accessible to everybody. I think we're taking the steps to try and address those things and overcome them. But yeah, really asking the question...If I'm a single producer, am I going to use the [DSS] for anything? Or is it really a tool for people like our watershed coordinators and district conservationists and people like that to use as an outreach tool? Researchers As researchers, individuals on a DSS development team often publish their work related to the system. The DSSs also enable additional research endeavors beyond the initial intent of the developers. For example, one social science researcher is using DSS outputs in an experiment for farmer engagement strategies. Other researchers are exploring ways to pair DSS outputs with other models to quantify the environmental benefits of recommended practices to better understand how much conservation is needed to meet nutrient reduction goals. 22 The Public With three of the DSSs being readily available online and free, they are accessible to the general public. Despite not being an intended user group, one developer shared that community members in their state are purportedly using a DSS to catch noncompliant farmers: One of the things that we hear from some of the bigger operators that use the tool, and there aren't all that many of them, but occasionally we'll hear that community members will be watching the site and comparing what they see happening in the field to what color the map was that day. They're trying to catch them being bad. 3.2. Outreach and Education Promotion Promotion for the DSSs was multi-faceted and consisted of three approaches: (1) promotion to intended user-groups of the system; (2) promotion to the broader agricultural and conservation community where developers and partners may interact with potential collaborators and funders that are interested in expanding the DSS to new areas; and (3) promotion to the research community through publication. Promotion to End-Users The strategies used to reach end-users depends on funding, staffing, time and target audience. For systems with limited outreach resources, the DSS are often promoted through presentations at various events. Other times, the development and implementation of a grant project is the method of promotion. For example, a DSS project manager connects with and gets commitment from an NGO to utilize the system for a conservation program. In turn, that NGO trains the boots-on-the ground technician that uses the system regularly. All systems have fact sheets, and those with additional resources have developed promotional videos, purchased giveaways for exhibits, and/or advertised through radio, TV, and 23 print ads in agricultural news services. The DSSs are also promoted through one-on-one interactions and trainings. With three of the DSSs having dedicated online interfaces, and one having an in-depth informational website, the Internet is also a means of promotion. Some users discover the systems through internet searches. Promotion to Conservation Stakeholders Presentations are the most common way that the DSSs are promoted, whether for academic or practitioner audiences, at conferences, webinars, field days or other events. For two of the systems, presentations opened the door to new partners, possible expansions and funding opportunities. For example, one DSS expanded into four other states from the interaction with interested parties at conferences, regional meetings and events. For another DSS, one presentation to a key decision-maker helped significantly expand the system, as one developer recounts: We went down to present what we had done at the [granting agency center]. There was one of their lead people that was just in the building and decided to sit in on one of the presentations we were giving. And by the time it was over, he decided that it would be good to support this. And he did. And so, it's ended up with about two million dollars in initial funding…And some of these things end up being serendipitous, really. Having the right person in the right place who goes, 'Wow. I can use that, and I can see a way I can help you help me use it.' And that's kind of how this started off. Promotion in Research For some researchers on the development team, publishing research is seen as their primary strategy for promotion. As researchers, advancement of knowledge and publishing in peer-reviewed articles is essential to their job, as one researcher describes: Well, writing papers, getting those published. You know, my job is a research scientist and the most important thing for me is to publish in the peer reviewed literature. And so, you know, if I couldn't use this as a mechanism to do that, there would have been little reason for me to do it. 24 Promotion Challenges Promotion can be hampered by the accuracy of the DSS. Across all four systems, the underlying models represent an evolution of years’ worth of research and effort, and of course, no model is perfect. However, end-users do not necessarily care about the scientific processes involved in modeling; they need a tool that is accurate and helps them in their work. A bad first impression can result in a one-time user. This is especially true for farmers, as one DSS manager details: We have definitely seen signs of where it's been overpredicting…We know we only really get one chance with people, but we don't want to really sell hard and then have it predict runoff five days straight for somebody and it doesn't rain at all. And then they'll never look at it again. And [developer] did do some tweaks…And I think this year it's been exceptionally better…So I would say that could have been a little bit of a stumbling block in the beginning phases of just trying to make sure that we feel like it's working well enough, that we want to stake our reputation on selling it to somebody. Lack of funding can also hinder promotion, and not just in the traditional sense of not having money to develop print or digital materials, but in terms of making outreach a priority. With no budget, managers aren’t necessarily thinking about it, as this individual describes: But because it's also a tool that basically runs free right now…We don't have to have these conversations every month of like, ‘Oh, how are we going to pay for this?’ Which then I think would also leads to a bigger conversation that we need to get out there and publicize it, because it's just information that we get for free and the website is already up and running. Finally, research publications may be challenging for developers as they juggle the management of the system. One developer noted that the time required to manage the system as it has expanded into other areas has decreased their team's ability to write manuscripts. Their solution to this problem leverages international scholars that can apply for grants in their country to work in the U.S. for a year and lead publication efforts. Relatedly, funders that invest in system expansions may be focused solely on the DSS as a product rather than a research 25 application. One developer is cognizant of not losing focus of their research mission, as they explain here: It's money that's coming from expansion. And other things from other states that…aren't that interested at this point in time in terms of new technological or scientific advancement...So we have to be on guard against being more than just a service. We are an academic institution and people that are working with us get promoted based on scientific advancement. So there's a balance there that we need to maintain. Training and User Support The other key component for outreach and education is training and user support. Given the variety of features and varying complexity across the four DSSs, some require more training and expertise than others. Some systems have informal training and user support. For example, one developer onboards stakeholders and project managers at the start of a new project and works with them throughout the project to address issues and questions. From the end-user perspective, this system is “easy to use” and “straightforward” and requires little training, as one manager describes: So we hired a new assistant earlier this year and no training was hardly required. Like I thought, 'OK. Here's another technician we've got to bring on board.' We set him up with the [DSS] in front of him and showed him a couple of the main buttons to get to the models that we need. By the end of the afternoon, he was he was fluent in it, and I think that's a real testament to the usability of it. Other systems have established training programs and materials. In one case, universities lead the training efforts. A technical training details how to use setup and generate outputs with the DSS, whereas an output training educates practitioners on how to use the results when communicating with producers. This system also has a user manual and extensive video library, which multiple users said were very helpful resources. 26 Overwhelmingly, end-users praised the developers of all four systems for their level of responsiveness when they had questions or issues. For users that encountered bugs or issues, the interaction with and response of the developers factored into their experience, as described by this user: There was always some sort of small issue when I used it, which it was frustrating when I was mapping a lot of acres because that's when it would really freeze up on me or just wanted to stop working if I was on it for a long time…The most frustrating thing was having to kind of quit in the middle of a project and go back to it. But there were, I mean, there was always updates. I know they would always look for feedback from us. And it seemed like they always provided those updates or were working on them. Training and User Support Challenges Few participants shared challenges related to training and user support. One manager described an instance where the internet was too slow at a workshop site, and none of the participants could actually load the interface to use it. Another user cautioned about having the appropriate infrastructure for user support: And I think you have to have some type of structure where there are some go-to people in the agencies that when other people to try use this and develop the maps and they run into problems, that they got some place to go to get some hands on, one-on-one in-person assistance. You know, [the developers], they had four people out there in that shop year and a half ago. And, you know, now they're down to two. And so they haven't been able to do the follow up that they used to as the tool grows. So there's some work going on to try and set up a structure to keep that going to do that. 3.3. Adoption Adoption broadly describes the extent of use of a DSS. The most common way that the developers and managers thought about and evaluated adoption is by the number of users, typically assessed using website analytics. However, number of users and website hits only provide so much insight into what it means for an individual or an organization to adopt a DSS, and true measurement of adoption can be difficult. For instance, one manager shared an anecdote 27 about a user that had a bad experience with a system, and never used the system again. Should that one-time user be considered an adopter? Further, infrequent use of a system may not imply a lack of adoption but reflect the temporal nature of the decisions the system is intended to support. A more meaningful way to evaluate adoption is the extent that a system is embedded within the planning and decision-making processes for an organization or an individual. I considered both approaches when assessing the nature and extent of adoption for the systems. While each DSS has been and continues to be used by various entities, there is significant variation in adoption across the four systems. Evaluating Adoption by Number of Users For the systems available online, analytics data is typically collected and used as an indicator of adoption. Multiple participants noted that the analytics data showed few users. The sentiment expressed by this participant was common among DSS developers and managers: You know, just to be completely honest with you, there haven't been as many users as we would have hoped…You know, we have some people that use it and I talk about it pretty much at any presentation I can give. But still…I'd like to see more users. While not supported by analytics data, one DSS promoter had doubts of system use based on the receptiveness of potential end-users during demonstrations of the tool: So I've been at this a while. And I can usually tell by looking at their eyes, sitting in the room when I'm talking, whether or not they're listening. And every now and then someone lights and you go, 'OK, that person is going to use this tool.' But it doesn't feel like there's a whole lot of people using the tool…It concerns me that when you bring a nice like this to them, there should be, you know, 30 people that day who get on it. And I don't know that that's happening. Measuring adoption purely on total hits or users may oversimplify the nature of how these tools support conservation decisions. For example, decision support for runoff risk varies temporally throughout the year: 28 We do analytics on the website, which is, kind of, often disappointing. For instance, we had 12 users today. I mean, you know, the massive rains that we're getting. I don't know, maybe people have internalized what the tool does by now or—But we've never had more than a few dozen users except in some really unusual circumstances…like when we had really up and down snowmelt…Then we'd see more users, and occasionally we'd see spikes if there was some sort of press release or radio interview…Back in last January, that might be our all-time record. We had 400 in January '19. Temporal variability is also evident for systems supporting conservation programs, where overall use is infrequent, but increases during sign-up periods: It goes through spikes of usage. There are times where it might not get any hits on a particular day, but then there might be an enrollment period for a particular conservation program that a conservation district may be running. And then we'll get a lot of traffic on the system…Some days it might just get a trickle of users who stumble upon it through Google or some other means. But it's really been driven by the needs of our partners and their projects. Evaluating Adoption by Level of Institutionalization Indicators of institutionalization include buy-in from top management or key influencers that recommend or require the use of a DSS, continued funding over multiple years by a government agency or NGO, and use of the DSS in routine conservation-related decisions. Two of the DSSs have received significant investment and buy-in from government agencies. For one DSS, a multi-agency and interdisciplinary team is conducting an assessment to evaluate an organization’s readiness to deploy the system nationally, and state agencies in at least three states recommend the system for developing watershed management plans. Another DSS serves as an NGO’s primary conservation prioritization tool, and the organization has heavily invested in the system’s development and expansion over multiple years. While these systems have strong support at an organizational level, the routine use of system outputs in decision-making processes at the field-level is not always realized. For example, this state agency coordinator 29 reflects on their organizational commitment to utilize a DSS but is less certain of field-level users: We had one project where we've done the full-blown version of this. We have probably three or four underway right now. And we have absolutely built this into our workflow at this point. This is standard fare for our agricultural areas now. And so we haven't really gotten into that next step. We had hoped this initial grant we gave these people would have...they would have taken that next step of BMP implementation. And we're not sure whether or not they've gone down that road. We have some high confidence in a couple of the ones we have going on right now that that's exactly how our grantees are going to use it. Once we get everything done, we have a high expectation they're going to be approaching producers. So, I mean, that's the ultimate goal for us. There are also mixed results with intermediary users that generate outputs for use by others in their conservation work. One intermediary user has provided DSS outputs to field staff that are actively using the information and want to develop capacity of their own to run the DSS. He also knows of a county that has “run every watershed for prioritizing their staff work.” Both examples suggest repeated use of the DSS for conservation planning and delivery. However, the user has also had the opposite experience, in part because of miscommunication, as he describes here: So I handed them over the actual output. And they got mad at me because they couldn't run it on their GIS system…They're like, 'Well, what did we pay for, you know?' And I'm like, 'Well, you paid for an output from a program that I gave.' And I gave them hard maps, too… But they're like, 'Yeah, but we wanted the interactive stuff.' And I'm like, 'Well, then you got to pay for a GIS license, I'm sorry.'… They did a 9-element plan. And all they did was they stuck some of the hard maps I gave them into it. And that was the end of the use, and I was like, you know, this is worth so much more than that...And that's why I call it kind of a failure. And part of that's on me…And now when I talk to people about it, I'm like, ‘I need to know what you want to do with this and what capabilities you have internally. Because I don't want to just give you something that's going to end up getting stuck in a computer or an extra drive somewhere and nobody's ever going to look at.’…And again, this is not me vilifying the people who don't have the capability or program. It's again, is it really worth it to make the runs if we're not going to actually have you be able to use them for anything? And a lot of times those agreements also are proprietary. So that means once I give it to them, I can't even use it for anything else. So again, what was the end game? For us to store this somewhere and then nobody gets to use it? 30 This vignette illustrates multiple challenges related to deploying DSS and working with stakeholders, but poignantly exemplifies the difficulty in defining and assessing adoption. The system technically was used by the organization in their watershed plan, but the user highly doubts that the outputs will actually be used to plan and delivery conservation. The nebulous nature of intermediary users was also evident for another DSS, which is readily utilized in multiple programs coordinated by an NGO to document the environmental outcomes that are important to their funders. However, the primary users of the system are field staff that run the tool when they recruit farmers to participate in a conservation program. The system has not been institutionalized at the field level. For example, one conservation technician used the DSS during a five-year project with over 100 producers. The project recently concluded, and while she thinks the system is useful beyond that particular program, she hasn’t used the system since. One manager noted that conservation districts are extremely underfunded and “running super lean” and “can only really do what somebody is paying them to do.” This participant was also familiar with a district that saw value in getting the environmental metrics, but requested that the NGO provide the outputs because they “didn’t have the time or bandwidth”. Capacity and resources appear to be a limiting factor in long-term adoption of the system at the field-level. Understanding and measuring adoption is further complicated by ensuring that the DSS is adopted by the individuals that have decision-making authority. While I was only able to interview one farmer, his experience was very enlightening. He registered to receive text messages from the DSS with the help of an Extension educator. While he receives and looks at the text messages, he is no longer the day-to-day decision maker for manure applications. That 31 responsibility now belongs with his son who uses other methods to make the determination of when to apply manure. A possible barrier to adoption is the mere existence of too many tools. Furthermore, some potential end-users may already have methods in place that help them in their conservation endeavors. One user explains: There's a lot of tools out there. Sometimes when you get so many tools, it's number one, annoying. Number two, you're not going to go look for all of those. You're only going to look for those that actually make sense for you. And if there's already another tool that's working for you or you have a method that's working for you already, why go search for something else?...You have to ask the question, is this actually going to help the farmers or the end-user, or is this just gonna be another tool out there that people are just like, 'This tool isn't really going to add anything that I don't have already.' 3.4. User Satisfaction Ease of Use User satisfaction is often related to the ease of use a system. In many instances, participants felt the DSS they utilized was easy to use. Nine participants described this in multiple ways, stating a given system as “straightforward”, “easy to use”, “not convoluted”, or “simple”. In other instances, the “helpful prompts”, “few inputs” or ability to be “self-taught” and “plug and play” is what made a DSS easy to use. One conservation technician elaborated on this: It does a good job of stepping you through what you need to input. So when you're running a model on a field having the first step be its own window. And when you complete that, you move to the next thing and everything seems very manageable. And it's not as overwhelming as just looking at a spreadsheet and seeing a bunch of numbers. Ease of use is also important to the intermediary end-users that help promote and train individuals the DSS. One DSS promoter explains their appreciation for having a simple DSS: 32 As an educator, it's really nice because it's easy to explain. If it's a complicated tool, I stumble over my words. Sometimes I stumble over what I'm trying to say, which doesn't help when I'm trying to educate other people on how to use it. So the fact that I can express what the tool is very readily, I think it's very telling to the fact that it's user friendly. The technical skills and experience of an end-user heavily influence a tool being perceived as user-friendly. An easy to use tool for one person may be extremely difficult to navigate for another. This dynamic occurred across the four DSSs. Individuals that perceived a DSS as difficult to use, particularly at the field level, expressed that they would not run the tool unless it became simpler to use, but did feel comfortable using the outputs if they were generated by someone else. One user reflected on re-evaluating their GIS skills based on their experience using one of the DSSs: It was a wake-up call, really working with it. I thought that I had a pretty decent GIS background, so I thought I was gonna be good. And when I got in there, I started realizing that, yeah, you're getting into the craziness of this actual program where even the way you name things and where you put them will completely throw the thing off…So, that might not sound like a big deal, but it becomes a very big deal for folks who don't really understand. 'Oh, wait. Where the hell did that go?' In your computer, these are hidden folders. And you're like, 'I don't know where the hell it is.' Confidence in DSS Outputs Most users across the four DSSs were satisfied with a system’s predictions and outputs. With regard to systems that assisted with BMP prioritization, many conservation field staff noted that this kind of information still needed to be verified through ground truthing. Seven end-users conducted informal ground truthing on their own, either looking at aerial imagery and comparing what they saw with the system’s recommendations or actually verifying it in the field. While users found some false positives and false negatives, there were multiple instances where a given DSS was accurate, or accurate enough. End-users found the outputs to be “pretty good” and felt 33 “comfortable” with the outputs. In some cases, the DSS outputs were perceived as superior to past methods, as one user explains: I'm confident. Yeah, the science behind it is very robust…In the past, what we had done is kind of the expert judgment factor based on our field and desktop GIS assessments of the watershed, these are the areas where we have really good reason to believe that would be priority locations for this practice or this set of practices, for example. I won't say the previous approach was subjective, but [DSS] is even more objective. For another DSS, reactions from farmers helped field-level staff feel more confident in the system’s outputs, as described by this participant: I can tell you anecdotally, we've never had a farmer dispute the results that we've presented. Not necessarily like the precise numerical quantifications, those are obviously debatable. But in terms of like relative priority, you know, we rank a farmer's 10 fields and there's never dispute that, 'Yeah, that one that you say is the worst is the worst.' Or we'll identify an area that's a hotspot and they'll say, 'Yeah, actually that's my ridge of blow sand that I always have trouble with.' So in terms of just a relative identification prioritization, 100% confident with it. And over the last six years using it with farmers, they've really reinforced that confidence. The nature of the decisions a DSS is intended to support may influence how accurate its outputs need to be to satisfy end-users. The consequences of a false positive or negative when engaging in conservation prioritization may not be as significant as those associated with manure runoff risk predictions. For example, one manager summarizes the consequence of a false positive: But we'd rather have a couple of misses because false positives tend to drive people away. When a farmer looks at the map and says, 'It told me not to spread and it didn't rain.' And then they're sitting with a pit full of manure that they could have unloaded. Applicable to all of the DSSs, is the fact that they are built on models which cannot know and perfectly reflect the management of a given farm. This sentiment that was often brought up when talking about system outputs with farmers, as illustrated here: You know, you have to get people to understand, especially the farmers. 'OK. This is saying that your field is high risk. That doesn't mean it actually is. That's the model's assessment of your field. The way you're managing that field may not be that.' 34 The underlying data also may lessen a user’s confidence in the DSS. For example, one user felt a dataset no longer reflected the condition of a watershed with widespread BMP adoption: And I've looked at them and said, well, you know, actually that's not what happens in this watershed anymore. So, it's basing what it's looking at off of something that I know is not correct with what's happening on the farms…I'm not being critical of the [DSS], but that just makes me start questioning a little bit…maybe it's starting to site things in places that don't need to be there just because those farms are already in a different condition in my mind. 3.5. Broader Outcomes In theory, DSSs are developed to support decisions with the hope that those decisions improve some aspect of coupled human and natural systems. To that end, I explored the ways in which the development and use of these systems have led to outcomes beyond mere adoption of the DSS. The DSSs are improving conservation delivery and watershed planning, which in turn will lead to positive social and environmental outcomes. The systems also enhance collaboration and learning. Conservation Delivery Innovation in Conservation Program Design Two of the DSS are enabling more effective ways to plan and deliver conservation. Program managers and end-users described how these types of tools are helping them to move toward performance-based conservation, applying conservation treatments on the land that needs it the most. One user shares their perspective: I do a lot of work in what…we're terming now performance-based conservation. So a lot of the tools I'm using are quantification tools…I've always termed [DSS] as a prioritization tool. We need to prioritize effort. We've spent so much time shotgunning 35 conservation onto the land and it really, to be completely honest, hasn't gotten us very far. We've got random practices being placed in places that they have no effect, really. And that's not me knocking the people who've done it. It's just the way we've always done things…And I'm like, let's start looking at results-based conservation. Find what's going to give us the best result and then move in that direction. For another DSS, it also enables organizations to report the environmental benefits of implemented practices, as described by this program manager: It's allowed us to speak about our programs in a different language than NRCS or conservation districts talking about the success of their programs. So even in 2020, you'll have very high level NRCS folks talk about the success of whatever program based on dollars spent, number of contracts, and acres enrolled is about as refined as is you can get from them…And we're speaking in, ‘These programs have allowed us to reduce 3,000 tons of sediment and 17,000 pounds of phosphorus.’ And that, just the value of that is beyond belief…It's advanced our work. Because we've been able to appeal to other funders that way…It's not that it's actually allowing us to do more conservation. I mean, pound for pound, NRCS is still doing so much more volume wise to reduce erosion and address resource concerns and all these great benefits. They're just not quantifying. They're not capturing it. They're not talking about it. Watershed Planning End-users of one DSSs described how it helped improve watershed planning by helping organizations develop more robust plans and assisting groups that lack capacity to develop formal watershed plans. A state agency coordinator also believes this DSS improves watershed planning, but notes the importance of field inventories: We got frustrated with some of the lack of specificity in our plans. They were very generalized plans “anywhere USA” – the usual suspects, resources and causes…You got a million dollars. Do you know where you're going to want to go work? And that really stuck with me because if you looked at some of our plans, we could not answer that question…This is augmenting the information we were gathering. It's not just the [DSS]. It's the [DSS] in conjunction with the field information we're collecting that, in our opinion, makes it so powerful. 36 Farmer Engagement Users of three of the DSSs also expressed that the systems assisted with farmer engagement around conservation practices, often referring to them as “visual aids” that supplement their conversation. One conservation technician described it like this: If I was working with a producer that was interested in putting some cover crops on, but they didn't want to put them on all their acres, this would be definitely helpful to pinpoint, 'Hey, you'll get a lot of reduction and the most out of your cover crops if we try them in all these areas here versus on some of the flat ground that might not show any reduction.' It helps the producers see exactly, 'Oh, I am losing X amount of tons of sediment here. Let's try to put something on it and keep it in place.' Environmental, Economic and Policy Outcomes With the ability to quantify the environmental benefits of conservation practices by using the DSS or coupling outputs with other models, users noted their ability to connect their program results with real-world impacts. This, in turn, is useful in assessing progress toward state nutrient reduction strategies. One manager describes this here: It's helpful to frame the nutrient reduction conversation in the state because we have this nutrient reduction strategy that says, ‘We're going to reduce nitrogen and phosphorus by 45%. We're going to need 15,000 bioreactors.' OK, well, can we even put in 15,000 bioreactors? Where would they even go? How would we know where to begin with those things?...So you can look at your watershed and say, well, this watershed can contribute, you know, 47 bioreactors to that. Another user indicated that this kind of information was useful in approaching state leadership about the resources needed to achieve nutrient reduction goals: And I would go to meetings with the state legislature or the Department of Ag, just as an example, that type of high-level leadership and say, 'OK, you say you want to solve this problem. You're putting enough resources in here to install, you know, X number of practices per year. But that's only two percent of what the conservation assessment says the need is.' 37 Enhancing Collaboration Multiple developers and managers felt the process of developing a given DSS enhanced their ability to collaborate. Where government agencies are concerned, participants noted that the development process overcame stereotypes that government agencies don’t talk to each other or that they’re unwilling to work together. One user shared that, “It made me much less of a cynic about how this stuff works. Everybody wanted the product to succeed and everybody did what they could to contribute.” Managers expressed that working on the systems brought together partners that traditionally wouldn’t work together, as described by this participant: We're getting people that normally may not have talked to each other, to talk to each other. We're getting modelers talking with people doing edge of field monitoring with agriculture extension agents, which they never really would have ever had any kind of overlap in connection with. And then state agricultural people…I've gotten a lot of people kind of involved in communicating on things they probably never would have without this project. In bringing together diverse stakeholders related to agriculture, conservation and water resources, one user feels the DSS is helping “bridge” divides between sectors that are commonly at odds: The tool helps get at the environmental concerns that a lot of people have and regulatory agencies have. But it also gets at the idea of doing the best with the resources you have from an agricultural standpoint. Having a tool that gets at both of those things, that's hard to do. Because a lot of times the environmental and the economic portions of agriculture do not always match. And so that's where we see a whole lot of disconnect. And so I think having a tool like this that helps kind of bridge that gap and gives a shared consensus…It reaches a lot of end goals that people are looking for in different ways, which is really nice because it helps give a unified front. However, collaboration brings its own challenges. For one, it requires time to sort through the responsibilities of the various partners. One manager recalls their experience in coordinating partners to help sustain the DSS: It was a lot of let's not step on anybody's toes at the beginning. Nobody is trying to take anything from anybody else…Like, we can all live together and work together and like, 38 this is a community that supports [DSS]. It's not a territorial thing. And we were able to work through that fairly quickly I feel like. But it still takes some intentional, like realizing that everybody is working toward the same goal just in different pieces. A potential barrier to collaboration is competition. One end-user offered this astute observation: We need to start working together instead of individual development of models everywhere. You know, when I'm assessing these things, I'll be completely honest, each model has some very strong things in it and failures in other things. And that's what frustrates me. I'm like, ‘If this model and this model were actually together, they'd fix each other's problems.’ I know it's not quite that simple…But the bottom line is, is maybe the people who developed each of these models, if they talked to each other, but they're in competition because whoever comes up with model that gets used is the one that gets the money. Learning DSSs may also enhance a user’s understanding of conservation benefits. Field staff talked about how showing the outputs of these systems to farmers helped some of them better understand what was occurring on their field and/or validated that their actions were making a difference. The DSSs can also help users start thinking from a watershed perspective: I think being able to present that view of the natural resource to people, I think starts to get their thoughts changing a little bit. It helps them get out of their, ‘This is my field. These are the decisions I make for my field.’…So you can look not just at your field, but you can look at your neighbor's field. I like it because I am all about watershed literacy and getting people to understand how the watershed is connected and how those things work together. One developer wondered if system users had “internationalized the model’s recommendations,” though they lacked evidence to support it. They explain: And so along with just providing people a tool, it's also a means of educating them…It may be that a lot of our users have internalized the model’s recommendations by now. And there is a lot more that goes into the model than just whether it's going to rain tomorrow. You know, soil saturation and previous history of rain and to some extent the soil characteristics and whether snow is melting or not. If you do it for long enough and you pay attention, you kind of know when the model's going to turn red in your area. And so ideally, it would, as an educational tool, if all these people that are making decisions 39 about spreading internalize that and they learn it, that's great. That's more than good if they don't need the website anymore. 3.6. Sustainability The four DSSs require ongoing investment and maintenance to stay running and relevant. Investment is needed not just in terms of keeping the interface running, but also in outreach and promotion to ensure it gets into the hands of end-users. Staffing and funding are the primary threats to DSS sustainability. Given the evolution of all four systems over the last 15-20 years, they have had some degree of success thus far. In addition, developers and project managers have learned and are applying strategies to design for longevity. Staffing Adequate staffing is a challenge for all organizations involved in the development, deployment and use of a DSS. At the field level, conservation field staff may not have the time to generate outputs from the systems. One user describes how other job responsibilities and limited staff prevent them from using a DSS in more watersheds: One of the things that held us back here is staffing. You know, there is a considerable amount of setup time to run one of those for a HUC12…With the last year, with the prevented planting and the disaster program and all the emphasis on installing conservation practices in the [watershed] and then the program dollars that we've gotten without staff support to implement it. You know our people have just been covered up. And so we've really not had anybody that has been able to take it and make it their job to run the tool. There are also capacity challenges at the manager and/or developer level. The DSS is one part of their job, and in some cases, it’s a very small part. One project manager described their role as being “a very, very side gig”. In another instance, the developer is juggling modeling and 40 development, along with multi-state coordination, and serving as the manager to develop the next version of the tool, on top of the primary responsibilities for their job. Probably most concerning related to staffing is turnover. These systems are extremely vulnerable to the loss of key personnel that have the technical expertise or social influence to keep the systems going. Eleven participants mentioned some form of turnover in positions related to the management, maintenance or promotion of the systems. In some cases, turnover was severe. One system lost three IT project managers during the interface development, which caused major delays and impacted their ability to engage with stakeholders that could have provided valuable input during development: You know, people are excited when you say you're going to do something, but when it's two and a half years later and they still haven't seen anything. And you're still talking about it? I think I lost some people. One state is on its fourth program manager for a DSS. Not only has it changed hands multiple times, but for this individual, it was handed off to them on very short notice: This was something that was handled by a previous section chief and then handed off to a colleague of mine…And this was literally something that was handed to me like two days before [person] left. Like, ‘Oh, by the way, you're going to have to be the guy.’…And essentially what happened is when he left, he's like, ‘Okay, here's a project folder. You need to take this over and we update the grant every year.’ The systems are also vulnerable to losing key modelers and programmers that are essential to keeping the systems running. In some cases, the modeler(s) or programmer(s) that initially developed the system are nearing retirement. Users are aware of their impending retirement and recognize it as an issue. In one instance, a user helped initiate conversation about pending retirements, which sparked a collaborative effort to develop a business plan to keep the system going. For another participant, they’re concerned about a new programmer taking over: He'll retire within a few years and we’ll lose his corporate knowledge as well. I can only imagine what it's going to be like to hand that off to another programmer. I mean, that's 41 like buying a used tractor. You have no idea what somebody has done inside that thing until you open it up and take it apart. In addition to keeping technical staff, the DSS are vulnerable to losing a champion within an organization. One manager offered this reflection on her role: I would not say I'm the key person for [state]. But like if I was out of my position, I don't know if there's anybody at the [organization] who would champion this because I'm the one who knows about it. One user summarized these dilemmas succinctly, “So it's really this integrated machine that works well as long as all the cogs are there.” Funding It's difficult to quantify how much it costs to develop, maintain, improve and promote the systems. Some developers and project managers have agreements in place that are very specific as to the deliverable. For example, having an $8,000 grant to an external party to host and maintain the online interface. In many cases, however, the systems are built by modelers and researchers whose staff time spent working on the DSS is not well documented and lumped in as part of their overall duties. What is clear is that long-term funding for these systems is difficult to obtain. All four systems have been supported by grants and typically have evolved and been funded in phases. Two systems purely rely on soft money, often supported with projects that expand the DSS into a new area. While both have been able to retain key programmers and modelers, this represents another vulnerability for a DSS. One developer acknowledged how critical a partner was for the DSS: Without their support, it would've just probably languished on a web server. But really through the funding that they provided for its continued support, maintenance and outreach allowed us to get a larger foothold amongst conservationists in the region. 42 Another concern related to funding is having adequate support for outreach and education for a DSS. In some cases, funding focuses on the technical aspects of the system, supporting modeling, expansion into new areas, and interface updates, and outreach is a small component. Designing for Longevity Recognizing the challenges associated with keeping these systems running and up-to- date, developers and project managers offered some insights and lessons learned. The DSSs have been expanded and improved through a phased approach. Typically, a system had a pilot phase and then was applied in another location, allowing developers to identify and address any issues. One developer shared that they initially developed individual interfaces on a watershed by watershed basis, and later decided to move toward a one-platform approach. Another developer shared how they are reducing future maintenance by writing code that enables the DSS to be automatically updated anytime the underlying model is updated. While developers and managers are thinking about longevity of the systems, some are further along in taking steps to ensure it. Only one manager referenced creating business and development strategies to consider expansion and sustainability. Committed Stakeholders Three of the DSSs have very dedicated user and stakeholder bases. They find value in the DSS and want to see it continue well into the future. Some of these groups are better positioned than others to take action to address sustainability concerns, such as the participant that recognized the vulnerability of having the main developers retire with no plan in place to ensure 43 the system’s continuation. Others are thinking about how to institutionalize a DSS but are looking for recommendations on how to accomplish this. One manager explains: And something that is always top of mind for us is how do we institutionalize this? And that's the real threat. You know, even having this out there in people's faces for the last six, seven years, it's still not readily used by other people other than [organization] and the people that we contract with them to do it. How do we institutionalize this thing and how do we get other groups utilizing it the way that we are?...And it stinks because we've gotten a ton of interest from other places…We've shown them the system and how we use it…They just never pulled the trigger on it. So, again, it's a deeper conversation…Are there other entities out there that are going to pick this up and use it?...This is going to sound trite, but if we're really going to change the world like there's got to be an exit strategy here. They've developed an incredible tool, so how do we take it mainstream? How do we hand this sucker off to an entity that can run with it? 44 4. DISCUSSION The findings showcase the multifaceted nature of decision support. While each of the four DSSs utilize different technologies and vary in functionality, the outputs for all systems are supporting some type of decision for their respective end-users. The findings illustrate that these are not static systems. A DSS does not have an end point, it is something that is updated, refined, and evolves. The systems represent years of research and work, and the developers are dedicated to improving them. As one developer puts it, “We would hope that it would be evolving, you know, 20 years from now.” The systems exist within a dynamic and complex web of stakeholders that support, promote and utilize the systems. 4.1 Outcomes The four DSSs highlighted in this study demonstrate the potential to achieve outcomes beyond adoption of a DSS. These systems can change the way conservation is delivered and assessed, and in two cases, the DSSs are actively being used to do just that by moving toward performance-based conservation. Applying conservation treatments on the land that is most susceptible to contributing nonpoint source pollutants to nearby waterways not only will yield the greatest environmental benefit but will result in the most cost-effective use of resources. However, the ability to directly connect DSS use to environmental benefits relies on a series of assumptions. As a general example, consider the following steps that must occur to achieve an environmental benefit from one of the BMP prioritization tools: a user generates outputs from a DSS. They then must integrate that information into their mental model and choose to use it when deciding who they’ll approach about a conservation program. While the landscape conditions make a particular location a higher priority than other areas, there needs to 45 be a willing landowner that will actually implement the recommended practice(s). The environmental benefit will only be realized once the practice is implemented and maintained. There are multiple places where that chain of events can break down. The user simply might not have the time to even use the system and continues with business as usual for conservation delivery. Perhaps the only willing landowners they’re able to engage have fields that are low risk for agricultural runoff, and little water quality benefits will be result from implementing a BMP. These hypotheticals don’t even address severe weather that could upend a farmer’s ability to implement a practice, or staffing and funding constraints that decimate a local office’s capacity to provide conservation assistance. Evaluating environmental benefits in this way is still difficult almost a decade after Matthews et al. (2011) called attention to “raising the bar” for EMS outcomes. However, a positive step in this direction is the ability to model pollutant reductions from implemented BMPs, assuming the models are accurate, and the conservation practice is implemented and sustained. The case studies also speak to the broader benefits of going through the process of developing, deploying, promoting, improving, evaluating and managing the various aspects that sustain a DSS. Multiple participants described the ways in which a DSS brought together groups of people and organizations that normally have no reason to collaborate. For the developers and program managers that talked about this, some of them were surprised that they were able to successfully work together and get something done. For them, that was the biggest success. 4.2 Challenges The challenges characterized in past DSS research remain as applicable today as they did 10, 20 and 30 years ago. McIntosh et al. (2011) delved into challenges and best practices for 46 environmental DSSs including engagement, adoption, evaluation, and business and technology issues. Note that the key challenges they identified had less to do with the science associated with modeling, validation, uncertainty, etc., and everything to do with the actual management, maintenance, promotion, and adoption of the systems. This study confirms that these challenges are not only persistent but can overshadow concerns related to the technical underpinnings of a DSS. When the consequences of an inaccurate output were low, end-users were often forgiving; the DSS were correct enough times, that the users still found value and comfort in using them. The key struggles from this study focus on how to get DSSs into the hands of users that have decision-making authority and keep them running. Adoption The study reveals that adoption is not a black and white concept. What does it mean to be a user and “adopt” a DSS? Is it the number of times they use the system? Is it that they use the system more than 50% of the time when making a certain decision? Are individuals that look at DSS outputs but don’t act on them considered “users”? All of these questions bring to light the need to make explicit the underlying assumptions researchers have about the DSSs they develop and how users are interacting with them. The findings also reinforce the need for standard metrics and approaches for assessing the nature and extent of adoption. Number of users is one indicator, but it doesn’t necessarily confirm that people are acting on system outputs. Further, analytics data need to be applied thoughtfully based on the nature of the DSS. Intermittent use doesn’t necessarily mean that users have abandoned a DSS; rather, it could be that the system supports their decision processes during certain times of the year. 47 The best practices that McIntosh et al. (2011) suggest for overcoming adoption challenges are good recommendations. But do they provide developers with enough specificity to be successful in following them? For example, it is very easy to recommend finding a champion to promote a DSS, but in practice can be incredibly difficult, as one of the participants in this study experienced. There are varying levels of commitment for key users and managers that could be deemed champions, from “side gig” managers to those that talk about the DSS in every presentation they give. Another best practice recommends “creating a plan for continuity…including planning for the transition from the development team to stakeholders and clients for adoption”. This certainly is good practice, but I would add that the stakeholders need their own continuity plan and documentation processes so that they, too, are successful when handing it off to a predecessor. This is especially important when turnover appears to happen regularly regardless of organization type or position. Lastly, outreach and education need to be at the forefront for a DSS. The type of outreach and user support depends on the intended end-users, ease of use, and the technical expertise required to run the system. Financial investment in promotional videos and ads may be most useful when trying to reach farmer and agribusiness audiences. Promoting conservation planning tools to field staff are likely best reached through professional networks and conferences, where social capital is more important. No matter the approach, funders must invest in and encourage grantees to build outreach and education into their DSS projects. 48 Sustainability While sustainability presents a challenge for the DSS in this study, the fact that the systems are still technologically sound, have been updated and are still used is a positive sign. However, the vulnerability of losing key personnel, either through retirement and turnover or lack of funding, could cripple some of the tools. To be sustainable, the systems really need the institutional support of a large organization or a government agency. The findings also reveal that DSSs enhance collaboration. In the context of sustainability, collaboration is essential. It takes a community to improve, maintain, and promote a DSS. The best practices from McIntosh et al. (2011) and Merritt et al. (2017) encompass numerous recommendations and strategies for interacting with stakeholders, including the need to clearly define stakeholder and end-user roles. However, there haven’t been detailed examples of what that looks like in practice. Table 3 contains a non-exhaustive list of 12 roles related to development, management, support and use of a DSS. Identifying roles to that level of detail can assist developers and stakeholders over the DSS lifecycle. For the case studies, the various roles are assumed by all types of individuals and organizations, many of which derive value and benefits from the system. The literature emphasizes the important role in engaging end-users throughout development, and I’d add to that to some degree, users need to take some shared level of responsibility to support the DSS. This study provides evidence that this is happening for some DSSs. Key stakeholders recognize that the tools require support and investment. In some cases, they are going out and finding funding to expand a system to new areas or are the ones leading the charge to develop business plans and find ways to institutionalize the system. To have that level of dedication suggests that the developers really have created a useful and powerful tool. 49 This study provides further evidence that DSSs can serve as boundary objects, bringing together diverse actors that utilize the system in tailored ways to meet their needs. Per Jakku and Thorburn (2010), DSS development crosses boundaries and shapes stakeholder experiences and understanding around the issue the DSS is trying to address. This is a powerful outcome that should not be ignored. In addition, the ability for DSS to be interpreted in multiple ways by the various stakeholders and actors that develop, manage, promote, and use these types of tools can be an advantage, particularly if it facilitates the sharing of resources to sustain the DSS. However, the interpretative flexibility of DSSs could have the opposite effect by diminishing ownership and responsibility for the system, where stakeholders assume that “it runs free” or only consider their specific use of the DSS instead of the collective benefits that it provides. This, in turn, could influence what updates are invested in that may enhance the use for some user groups but diminish it for others. 50 5. CONCLUSION DSS evaluation is seldom researched and has been heavily skewed to the experiences of developers. While developer perspectives are useful, they cannot fully illuminate user satisfaction, adoption, and engagement. By including both developer and end-user perspectives, this study provides a more complete account of the nature and impact of four DSSs. While the four systems focus on conservation delivery and implementation decisions to address water quality impairments, the findings are widely applicable to environmental DSS and EMS products. Adoption and sustainability challenges described in past studies were also documented in a variety of ways for the four DSS included in this study. DSSs are not static systems, but constantly evolving in response to research development, user needs, and expansion opportunities. These processes are rarely documented in detail within the literature but could provide valuable insights into what makes a given system successful. More case studies are needed, including successful and unsuccessful cases. Having greater understanding of why a DSS is no longer operational could further tease out best practices for developers and illuminate how factors such as system design, stakeholder engagement and extenuating circumstances hindered or helped a “failed” system. Future research involving DSS evaluation should consider three aspects: (1) documenting unsuccessful DSS cases, (2) collecting end-user experiences related to DSS use, and (3) defining metrics and approaches for assessing adoption of a DSS. 51 APPENDICES 52 APPENDIX A Agricultural DSS in the North Central Region 53 Table 4. Agricultural Decision Support Systems Available in North Central Region DSS Name Water resource issue Location Scale Audience Source Agricultural Conservation BMP siting; Planning Framework Watershed planning Multistate HUC12 Conservation Staff, Government, NGOs Researchers Tomer et al. (2015) Gelder et al. (2018) Nelson et al. (2014) State Climate Office OH (2020) Daily Erosion Project Erosion Vulnerability Estimate soil loss from water erosion Multistate HUC12 Government, Farmers, NGOs, Researchers Assessment for BMP siting for soil erosion WI Agricultural Lands Field-scale; HUC12 Watershed Managers Field Application Resource Monitor Reduce runoff risk of nutrient applications OH Field-scale Farmers, Nutrient applicators Great Lakes Watershed Quantify environmental Management System benefits of BMPs Multistate Field-scale; watershed-scale Conservation Staff, Fales et al. Farmers, NGOs (2016) Nutrient Tracking Tool Runoff Risk Advisory Forecast Prioritize, Target, and Measure Application SnapPlus Useful2Useable Quantify nutrient and sediment losses from agricultural land Reduce runoff risk of nutrient applications Prioritize and identify BMP locations, and evaluate water quality benefits Nutrient Management Planning Decision support to enhance agricultural resilience in face of climate change Multistate Field-scale; Farmers, Conservation Staff, Saleh et al. watershed-scale Government (2011) Multistate 6 mi2 grid Farmers, Manure Haulers, Nutrient Applicators, Extension Goering (2013) MN Field-scale; Conservation Staff, watershed-scale Government, Planning Staff Srinivas et al. (2020) WI Field-scale Multiscale Farmers, Nutrient Applicators Good et al. (2012) Multistate depending on Farmers, Agricultural Advisors DST 54 Prokopy et al. (2017) APPENDIX B Participant Consent Form 55 Research Information and Consent Form – Water Resources Decision Support Systems 1.  Purpose of Research                                                                         The purpose of this research study is to better understand how DSS have been deployed to address agricultural water quality and quantity issues within the North Central Region of the United States. This will help inform the development of more effective decision support systems by university researchers and conservation agencies. 2. Your Participation is Voluntary                                                                         Your participation in this study will take about 30-45 minutes. During an in-depth interview, you will be asked a series of questions about your thoughts and experiences as they relate to the [DSS]. Your participation in this project is voluntary. You have the right to say no. You may change your mind at any time and withdraw. You may choose not to answer specific questions or to stop participating at any time.   3. Potential Benefits and Risks You may not benefit personally from being in this study. However, we hope that in the future, others may benefit from having access to improved decision support tools. There are no foreseeable risks to participating in this study.     4.  Privacy and Confidentiality                                                              The interview will be recorded. During the research process, the recordings will be used for data analysis purposes, and will only be accessible by the researchers. When this study is complete, the recordings will be destroyed. The transcripts will be retained indefinitely by the researchers. Any personal information that could identify you will be removed or changed before files are shared with other researchers or results are made public. Confidentiality cannot be guaranteed. It may be possible, though unlikely, to indirectly identify you by connecting information from your interview to the individuals in your community/organization, even if your name is removed. However, we will make every possible effort to maintain your confidentiality should you choose to exercise that option, by assigning you a pseudonym.   5.  Contact Information                                                                          If you have concerns or questions about this study, such as scientific issues, how to do any part of it, or to report an injury, please contact the lead researcher:   Steven Gray grayste1@msu.edu   If you have questions or concerns about your role and rights as a research participant, would like to obtain information or offer input, or would like to register a complaint about this study, you may contact, anonymously if you wish, the Michigan State University’s Human Research Protection Program at 517-355-2180, Fax 517-432-4503, or e-mail irb@msu.edu or regular mail at 4000 Collins Rd, Suite 136, Lansing, MI 48910.   6.  Documentation of Informed Consent You will provide your verbal consent to participate at the start of your interview. 56 APPENDIX C Interview Guide – Developers 57 INTERVIEW GUIDE - DEVELOPERS Interview Participant: Date: Interview Start Time: Interview End Time: Introduction and Informed Consent Script Thank you for agreeing to participate in my study, which is supporting my training as I complete my thesis. I’d like to briefly review the information in the informed consent statement that I attached with your calendar appointment before we begin. To recap, the purpose of this study is to explore how different decision support systems and models are used throughout the North Central region to improve decision-making related to agriculture and water resources. I am interviewing both developers and end-users about their experiences with different decision support systems. The interview will take approximately 30-45 minutes, and you will be asked about your thoughts and experiences related to Name of Decision Support System. Your participation is voluntary. You may choose not to answer specific questions or stop participating at any time. There are no known risks to participation. While you may not directly benefit by participating in this project, I hope to identify gaps and opportunities to improve these kind of tools in the future. I will publish the results in my thesis and in a final report to the Institute of Water Research at Michigan State University, which is supporting my project through a U.S. Geological Survey grant. Any personal information that could identify you will be removed or changed before results are made public. However, confidentiality cannot be guaranteed. It may be possible to indirectly identify you by connecting information from your interview to the individuals in your organization, even if your name is removed. For example, if I name Decision Support System in my thesis and indicate that I interviewed one of the developers or project managers associated with the system, I cannot guarantee that you will remain anonymous. I will make every possible effort to maintain your confidentiality and will assign you a pseudonym in my project. Are there any concerns about maintaining your anonymity and confidentiality? The interview will be recorded with your permission. I’d like to ask for your verbal consent to record. [Obtain consent]. Do you have any questions for me before we begin the interview? [Answer any questions] Ok, with that, I will start recording and we’ll begin. 1. What is your specific role with [DSS]? 2. Can you talk about how the system got started and how it has evolved? Additional prompt: Who initiated the project? Were end-users engaged? 3. Tell me about the [DSS]. Describe the purpose of the system. 58 4. Tell me about how the system works. What kind of data inputs are fed into the system? What are the outputs? What kind of technologies and features are used in the system? 5. Tell me about the limitations of the system and model predictions. How confident can end-users be in the predictions made by [DSS]? 6. Who are the intended end-users? 7. What decisions are intended to be supported by [DSS]? 8. Describe a typical instance where an end-user would use the system. 9. Has [DSS] supported any specific programs or projects? Can you talk about those? 10. Can you talk about the funding for the system? If you are willing to share, approximately how much did it cost to develop? How are you supporting ongoing maintenance? 11. How is the system promoted to end-users? Are end-users trained in using the tool? Can you describe more about that? 12. Has the system been evaluated at all by end-users? Do you collect feedback from end- users as they use the system? 13. What do you see as the biggest impacts from developing, deploying and expanding the system? Alternative prompt: Put another way, what would you describe as the big successes from developing and launching [DSS]? 14. Are there any challenges you’d like to share about managing [DSS]? 15. Are there other tools like [DSS] that you know are used to address similar water resource and agricultural issues? 16. Do you have any final comments about [DSS] that would be helpful for my research? 17. Do you know of other end-users that would be willing to be interviewed about their experience with [DSS]? 18. Thank for participating and note intention to share final report. 59 APPENDIX D Interview Guide – End-Users 60 INTERVIEW GUIDE – END-USERS Interview Participant: Date: Interview Start Time: Interview End Time: Introduction and Informed Consent Script Thank you for agreeing to participate in my study, which is supporting my training as I complete my thesis. I’d like to briefly review the information in the informed consent statement that I attached with your calendar appointment before we begin. To recap, the purpose of this study is to explore how different decision support systems and models are used throughout the North Central region to improve decision-making related to agriculture and water resources. I am interviewing both developers and end-users about their experiences with different decision support systems. The interview will take approximately 30-45 minutes, and you will be asked about your thoughts and experiences related to Name of Decision Support System. Your participation is voluntary. You may choose not to answer specific questions or stop participating at any time. There are no known risks to participation. While you may not directly benefit by participating in this project, I hope to identify gaps and opportunities to improve these kind of tools in the future. I will publish the results in my thesis and in a final report to the Institute of Water Research at Michigan State University, which is supporting my project through a U.S. Geological Survey grant. Any personal information that could identify you will be removed or changed before results are made public. However, confidentiality cannot be guaranteed. It may be possible to indirectly identify you by connecting information from your interview to the individuals in your organization, even if your name is removed. For example, if I name Decision Support System in my thesis and indicate that I interviewed one of the developers or project managers associated with the system, I cannot guarantee that you will remain anonymous. I will make every possible effort to maintain your confidentiality and will assign you a pseudonym in my project. Are there any concerns about maintaining your anonymity and confidentiality? The interview will be recorded with your permission. I’d like to ask for your verbal consent to record. [Obtain consent]. Do you have any questions for me before we begin the interview? [Answer any questions] Ok, with that, I will start recording and we’ll begin. 1. How did you find out about and get involved with [DSS]? 2. Can you describe the purpose of [DSS]? 3. Can you describe a typical instance where you would use the system? 4. How often do you use [DSS]? 61 5. Does [DSS] help you make decisions in your job? If so, in what way? Alternative prompt: How do you use the information from the system? 6. Tell me about how the system works. What kind of information do you need to run the system? What kind of information do you get back? 7. How confident are you in the predictions made by the system? 8. Did you receive training to use the system? Can you tell me more about that? 9. In using [DSS], are there features that particularly useful? Any that aren’t very useful? 10. Have you provided feedback to the development or management team about [DSS]? 11. Would you recommend [DSS] to others? Tell me more about that. 12. To what extent do you know other people that are using the system? 13. Does using [DSS] help you in your conservation work? Can you tell me more about that? 14. In instances where you used [DSS] with farmers, what were their reactions? 15. When conservation professionals and farmers use tools like [DSS] to aid decisions, do you think it can result in better farm outcomes? 16. Are there any challenges you’d like to share about using [DSS]? 17. Are there other tools like [DSS] that you use to support your conservation work? 18. Do you have any final comments about [DSS] that would be helpful for my research? 19. Do you know of other end-users that would be willing to be interviewed about their experience with [DSS]? 20. Thank for participating and note intention to share final report. 62 APPENDIX E Recruitment Email 63 Subject: Name of Decision Support System - Interview Request for Thesis Research Greetings Name of Participant, My name is Laura Young, and I am a graduate student at Michigan State University. I would like to invite you to participate in an interview (~30-45 minutes) about the Name of Decision Support System, as part of my thesis research project. The purpose of this research is to better understand how tools like Name of Decision Support System have been used to address agricultural water resource issues within the North Central Region of the United States. This will help inform the development of more effective online tools and decision support systems by university researchers and conservation agencies. If you would like to participate, are you available suggested dates and times? Participation is voluntary, and you can withdraw any time if you change your mind. There are no known risks to participation. Please find attached an informed consent document that I will review with you should you wish to participate. If you have any questions, please reach me at youngla9@msu.edu or the lead researcher, Steven Gray, at grayste1@msu.edu. Thank you, Laura Laura Young Graduate Student Department of Community Sustainability Michigan State University 64 APPENDIX F Participant Thank You Letter 65 Participant Thank You Letter Subject: Name of Decision Support System Research Follow-Up and Final Report Greetings Name of Participant, Thank you for participating in an interview about Name of Decision Support System, as part of my thesis research project. Your participation provided valuable insights regarding the challenges and benefits of agricultural decision support systems. Please find attached the final report that I have prepared for this study. Please reach out with any questions or comments you have about the study. Thank you, Laura Laura Young Graduate Student Department of Community Sustainability Michigan State University 66 APPENDIX G Codebook 67 Table 5. Codebook Name Description Rule Example Challenges Challenges or issues encountered by developers or end-users as the DSS is developed, deployed, used, maintained, improved or evaluated. Code any response for question explicitly asking about challenges; code additional responses that allude to difficulties in developing, deploying, using, maintaining, or improving a case study DSS Bugs Technical issues, bugs with the DSS Anytime participant brings up bugs with a case study DSS Data Challenges associated with data processing for the DSS Anytime participant describes issues with input data for case study DSS Turnover Specific instances of turnover occurring Apply for development team, intermediary users and end- users You know, people are excited when you say you're going to do something, but when it's two and a half years later and they still haven't seen anything. And you're still talking about it? I think I lost some people. Well, I mean, it's extremely buggy…I mean, a lot of the issues have been resolved, but it's really buggy. It causes headaches for our conservation districts that we contract with to be the boots on the ground and do the programs for us. You know, having good LIDAR information is critical. And having that available in a form that is usable when they put things together. So I think that slowed our process down a while because he had so much clean up to do with that. I think well, we lost, I think, three different project managers. So then that, you know, we'd start making progress. And then that person quit. And so then we'd have to get somebody else back up to speed. So that was another thing that took longer. 68 Table 5 (cont’d) Name Description Rule Example Resources Challenges related to having resources (time, money, staff) to be able to use or support the DSS. Anytime participant discusses resource challenges related to case study DSS But, yeah, there are there are no recurring funds from the university or elsewhere to for its long- term maintenance. So we're kind of patching it together. DSS Sustainability Strategies or challenges related to the long-term viability of the DSS Participant describes strategies or challenges that impact DSS sustainability And something that is always top of mind for us is how do we institutionalize this? And that's the real threat. And that's the real thing that we worry about here. You know, even having this out there in people's faces for the last, you know, six, seven years, it's still not readily used by others and the people that we contract with them to do it. How do we institutionalize this thing and how do we get other groups utilizing it the way that we are? Funding Financial support for the development, deployment, maintenance, improvement or evaluation of a DSS Anytime funding is discussed for case study DSS. Can be formal grants, interagency agreements, in- kind funding And it was really lucky because at that time we actually did have a funding source, some [funding source] dollars that were available that I could get a bid for. So we crafted our proposal and then we were awarded like $82,000. Ownership Nature of perceived ownership of a DSS Apply when participant describes or implies varying levels of ownership of case study DSS 69 It was a lot of let's not step on anybody's toes at the beginning. Nobody is trying to take anything from anybody else…Like, we can all live together and work together and like, this is a community that supports [DSS]. It's not a territorial thing. And we were able to work through that fairly quickly I feel like. But it still takes some intentional, like realizing that everybody is working toward the same goal just in different pieces. Table 5 (cont’d) Name Description Rule Example Audience Intended or actual end- users of the DSS Apply when possible, actual or intended audiences are mentioned for case study DSS Features Specific features or functions of a DSS Anytime participant mentions specific feature of a case study DSS Purpose The purpose of the DSS from the perspective of developers and end-users Code in response to question directly asking about the purpose of the DSS Conservation planners primarily, you know. As we look at this, though, we realize that the number of planning agencies who need broader scale information on planning...conservation opportunities out on their landscapes. So we think that the state and federal agencies have the opportunity to do this. But the interactive ability of the tools is a little different. Some of them have like a pop up box with, like, graphical. Some of them have more of a table format. And it just depends on what platform it was created on in each state. [State] is, though, clear, leader, in trying to get like email and text messaging where other states really haven't even tried to do that at all. Personally, I think that's the future is to kind of get that, get to a way to push data instead of relying on people to come to a website. It is a tool for predicting pollutant loads on specific pieces of land. 70 Table 5 (cont’d) Name Description Rule Example And then again, when it gets down to where we actually have a farmer or a land owner on the hook wanting to do stuff, we do batch evaluations of their whole farm. And then that gets boiled down to just a subset of fields because [DSS] again, simulates where kind of hotspots are, those areas that are going to contribute more. And then we contract those individual fields. So every step of the way, it helps us to kind of hone in and really refine where you work. And so what we want users to do is incorporate this tool in their day to day decision making process. And if we show a high risk, we want them to hold off on applying until that risk is passed or if they have to apply, to apply to safer fields. So with the whole goal of not having recently applied nutrients immediately lost. Using it for prioritizing the landowners that we needed to approach was extremely helpful, not only in getting them invested, but just kind of giving us a general idea of where we might focus future conservation work as well. BMP Prioritization Use of DSS to prioritize BMPs on the landscape Participant describes use of DSS for BMP prioritization Nutrient Application Use of DSS to support nutrient application decisions Participant describes use of DSS for nutrient application Outreach Prioritization Use of DSS to prioritize conservation outreach Participant describes use of DSS for outreach prioritization 71 Table 5 (cont’d) Name Description Rule Example Research Use of DSS to support research Participant describes use of DSS for research Watershed Planning Use of DSS to support watershed planning Participant describes use of DSS for watershed planning Other DSS DSS, DST, or models that are referenced by the participant and are not a case study system Participant brings up other EMS product Other Outcomes Possible outcomes as a result of the development, deployment, maintenance, improvement, use and evaluation of a DSS Other outcomes that do not fit other outcomes categories We think it's a good research platform as well. And we're actually doing some work now to help demonstrate that we do have a special section. You know, you mentioned papers. We've got seven papers being published in [Journal]. Yeah. The state agencies want to see it there. They haven't written that into policy. But they want to they want to see that information to assess plans. Trust me, because [DSS], [DSS], all these other models I work with. Yeah. There are folks who literally get confused by them. And it's taken me a long time to understand some of the nuances. Every model has nuances that you have to understand how to work around or work through. And quite honestly, this is going to sound like...just uh, I don't know, hyperbole here, but it really is half the reason I work at [organization] is because of [DSS]. Because it offered something so different, so accessible, so user friendly that I hadn't seen working in the conservation district NRCS space before. 72 Table 5 (cont’d) Name Description Rule Example Adoption Extent and nature of adoption of DSS by those outside the development team Apply when referencing number of users or instances where it is routine practice to use case study DSS You know, just to be completely honest with you, there haven't been as many users as we would have hoped. Collaboration DSS serves as a collaboration tool among developers, partners, stakeholders and/or end- users. Participant describes instance of case study DSS facilitation collaboration among multiple entities Communication DSS use, results, or end products contribute to communication among partners of agriculture Participant describes instance where case study DSS facilitates communication between conservation professionals and farmers Economic Use of DSS leading to economic change Participant describes connection between case study DSS and economic outcomes 73 So there there's been this, I think, just ability for us to kind of talk about that. We had this project to focus on and then it has these that are products that we can use and have potential use for. And I think it's helped kind of foster some of those conversations. It's just a visual aid that brings it full circle to the producer so we can talk about sediment leaving their field. We can talk about how cover crops or no till is really going to help. But when they look at it, they can see the numbers. They can see their problem area showing up on the program as far as where they see them in the field. I think it just helps kind of complete the conversation. So with the whole goal of not having recently applied nutrients immediately lost. Across the region, it's kind of well established, every time you know, you have runoff, you're more likely going to lose some nutrients as a legacy nutrients. We just don't want to apply more nutrients right before a runoff event because it's economically and environmentally, it's a benefit. Table 5 (cont’d) Name Description Rule Example Enabling Existence and/or use of DSS enabling new projects or programs to happen Participant describes instance where case study DSS enabled another project to occur But basically, the [funder] wanted to see, 'Can we better engage farmers? What are some strategies for that?' And this this idea of tailored information seemed like a kind of neat idea. And the only reason I could do it is because of the [DSS] data. Yeah. Other things exist in. Like at the HUC8 level, but that's just not as specific as one might want. Environmental Use of DSS leading to environmental change Participant describes connection between case study DSS and environmental outcomes And we're speaking in, ‘These programs have allowed us to reduce 3,000 tons of sediment and 17,000 pounds of phosphorus.’ And that, just the value of that is beyond belief. Learning DSS development, deployment, maintenance, improvement processes contribute to learning either of developers, partners, and/or end-users related to water resources and agriculture Participant describes instances where case study DSS contributed to learning or is perceived to contribute to learning Yeah, I think that it definitely helped them kind of put into perspective why they're seeing different things on their land, so they might point to a certain problem that they have and they say, 'Oh, that's because, you know, there's a lot of sediment running off of that field. That's why I always have topsoil issues there," or something like that. 74 Table 5 (cont’d) Name Description Rule Example Policy Extent that a DSS supports or addresses policy initiatives Anytime participant mentions case study DSS in broader context of policy impacts Confidence Level of confidence in the data Groundtruth Verification of data inputs or outputs associated with DSS Code responses for question explicitly asking about confidence in DSS predictions Apply when participant specifically mentions process of ground truthing predictions. Does not have to use ground truth explicitly, but describe the process. So, you know, we're incorporating the information, you know, that's gonna be part of the nutrient reduction strategy. This is one of the things we're using to measure progress or change and everything. And so we're doing an update to that to our nutrient reduction strategy right now, a five year update. And we're including this work as part of one of our initiatives. So, for example, if I have two different fields and I run the same model on them both, I can say with reasonable certainty which field is going to be experiencing more erosion relative to the other. I'm not confident that those numbers, the sediment loading numbers, for example, are exactly precise. We're going to start actually vetting what it was telling us and whether or not, you know, like, OK, so 50 percent of the time it's doing it right. The other 50%, you know this is a random, not just you, you know, as an example. And I'm finding that it does a pretty decent job. But there's still something to be said for old school groundtruthing. 75 Table 5 (cont’d) Name Description Rule Example Privacy Statements made about perceived privacy concerns for users and/or stakeholders Participant mentions possible concerns over data privacy with case study DSS Ease of Use Perceptions about the user- friendliness and ease of use of a DSS Participant describes positive or negative thoughts about ease of use for a case study DSS Top Management Support Instances where leadership approves or supports of staff use of DSS Participant describes instances where top management support bought into case study DSS use for their organization Computer experience Nature of technical expertise required to use DSS Participant describes computer experience needed to run case study DSS With some people, like, some people don't. You can register on the tool. And so it actually kind of stores your previous forecast, but it's not a requirement. You go in anonymously and do it. There was some concern early on about people being targeted as far as maybe making manure application or fertilizer application, things like that. I know I've said this a million times, but it's easily accessible and user friendly. It's really easy to put it in their hands and say here is a tool to use to help you obtain these recommendations and to uphold these recommendations for nutrient management and manure management. And, you know, they had an interest because the GIS specialist, in particular, likes it. He's big on it. He thinks it should be done. So him and I kind of came together and, you know, we came up with this concept of I'll do the eastern side of the state. Him and his staff will try to do the western side of the state. So we provided some training to them. I think it's going to depend on the farmers'...I'm not going to say age, but receptiveness to technology...that can be associated with age. 76 Table 5 (cont’d) Name Description Rule Example Promotion Nature of how a DSS is promoted to potential end- users or new partners Participant describes how they found out about the DSS and/or how the system is promoted to others I talk about it pretty much at any presentation I can give. Training Extent and nature of training provided to end- users to use the tool Participant describes training offered or received for case study DSS User Engagement Communication and engagement among development team and end- users, partners and/or stakeholders Participant describes process of engaging end-users and/or interacting with developers Yeah, I mean, we've given training. We're giving in-person trainings on this. The minimum was two days. We have had full week sort of training sessions that involve watershed tours and looking at results and that sort of thing. And we talk about lessons learned. What have you guys done with your tool for outreach? What are problems with the tool? You know, in the early days is like, well, what do we want to take a look like, you know? And what I really want is the state representatives to communicate to developers what they really want out of the tool, because this is like the one and only time and get really feedback kind of incorporated. 77 REFERENCES 78 REFERENCES Bessette, D., Wilson, R., Beaudrie, C., & Schroeder, C. (2019). An online decision support tool to evaluate ecological weed management strategies. Weed Science, 67(4), 463-473. Bianchi, T.S., DiMarco, S.F., Cowan Jr, J.H., Hetland, R.D., Chapman, P., Day, J.W., & Allison, M.A. (2010). The science of hypoxia in the Northern Gulf of Mexico: a review. Science of the Total Environment, 408(7), 1471-1484. Car, N.J., Christen, E.W., Hornbuckle, J.W., & Moore, G.A. (2012). Using a mobile phone Short Messaging Service (SMS) for irrigation scheduling in Australia–Farmers’ participation and utility evaluation. Computers and electronics in agriculture, 84, 132-143. Díez, E., & McIntosh, B. S. (2009). A review of the factors which influence the use and usefulness of information systems. Environmental Modelling and Software, 24(5), 588–602. Easton, Z.M., Kleinman, P. J., Buda, A.R., Goering, D., Emberston, N., Reed, S., ... Sommerlot, A.R. (2017). Short-term forecasting tools for agricultural nutrient management. Journal of Environmental Quality, 46(6), 1257-1269. Fales, M., Dell, R., Herbert, M.E., Sowa, S.P., Asher, J., O'Neil, G. … Wickerham, B. (2016). Making the leap from science to implementation: Strategic agricultural conservation in Michigan's Saginaw Bay watershed. Journal of Great Lakes Research, 42(6), 1372-1385. Fassio, A., Giupponi, C., Hiederer, R., & Simota, C. (2005). A decision support tool for simulating the effects of alternative policies affecting water resources: an application at the European scale. Journal of Hydrology, 304(1-4), 462-476. Finlay, P.N., & Forghani, M. (1998). A classification of success factors for decision support systems. The Journal of Strategic Information Systems, 7(1), 53-70. Gelder, B., Sklenar, T., James, D., Herzmann, D., Cruse, R., Gesch, K., & Laflen, J. (2018). The Daily Erosion Project–daily estimates of water runoff, soil detachment, and erosion. Earth Surface Processes and Landforms, 43(5), 1105-1117. Gibson, F.L., Rogers, A.A., Smith, A.D.M., Roberts, A., Possingham, H., McCarthy, M., & Pannell, D.J. (2017). Factors influencing the use of decision support tools in the development and design of conservation policy. Environmental Science & Policy, 70, 1–8. Goering, D. C. (2013). Decision Support for Wisconsin's Manure Spreaders: Development of a Real-Time Runoff Risk Advisory Forecast [Master’s thesis, The University of Arizona]. UA Campus Repository. http://hdl.handle.net/10150/305874 Gold, D. (2020). Researching the Effectiveness of Agricultural Programs in GLRI Priority Watersheds: REAP Final Report. https://www.glc.org/wp-content/uploads/REAP-Final- Report-20200117-WITH-Appx.pdf 79 Good, L.W., Vadas, P., Panuska, J.C., Bonilla, C.A., & Jokela, W.E. (2012). Testing the Wisconsin Phosphorus Index with year‐round, field‐scale runoff monitoring. Journal of environmental quality, 41(6), 1730-1740. Gorsevski, P.V., Cathcart, S.C., Mirzaei, G., Jamali, M.M., Ye, X., & Gomezdelcampo, E. (2013). A group-based spatial decision support system for wind farm site selection in Northwest Ohio. Energy Policy, 55, 374–385. Inman, D., Blind, M., Ribarova, I., Krause, A., Roosenschoon, O., Kassahun, A., ... & Jeffrey, P. (2011). Perceived effectiveness of environmental decision support systems in participatory planning: Evidence from small groups of end-users. Environmental Modelling & Software, 26(3), 302-309. Irani, Z. (2002). Information systems evaluation: navigating through the problem domain. Information & Management, 40(1), 11-24. Jakku, E., & Thorburn, P. J. (2010). A conceptual framework for guiding the participatory development of agricultural decision support systems. Agricultural systems, 103(9), 675-682. Kochilakis, G., Poursanidis, D., Chrysoulakis, N., Varella, V., Kotroni, V., Eftychidis, G., ... Makropoulos, C. (2016). FLIRE DSS: A web tool for the management of floods and wildfires in urban and periurban areas. Open Geosciences, 8(1), 711-727. Lautenbach, S., Berlekamp, J., Graf, N., Seppelt, R., & Matthies, M. (2009). Scenario analysis and management options for sustainable river basin management: application of the Elbe DSS. Environmental Modelling & Software, 24(1), 26-43. Le Bars, M., & Le Grusse, P. (2008). Use of a decision support system and a simulation game to help collective decision-making in water management. Computers and electronics in agriculture, 62(2), 182-189. Matthews, K., Rivington, M., Blackstock, K., McCrum, G., Buchan, K., & Miller, D.G. (2011). Raising the bar?–The challenges of evaluating the outcomes of environmental modelling and software. Environmental Modelling & Software, 26(3), 247-257. McIntosh, B.S., Ascough, J.C., Twery, M., Chew, J., Elmahdi, A., Haase, D. … Voinov, A. (2011). Environmental decision support systems (EDSS) development – Challenges and best practices. Environmental Modelling & Software, 26(12), 1389–1402. Merritt, W.S., Fu, B., Ticehurst, J.L., El Sawah, S., Vigiak, O., Roberts, A.M., ... Jakeman, A.J. (2017). Realizing modelling outcomes: A synthesis of success factors and their use in a retrospective analysis of 15 Australian water resource projects. Environmental Modelling & Software, 94, 63-72. National Agricultural Statistical Service (NASS). (2020). Quick Stats [dataset]. https://www.nass.usda.gov/Quick_Stats/index.php 80 Nelson, T., Mazurek, D., Ruesch, A., Kempen, S., & Evans, D. (2014). Erosion Vulnerability Assessment for Agricultural Lands (EVAAL) Methods Documentation, Wisconsin DNR. http://dnr.wi.gov/topic/Nonpoint/documents/EVAAL/EVAAL_Methods_v1_0.pdf Newman, J.P., Maier, H.R., Riddell, G.A., Zecchin, A.C., Daniell, J.E., Schaefer, A.M., ... Newland, C.P. (2017). Review of literature on decision support systems for natural hazard risk reduction: Current status and future research directions. Environmental Modelling & Software, 96, 378-409. NRCS. (2019). Mississippi River Basin Healthy Watersheds 2019 Progress Report. https://www.nrcs.usda.gov/wps/PA_NRCSConsumption/download?cid=nrcseprd1617624&e xt=pdf Olugbode, M., Richards, R., & Biss, T. (2007). The role of information technology in achieving the organisation's strategic development goals: A case study. Information Systems, 32(5), 641-648. Le Page, M., Berjamy, B., Fakir, Y., Bourgin, F., Jarlan, L., Abourida, A., ... Simonneaux, V. (2012). An integrated DSS for groundwater management based on remote sensing. The case of a semi-arid aquifer in Morocco. Water resources management, 26(11), 3209-3230. Portoghese, I., D'Agostino, D., Giordano, R., Scardigno, A., Apollonio, C., & Vurro, M. (2013). An integrated modelling tool to evaluate the acceptability of irrigation constraint measures for groundwater protection. Environmental modelling & software, 46, 90-103. Prokopy, L.S., Carlton, J.S., Haigh, T., Lemos, M.C., Mase, A.S., & Widhalm, M. (2017). Useful to usable: Developing usable climate science for agriculture. Climate Risk Management, 15, 1-7. Rabalais, N.N., & Turner, R.E. (2019). Gulf of Mexico hypoxia: Past, present, and future. Limnology and Oceanography Bulletin, 28(4), 117-124. Reiter, D., Meyer, W., & Parrott, L. (2017). Why do NRM regional planning processes and tools have limited effect? Presenting the perspective of the end user. Climate Risk Management, 18, 66–74. Rizzoli, A.E. & Young, W.J. (1997). Delivering environmental decision support systems: software tools and techniques. Environmental Modeling and Software, 12, 237-249. Rossi, V., Salinari, F., Poni, S., Caffi, T., & Bettati, T. (2014). Addressing the implementation problem in agricultural decision support systems: the example of vite.net. Computers and Electronics in Agriculture, 100, 88–99. Saleh, A., Gallego, O., Osei, E., Lal, H., Gross, C., McKinney, S., & Cover, H. (2011). Nutrient Tracking Tool—a user-friendly tool for calculating nutrient reductions for water quality trading. Journal of Soil and Water Conservation, 66(6), 400-410. 81 Sandink, D., Simonovic, S. P., Schardong, A., & Srivastav, R. (2016). A decision support system for updating and incorporating climate change impacts into rainfall intensity-duration- frequency curves: Review of the stakeholder involvement process. Environmental Modelling and Software, 84, 193–209. Smith, D.R., King, K.W., & Williams, M.R. (2015). What is causing the harmful algal blooms in Lake Erie?. Journal of Soil and Water Conservation, 70(2), 27A-29A. Srinivas, R., Drewitz, M., & Magner, J. (2020). Evaluating watershed-based optimized decision support framework for conservation practice placement in Plum Creek Minnesota. Journal of Hydrology, 583, 124573. Star, S.L., & Griesemer, J.R. (1989). Institutional ecology, ‘translations’ and boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39. Social Studies of Science, 19(3), 387-420. State Climate Office of Ohio, The Ohio State University. (2020). Field Application Resource Monitor [Website]. https://farm.bpcrc.osu.edu/ Tayyebi, A., Meehan, T.D., Dischler, J., Radloff, G., Ferris, M., & Gratton, C. (2016). SmartScape™: A web-based decision support system for assessing the tradeoffs among multiple ecosystem services under crop-change scenarios. Computers and Electronics in Agriculture, 121, 108-121. Thorburn, P.J., Jakku, E., Webster, A. J., & Everingham, Y. L. (2011). Agricultural decision support systems facilitating co-learning: a case study on environmental impacts of sugarcane production. International Journal of Agricultural Sustainability, 9(2), 322-333. Tomer, M. D., Porter, S. A., Boomer, K. M. B., James, D. E., Kostel, J. A., Helmers, M. J., ... & McLellan, E. (2015). Agricultural conservation planning framework: 1. Developing multipractice watershed planning scenarios and assessing nutrient reduction potential. Journal of Environmental Quality, 44(3), 754-767. Tomer, M. D., Porter, S. A., James, D. E., Boomer, K. M., Kostel, J. A., & McLellan, E. (2013). Combining precision conservation technologies into a flexible framework to facilitate agricultural watershed planning. Journal of Soil and Water Conservation, 68(5), 113A-120A. Watson, S. B., Miller, C., Arhonditsis, G., Boyer, G. L., Carmichael, W., Charlton, M. N., ... Matisoff, G. (2016). The re-eutrophication of Lake Erie: Harmful algal blooms and hypoxia. Harmful algae, 56, 44-66. Wenkel, K., Berg, M., Mirschel, W., Wieland, R., Nendel, C., & Köstner, B. (2013). LandCaRe DSS - An interactive decision support system for climate change impact assessment and the analysis of potential agricultural land use adaptation strategies. Journal of Environmental Management, 127, S168–S183. 82 Zhang, D., Chen, X., & Yao, H. (2015). Development of a Prototype Web-Based Decision Support System for Watershed Management. Water, 7(2), 780–793 83