. I .. Ilfihrrwrrl— 3232!. 5:33! ‘ A. t: r v ‘ . 1 . . 41.. :L. 1.... .5152. fir 23:5..6 :r an L113... : . 10 ill! a .- 1H». .. .F . . . ‘ fin . $9.3"- “Em: A . . ‘- .1» $3.12.“... 3 3.4»??- :5 D) .. I . kWh . .53.“: Vetvil :‘I‘h—r ‘1; Dirt . .2: . . . V..H...h...n§ m 3.5 |::1 an?» _ ; .a £1 . 2 .mruiuuiun uh ‘I . . r 3‘ «1, it... , If u . "Lawuvufisvfl ‘ . «a .e . . .. 1:18”... mic . «.5 V . , s . 3:... 1%.... =. .. .2........o..r it... ..: up"... u . . l- : -~ 7.; rlN‘ . . , ir . . . , . . and... . . A . shun“. . . f... q . .. _ . 7, . .regmat 4%.. . .33 . , . 2...} i) .311 . . 3.2. 1%-}? u . “u I '3‘: I}. . . viii.“ . A . . :i .l . _ . . . . . I... V . ‘ .. . igmfirnfflrz :1 '~ 3 I. ‘r . , (7...... huh?! . I §J~2fl1¢li . g! I .3 ‘ \ 3n»... “1.25:1... unis? Emmy!» hi . d? . 2r. bunt? . I 1375.. - 3.33.6 ‘ 3:23.25. : . :11! 1:. 523.337! . . :2‘: a... ‘AMBXPJWI. \ .u n . flair»: .to to!) . I‘lrosv xl Sittiqn. . , r "yr 3 u.m | tel AIS-nlfili‘: '1 .ul ‘3? Surf‘s! '00 2‘ tea-5:9.) .1. zitv».\.‘ . .. 7.... 113.24,. s. 2 y .. 3 . ~ \ ’L')’,l‘l ,{l‘i- . 6. l\ 3-(‘1 ;v 3.1,. Li: ‘9...“ F124... \.l: . _ :Iiv. .1: . ; . nh‘lia‘..!.$§. \ . {113%. «.v . I 33.37 .lvtlll. vb.ll.~y :l‘J: rl ‘ . .I‘Vrv... . n. Ntv......, ‘Y‘I‘A: . 1’2. 1A}? a} I ‘1. I mfififiuufia .1 .. ... :3... {Fifihmwm .fiwfidmmw Limp... $1.3: immzf. . , THEBlS 2 goat) Illill'llllllllllllllllllllllllll'lll 3 1293 02048 6472 LIBRARY Michigan State University This is to certify that the dissertation entitled THE EMERGENCE OF STAKEHOLDER CONSENSUS: EXAMINING ISSUES IN EVALUATING SUSTAINABLE AGRICULTURE RESEARCH AND EDUCATION (SARE) presented by Susan B. Smalley has been accepted towards fulfillment of the requirements for Agricultural and Ph . D . degree in Extension Educat: ion AR“? \V t > ajor professor Date August 21, 2000 MS U is an Affirmative Action/Equal Opportunity Institution 012771 -—1- .—~—- <—-— _r -_ —' —.——.—— —- _ PLACE IN RETURN BOX to remove this checkout from your record. To AVOID FINES return on or before date due. MAY BE RECALLED with earlier due date if requested. DATE DUE DATE DUE DATE DUE :’58 1 4 2008. <:- 3 ‘ 78:1 11 1531‘!) 8 302 JUN} 8 2002 j? 11m mus-p.14 THE EMERGENCE OF STAKEHOLDER CONSENSUS: EXAMINING ISSUES IN EVALUATING SUSTAINABLE AGRICULTURE RESEARCH AND EDUCATION (SARE) By Susan B. Smalley A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Agriculture and Natural Resources Education and Communication Systems 2000 ABSTRACT THE EMERGENCE OF STAKEHOLDER CONSENSUS: EXAMINING ISSUES IN EVALUATING SUSTAINABLE AGRICULTURE RESEARCH AND EDUCATION (SARE) By Susan B. Smalley Although stakeholder involvement in program development is considered important, there are few tested models for meaningful stakeholder involvement in program evaluation, even in sustainable agriculture, where the basic tenets call for such involvement. This research sought to determine whether it was possible to construct a framework to allow and support stakeholder definition of evaluation issues, problems and recommendations associated with the North Central Region Sustainable Agriculture Research and Education (NCR SARE) program. A sample of NCR SARE stakeholders was selected based on nominations, distribution across states and distribution across stakeholder groups. The sample included present and past NCR SARE Administrative Council members, Technical Committee members, grant recipients, state coordinators and staff. Three Delphi surveys were sent to the stakeholder sample to elicit their views. Content analysis was used to organize responses into 14 Issue/Question Clusters and their associated Recommendation Themes. Issue/Question Clusters included useful information; whole system questions; new ways of producing, consuming, being; long-term sustainability; specific, measurable impacts; empowering people; who is currently involved; diversity of approaches, practices, systems; practicality; greatest barriers and critical components; grounding in research; measuring; sustainable agriculture practices; and outside evaluators. Clear Recommendation Themes included: (a) NCR SARE should clarify project expectations, target funding, increase follow-up methods, better utilize media, and generally become more active in generating and sharing useful information; (b) it should adjust its investment portfolio to include projects that are longer term and that address system-redesign questions; (0) it Should strengthen its focus on working with and through people, especially those engaged with small to mid-sized farming systems, community- based food systems, and those already committed to a more sustainable future; ((1) it should clarify its identity as overlapping with but not synonymous with either organic or conventional agricultural systems; and (c) it Should push for increased goal clarity but avoid over quantifying. Delphi surveys were found to be useful in allowing and supporting stakeholders to identifyI NCR SARE evaluation issues, problems and recommendations. C0pyright by SUSAN B. SMALLEY 2000 ACKNOWLEDGMENTS I am deeply grateful for the infinite patience and continuing support provided by my major advisor, Dr. S. Joseph Levine, my supervisor and mentor, Dr. Richard R. Harwood, my spouse, James Artabasy, and my parents, Wayne and Irene Smalley. TABLE OF CONTENTS LIST OF TABLES ...................................................... ix CHAPTER 1 INTRODUCTION ....................................................... 1 Purpose .................................................... 1 Background ................................................ 1 Relevance of the Study ....................................... 5 Research Questions .......................................... 6 Limitations ................................................ 6 Assumptions .............................. 1 .................. 8 CHAPTER 2 REVIEW OF RELATED LITERATURE ..................................... 9 Program Evaluation .......................................... 9 General Program Evaluation ............................ 9 Evaluating Broad Programs ............................. l 1 Evaluation Utilization ................................. 15 Participation in Evaluation .............................. 18 The Delphi Technique ....................................... 22 Definition ........................................... 22 Process and Format ................................... 24 Panel Selection ....................................... 25 Limitations, Disadvantages and Problems .................. 26 Advantages and Applications ............................ 26 Use of Expert Judgment ................................ 29 Sustainable Agriculture ...................................... 30 Background ......................................... 30 Defining Sustainability ................................ 35 System Levels and Processes ............................ 38 Dimensions of Sustainability ............................ 42 Sustainable Agriculture Research and Education Program ..... 47 CHAPTER 3 METHODS AND PROCEDURES ......................................... 51 Introduction ............................................... 5 1 Research Questions ......................................... 51 Population and Sample ...................................... 52 Population Description ................................. 52 Sampling Procedure ................................... 53 Instrumentation and Data Collection ............................ 58 Round One .......................................... 58 Round Two .......................................... 59 vi Round Three ......................................... 59 Response Rates and Patterns ............................ 60 Data Analysis .............................................. 64 Round One .......................................... 64 Round Two .......................................... 64 Round Three ......................................... 65 CHAPTER 4: FINDINGS ................................................ 66 Introduction ............................................... 66 Round One: Identifying Issues/Questions ........................ 66 Issue/Question Cluster A: Who is Currently Involved? ........ 67 Issue/Question Cluster B: Diversity of Approaches, Practices and Systems ............................................ 68 Issue/Question Cluster C: Long-term Sustainability .......... 68 Issue/Question Cluster D: Empower People ................ 68 Issue/Question Cluster E: Whole-systems Questions ......... 69 Issue/Question Cluster F: Useful Information ............... 7O Issue/Question Cluster G: Greatest Barriers and Critical Components ......................................... 71 Issue/Question Cluster H: Specific, Measurable Impacts ...... 71 Issue/Question Cluster 1: Sustainable Agriculture Practices . . . . 72 Issue/Question Cluster J: Measuring Sustainability .......... 72 Issue/Question Cluster K: Practical ....................... 73 Issue/Question Cluster L: Grounded in Research ............ 73 Issue/Question Cluster M: New Way of Producing, Consuming and Being .............................................. 73 Issue/Question Cluster N: Outside Evaluator ............... 74 Round Two: Prioritizing Issues/Questions and Identifying Associated Problems ................................................. 74 Issue/Question Cluster F: Useful Information ............... 76 Issue/Question Cluster E: Whole-systems Questions ......... 76 Issue/Question Cluster M: New Way of Producing, Consuming and Being .............................................. 77 Issue/Question Cluster C: Long-term Sustainability .......... 78 Issue/Question Cluster H: Specific, Measurable Impacts ...... 78 Issue/Question Cluster D: Empower People ................ 79 Issue/Question Cluster A: Who is Currently Involved? ........ 79 Issue/Question Cluster B: Diversity of Approaches, Practices and Systems ............................................ 80 Issue/Question Cluster K: Practicality ..................... 80 Issue/Question Cluster G: Greatest Barriers and Critical Components ......................................... 8O Issue/Question Cluster L: Grounded in Research ............ 81 Issue/Question Cluster J: Measuring Sustainability ........... 81 Issue/Question Cluster 1: Sustainable Agriculture Practices . . . . 82 Issue/Question Cluster N: Outside Evaluator ............... 82 vii Issue/Question Cluster 0: Summarize & Share Information . . . . 82 Round Three: Recommendations to Address Problems ............. 83 Recommendation Themes for Useful Information (F) ......... 83 Recommendation Themes for Whole Systems (E) ........... 87 Recommendation Themes for New Ways of Thinking, Producing. Consuming ......................................... 91 Recommendation Themes for Long-term Sustainability (C) . . . . 94 Recommendation Themes for Specific, Measurable Impacts (H) 96 Recommendation Themes for Empowering PeOple (D) ...... 100 Recommendation Themes for Who is Currently Involved? (A) ....................................... 102 Recommendation Themes for Diversity of Approaches, Practices and Systems (B) ..................................... 104 Recommendation Themes for Practicality (K) ............. 107 Recommendation Themes for Greatest Barriers and Critical Components (G) ..................................... 108 Recommendation Themes for Grounded in Research (L) ..... 11] Recommendation Themes for Measuring Sustainability (J) . . . 113 Recommendation Themes for Sustainable Agriculture Practices (1) ........................................ l 15 CHAPTER 5: CONCLUSIONS, DISCUSSION AND IMPLICATIONS .......... 118 Summary of the Study ...................................... 118 Conclusions: Emergence of Stakeholder Consensus .............. 120 Evaluation Issues .................................... 121 Evaluation Problems ................................. 125 Evaluation Suggestions and Recommendations ............ 126 Recommendations ......................................... 1 32 Suggestions for Further Research and Related Work .............. 133 Implications .............................................. 1 3 5 APPENDIX A: LETTER OF INVITATION & RESPONSE CARD .............. 137 APPENDIX B: ROUND ONE SURVEY ................................... 140 APPENDIX C: ROUND ONE RESPONSES ............................... 143 APPENDIX D: ROUND TWO SURVEY .................................. 155 APPENDIX E: ROUND TWO RESPONSES .............................. 162 APPENDIX F: ROUND THREE SURVEY ................................ 173 APPENDIX G: ROUND THREE RESPONSES ............................. 184 BIBLIOGRAPHY ..................................................... 203 viii LIST OF TABLES Table Page 2-1: Sustainable Agriculture Concepts and Key Authors ........................ 32 2-2: Agricultural System Dimensions ....................................... 42 3-1: NCR SARE Stakeholder Groups and Nominations ......................... 54 3-2: NCR SARE Stakeholders Invited and Agreeing to Participate by State and Stakeholder Groups ..................................................... 56 3-3: Individuals Invited to and Agreeing to Participate in Study ................... 57 3-4: Stakeholder Participation in Delphi Survey Rounds ........................ 61 4-1: Pressing Issues/Questions Identified in Round One ........................ 67 4-2: Order of Importance of Evaluation Issue/Question Clusters .................. 75 4-3: Recommendation Themes for Useful Information (F) ...................... 84 4-4: Recommendation Themes for Whole Systems (E) ......................... 88 4-5: Recommendation Themes for New Ways of Thinking, Producing, Consuming (M) .................................................... 92 4-6: Recommendation Themes for Long-Term Sustainability (C) ................. 95 4-7: Recommendation Themes for Specific, Measurable Impacts (H) .............. 97 4-8: Recommendation Themes for Empowering People (D) .................... 100 4-9: Recommendation Themes for Who is Currently Involved? (A) .............. 103 4-10: Recommendation Themes for Diversity of Approaches, Practices, Systems (B) ..................................................... 105 4-11: Recommendation Themes for Practicality (K) .......................... 108 4-12: Recommendation Themes for Greatest Barriers and Critical Components (G) . 109 4-13: Recommendation Themes for Grounding in Research (L) ................. 1 12 ix 4-14: Recommendation Themes for Measuring (J) ........................... 1 14 4-15: Recommendation Themes for Sustainable Agriculture Practices (1) .......... 1 16 CHAPTER I INTRODUCTION Purpose Stakeholder involvement in program development is important. An increasing number of rules and regulations in both the public and private sectors mandating stakeholder involvement underscores this importance. Program planners and directors often focus on involving stakeholders in planning. Meaningful stakeholder involvement in other aspects of program development such as needs assessment, implementation, and evaluation is less common. In the area of sustainable agriculture, stakeholder involvement in evaluation is not only important, but consistent with its basic tenets. Sustainable agriculture requires a systems perspective that includes ecological, economic and social dimensions. Stakeholders are an essential part of the system. Programs that address issues of sustainable agriculture need to expand and enhance the ways in which stakeholders are involved. This research was designed to examine one strategy for effectively involving stakeholders in the development of an evaluation schema that is appropriate for use in the area of sustainable agriculture. Background The North Central Region Sustainable Agriculture Research and Education program (NCR SARE) is part of a national program. Its mission is “to create and manage a system designed to encourage the involvement of farm and non-farm citizens in the process of discovery and learning that leads to achievement of more sustainable, environmentally benign agriculture” (N CR SARE, n.d.). Three competitive research and education grant programs support this mission in the twelve-state north central region. The United States Department of Agriculture invests approximately $1.2 million annually in 10-12 NCR SARE Research and Education grants for interdisciplinary one- and two-year projects; the NCR SARE Professional Development Program awards about $390,000 annually to support 10-12 training projects; the NCR SARE Producer Grant Program awards about $250,000 annually to support 30-40 farmer projects. Program policy is established and funding decisions are made by an Administrative Council comprising research, extension, farmer, and other representatives from across the region. A Technical Committee reviews and makes recommendations about proposals submitted for funding. Both public and private research and extension interests are included at the table. At its inception, SARE was unique among federal programs in its decentralization of decision-making and its emphasis on meaningful collaboration among researchers, extensionists, farmers and nonprofit organizations. In addition, the nature of sustainable agriculture systems — integrated, holistic, site-specific, balancing environmental, economic and social factors — has called for research and education approaches that differ from many more conventional agricultural efforts. One result of these differences is that evaluation approaches used elsewhere in agriculture have not always been appropriate for use within sustainable agriculture. The manner in which NCR SARE is structured has also contributed to its decentralization. State representation on its governing Administrative Council (AC) provides for four-year terms. The sector — research, extension or producer — for a particular state to fill is specified and rotates between terms. This generally prevents one person from representing a state for more than a single term. Technical Committee (TC) terms are three years, but more flexible membership guidelines have permitted successive terms for several of its producer members. The AC meets three times each year, typically for two to three days each time. The TC meets once each year for about two days. The two groups generally do not meet together. Members of both bodies are spread across twelve states and generally do not interact much or at all with one another outside NCR SARE business. Although SARE has liaisons for its Professional Development Program in each state, they have only limited interaction with the AC and TC. NCR SARE has dealt with evaluation issues from its early years. A 1992 United States General Accounting Office (GAO) program audit of the SARE program included the following recommendation: To increase the impact of the SARE program, improve its effectiveness and help ensure its integrity, we recommend that the Secretary of Agriculture direct SARE program management to establish 1) guidance and systems to collect, evaluate, synthesize and report the results of the SARE research projects at regional and national levels; and 2) national standards for regional monitoring of SARE. (GAO, 1992). Even before this report was released, periodic discussions and deliberations on evaluation philosophy, approaches and implementation occurred within NCR SARE and are documented in its program records. Many components of evaluation are in place within the program. These include: (a) a requirement for each project to include an evaluation component; (b) conduct of occasional project site visits by staff and/or Administrative Council members; (c) occasional invitations for project coordinator presentations to SARE Administrative Council; (d) evaluation information requested for final project reports; (e) feedback on proposal process invited from external peer reviewers and authors of non-funded proposals; and (f) inclusion of program assessment and impact in Administrative Council strategic planning process. (Waller, unpublished). Nevertheless, there is currently no overall program evaluation framework for NCR SARE, especially one that is participatory in nature, involving stakeholders in defining the evaluation schema. It is difficult to assess overall program impact after more than a decade of operation. Each of the three programs within SARE — Research and Education, Professional Development Program and Producer Grant Program — has its own approach to evaluation with few apparent linkages between programs. Various evaluation approaches have been suggested. One approach that has gained favor in recent years would identify and use indicators of sustainability. Hart (1995) asserted that effective indicators of sustainability have in common the following characteristics: (a) relevance to sustainability (viewed as integrating economy, environment and society); (2) understandable to the community at large; (3) developed and accepted by people in the community; (4) linking economy, society and environment; (5) focused on long-range view; (6) advancing local sustainability, but not at the expense of others; (7) based on reliable information; and (8) based on timely information. Hart suggested that two key components of sustainability are the concepts of community capital and carrying capacity. Community capital, comprising natural, social and built capital, allows inhabitants to live and interact productively. Carrying capacity is the ability of a community's capital to provide for the community's needs over the long term. In assessing relevance to sustainability, good indicators will address whether a community is maintaining and enhancing the capital on which it depends. NCR SARE also examined an approach called outcome funding. This would require the AC to become much more explicit about its goals. An invitation for proposals would specify in advance: (a) outcomes sought and acceptable ranges of project performance targets; (b) project portfolio objectives and rationales; (c) minimum overall return on investment in terms of tangible public gain; (d) statement of any non-negotiable implementation givens; and (e) what NCR SARE desired to learn from the investment round (Williams, Webb, Phillips, 1991). Relevance of the Study One hallmark of SARE has been its explicit requirement to involve farmers, researchers, extensionists, non-profits, and others in key leadership and decision-making roles. Bringing together such a diverse group and adding another dimension of geographic diversity may create challenges for a program in defining desired outcomes and impacts against which to evaluate its work. In addition, developing institutional memory and building on past efforts may prove difficult in a situation where funding and other key decisions are invested in a volunteer board with rotating membership, supported by a small staff. Enhancing its efforts‘in evaluation could place additional stress on the organization. The SARE model represented a radical departure from standard government agency structure when it was developed. Twelve years later, however, more agencies and organizations have mandated increased stakeholder or constituent involvement using a wide range of mechanisms to implement their directives. This research modeled one way to involve geographically dispersed stakeholders in a participatory process to identify and prioritize evaluation issues for an organization and provide direction to address some of the problems related to those issues. Research Questions The primary research question for this study was: Is it possible to construct a meaningful participatory framework to involve stakeholders in defining evaluation issues and the problems associated with these issues? This primary question incorporated several subsidiary questions. Are there clearly recognizable issues regarding evaluation in sustainable agriculture? If so, what are the issues? D0 evaluation issues in sustainable agriculture clearly group in major themes? Are there clearly definable problems associated with sustainable agriculture evaluation issues? What are the problems? Is there consensus regarding the problems associated with sustainable agriculture evaluation issues? Are there differences in the types of problems associated with each sustainable agriculture evaluation issue? What are the suggestions and recommendations to deal with these problems? Is there consensus regarding the suggestions and recommendations associated with sustainable agriculture evaluation issues? Are there differences in the types of suggestions and recommendations associated with sustainable agriculture issues? Limitations Limitations of this study include factors related to its sample, its methodology and its procedures. The sample for the study comprised NCR SARE stakeholders. It was not randomly drawn, but intentionally drawn from people who had been involved with NC R SARE and who were nominated by NCR SARE leaders. Types of NCR SARE involvement included participation on the AC or TC; receipt of a SARE grant; involvement as a state SARE liaison; and involvement as a paid SARE staff member. Another limiting factor related to the sample was that respondents had to agree to participate in a study that would require about three hours of their time, spread over three separate occasions, over a span of several months. Some of the NCR stakeholders contacted chose not to commit their time. The research methodology used for this research required written responses. Participants varied widely in the length, specificity, and clarity of their responses. Some responses may have been different if the same questions had been posed face-to-face or by telephone. In addition, the choice of written communication meant that responses lacked additional levels of communication such as vocal inflections and body language that might have added meaning to the responses. The open-ended nature of the questions posed in each research phase tended to make it difficult to objectively code responses and formulate unbiased next phase questions. A further limitation was the number of missing cases. Each succeeding research phase required more of the respondents’ time because each questionnaire became longer and more complex. The number of returns diminished from phase to phase. Also, the process took several months from the initial request to participate until mailing of the final phase. The time coincided with the 1999 growing and harvest season for much of the region, an extremely busy time for many NCR SARE stakeholders. The length of the survey, as it evolved in the final phases, may have caused the last questions to receive less attention than the first questions. Assumptions It was assumed that responses to the surveys reasonably reflected NCR SARE stakeholders’ attitudes and beliefs about SARE and sustainable agriculture. All participants were involved at some point during the past 12 years with NCR SARE as grant recipients, AC or TC members, state SARE PDP coordinators or staff members. It was assumed that their SARE involvement indicated some awareness and understanding of NCR SARE activities and procedures. CHAPTER 2 REVIEW OF SELECTED LITERATURE This chapter includes a review of literature related to program evaluation, the Delphi technique and sustainable agriculture. The section about program evaluation focuses on general program evaluation, evaluating broad programs, evaluation utilization. and participation in evaluation. The section about the Delphi technique addresses its definition; process and format; panel selection; limitations, disadvantages and problems; advantages and applications; and use of expert judgment. The section about sustainable agriculture addresses its background, attempts to define sustainable agriculture, system levels and processes, dimensions of sustainability, and the Sustainable Agriculture Research and Education program. Program Evaluation General Program Evaluation Stake (1972) identified the basic characteristics of evaluation activities as “the evaluation acts, the data sources, the congruence and contingencies, the standards, and the uses of evaluation” (p. 33). He maintained that, “both description and judgment are essential — in fact they are the two basic acts of evaluation . . . . To be fully understood, the educational program must be fully described and fully judged” (pp. 33-34). He went on to provide a matrix to help organize the various types of data used for evaluation. He defined antecedent data as “any condition existing prior to teaching and learning which may relate to outcomes” (p. 36); transaction data as “the succession of engagements which comprise the process of education” (p. 37); and outcome data as “the consequences of educating — immediate and long-range, cognitive and conative, personal and community—wide” (p. 37). With Stake’s approach, the evaluator would state the program rationale. Then. in describing an educational program, he or she would use the matrix to record what educators intended (their program goals and objectives) as well as what observers perceived in terms of program antecedents, transactions and outcomes. These descriptive data could be analyzed by examining the relationships between intentions and observations for their degree of congruence. That is, to what extent did what was intended actually happen? The data could also be analyzed by looking for relationships or contingencies between antecedents, transactions and outcomes (p. 42). In judging an educational program, the evaluator would record both “what patrons generally expect and . . . what judges value the immediate program to be,” (p. 39) again in terms of program antecedents, transactions and outcomes. More recently, the Government Accounting Office (GAO) defined both program measurement and evaluation. Program measurement was defined as “the ongoing monitoring and reporting of program accomplishments, particularly progress towards preestablished goals.” Program evaluations were defined as “individual systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working” (GAO, 1998). According to the GAO, Performance measurement focuses on whether a program has achieved its objectives, expressed as measurable program standards. Program evaluations typically examine a broader range of information on program performance and its context than is feasible to monitor on an ongoing basis (GAO, 1998). 10 Evaluating Broad Programs Weiss and Rein (1972) discussed challenges in evaluating broad—aim programs. They warned against administering a pre- and post-program survey that seeks to find possible changes over a large number of narrowly-defined criteria. Instead, they proposed that, the need in the study of broad-aim programs is for a conscientious attempt to find the forces which shaped the specification of the program, the nature of the Opposition it encountered, the reasons for such failure as occurred, the program‘s unanticipated consequences. Then, in addition, the research might identify the anticipated changes which occurred and the ones which did not (p. 240). They recommended a methodological approach for evaluating broad-aim programs that is process-oriented, historical and comparative. Such an approach might: (a) use systems theory as a conceptual framework; (b) include analysis at concrete, process and theoretical levels; and (c) explicitly deal with program consequences and values. Andrews, Ashcraft, Thullen and Lock (1992) described an approach to cluster evaluation — evaluation that involves multiple sites and multiple methods. They stressed the importance of finding the logical or conceptual linkages which may be explicit or implicit across sites and projects to develop an evaluation framework. The framework then helps evaluators "focus on critical areas of inquiry appropriate to the field of study within which the project exists, thus building a broader understanding of practice" (p. 98). Program managers at the state and federal level have long faced the challenge of evaluating a range of programs, each planned to deal with aspects of the same broad problem but each tailored to specific local circumstances. One example is evaluating the 11 National Extension Water Quality Program. Its federally-funded state efforts from 1992 through 1995 were evaluated (Marshall and Bennett, 1998). The . . . program had five goals, which framed national program components to protect or improve water quality in state-defined geographic areas of need, through improved: management of animal wastes . . . ; management of commercial nitrogen fertilizers; management of crop pesticides; public issues education . . . ; and protection of drinking water. . . . Participating State Extension Services aimed typically at four or five Targeted Program goals, i.e., at those that pertained to priority water quality needs in their particular state (p. 2). The evaluation analyzed program components (developed around the five goals), program outputs (methods and activities implemented to address specified problems related to each component and content of information and education conveyed), program outcomes (changes in behavior patterns of participants and those they influence as well as related environmental, economic and social consequences at least in part attributable to the program) (pp. 8-9). Positive outcomes were defined as movements toward numerical targets . . . and trends consistent with the (program) objectives (p. 10). Several of Marshall and Bennett’s recommendations are particularly relevant to this study. They suggested that: (a) state and federal evaluators collaborate to jointly identify and use appropriate indicators; (b) factors such as records, expert assessment, industry changes and resources committed be used to make base projections; (c) evaluators strive for more consistent and direct reporting of specific practice changes; and ((1) end results “be defined in terms of appropriate immediate, intermediate, and/or ultimate ends that are feasible within the reporting time frame and within Extension’s evaluation capabilities” (p. 33). Young, Gardner, Coley, Schorr and Bruner (1994) provided a method to develop outcome indicators in models of integrated services. Their model was built on a theoretical framework linking elements of needs/assets, goals, resources, activities, short- terrn outcomes, and long-term outcomes. Elements of the model are ordered such that 12 outcomes can be defined only after the first four elements. Outcomes can be measured at five levels: (a) an individual client; (b) client outcomes aggregated into program outcomes; (c) program information aggregated into agency outcomes; ((1) agency outcomes aggregated into system or community outcomes; and (e) community-wide outcomes. Young et al. (1994) indicated that, " goals and outcome measures serve different purposes. The former represent what the community is striving for. The latter represent what the community will be held accountable for” (p. 19). So outcome measures represent minimal rather than maximal objectives. Their recommendations for developing outcomes included: (a) a wide range of policymakers, funders and citizens should consider the measures important and meaningful; (b) data about the measures Should be relatively easy to obtain and interpret; (c) clear distinctions should be maintained between outcome measures and process or capacity measures and the rationale for using any process or capacity measures should be clearly understood or stated; (d) the least ambiguous available measure of an outcome should be used; and (e) to the extent possible, outcomes that cannot be misused should be chosen. According to Taylor-Powell, Rossing and Geran (1998), varying interpretations of both “evaluation” and “collaboration” Often create misunderstanding. Evaluating broad problems often involves dealing with collaborations, which are appropriate for situations with complex problems, limited resources, social fragmentation, disengaged citizens and sweeping change. They defined evaluation as “a process to systematically collect, analyze and interpret information in response to critical questions to inform program and/or other organizational decision making, judgments, and learning”(p. 2), and 13 collaboration as a specific type of joint effort that develops shared vision, builds and interdependent system to address issues and opportunities, and shares resources (p. 5). They advocated developing a logic model that creates a theory of action to link the situation, inputs, outputs, outcomes and impact. Such a logic model created by members of the collaboration can help to develop understanding, monitor progress. serve as evaluation framework, bare assumptions, restrain over-promising, and promote communication. Kibel (1999) pointed out that different types of programs yield different evaluation opportunities. Programs designed to fix or cure something are frequently designed with a specific intervention or a narrow range of possible interventions applied in a fairly standard way to relatively passive recipients. In Situations where each client or participant receives the same service, advice, or intervention, with expectations of a similar result, the participant counts and success rates of the program provide a realistic picture of the program’s performance. It is often possible to argue that the program caused any changes observed in its clients (p.8). Other programs, however, are designed to help their clients grow, heal, become transformed, or prevent some negative consequence. These programs typically include a range of services, customized to meet the need of each client. They try to not only make short-term changes in clients, but to support the clients in longer-term, more fundamental change efforts. They may join forces with other programs or organizations in these efforts, and clients may be actively involved as well. Body counts, satisfaction surveys and numeric measures of specific outcomes do not adequately portray the performance of these programs. Attribution is complex, impacted by multiple services, wide variation in 14 services provided, strong outside influences, long distance from service to outcomes, multiple outcomes, rich diversity of outcomes, and active client roles. Most sustainable agricultural education programs are much closer to the second model than the first. Kibel developed results mapping, an evaluation approach designed specifically to capture performance of transforrnative programs. It can be used, to map, score, analyze, and provide feedback to improve the best work that a program does with its clients, be these individuals, families, teams, groups, communities, organizations, or systems. Each story features some of that best work. . . . It is a story that begins with the first interaction between program and the client and extends to further program-client interactions, to program interactions with the client’s support system, to client interchanges with others called on by the program to assist the client, and to personal client achievements in support of self or to benefit others (p. 45). Mapped data is scored and can yield information about interim, intermediate, longer-term and ultimate outcomes. Performance measures are available to examine overall points, services by level and provider type, and networking activities. Blending Stories, outcome data and scores provides data that are accurate, evoke confidence, and are consistent with current science. Evaluation Utilization Weiss (1972b) addressed factors that affect utilization of evaluation. She indicated particular interest in the nature of the evaluation itself. At present, evaluation usually examines conditions before and after the program and comes up with global findings on the extent of change. But rarely can it answer questions about which elements of the program amalgam worked or did not work, and how and why. Yet it is just such information that is vital for institutionalizing a program into routine practice and transferring it to other locations . . . . Utilization might be increased if the evaluation included such elements as these: 1. the explication of the theoretical premises underlying the program, . . . 15 2. specification of the ‘process model’ of the program . . . 3. analysis of the effectiveness of components of the program . . . (p. 323). Weiss went on to suggest several additional procedures that she deemed worthy to consider: 1. Early identification of potential users of evaluation results and selection of the issues of concern to them as the major focus of study . . . . 2. Involvement of administrators and program practitioners, from both inside and outside the project. in the evaluation project . . . . 3. Prompt completion of evaluation and early release of results . . . . 4. Efifective methods for presentation of findings and dissemination of information (pp. 324-325). Cozzens (1997) discussed some of the special challenges in evaluating fundamental research programs. By definition, the primary goal of any research program is to increase understanding of a physical, social, or technological phenomenon. While understanding itself is hard to quantify, knowledge production has proven to be at least in part measurable. Three aspects of the knowledge produced under research programs are generally of interest to agency program managers: quantity, quality, and importance (p. 80). Frequently used indicators for these dimensions include publication counts, citations per publication, doctorates produced, and user involvement and satisfaction ratings. These indicators may represent research outputs, but they say virtually nothing about research outcomes or impacts. Cozzens described a logic model for fundamental research which She called “the knowledge pool” (p. 86). As soon as they are produced, the outputs of research activities join a pool of knowledge and human resources that is fed, not just by one agency’s activities, but by the activities of many government agencies, a variety of private organizations such as industrial firms and nonprofit institutions, and the world research community. In the knowledge pool, ideas and people interact and produce innovation and discovery through unpredictable paths and at uneven intervals. The practical value of the knowledge pool is demonstrated only when someone trying to solve a practical problem dips into it for the needed resources . . l6 . . The dipping, like the appearance of discoveries, also happens at unpredictable and uneven intervals, and each dip pulls up a mixed product of the many contributing streams. . . . Research funding organizations can track the outputs of the activities they fund into the pool. But if they try to track each drop they have contributed through the pool to its outcomes, they will end up spending more money tracking than they spent to support the research (p. 87). Patton (1997) also dealt extensively with issues of utilization. He differentiated between program evaluation in general and utilization focused program evaluation. Program evaluation is the systematic collection of information about the activities. characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. Utilization-focused program evaluation (as opposed to program evaluation in general) is evaluation done for and with specific, intended primary users for specific, intended uses (p. 23). After examining a list of 11 factors derived from the utilization literature about the extent to which they affected the use of findings for 20 federal health evaluations, Patton found that two factors emerged as key —— political considerations and the personal factor. He defined the personal factor as, the presence of an identifiable individual or group of people who personally care about the evaluation and the findings it generates. Where such a person or group was present, evaluations were used; where the personal factor was absent, there was a correspondingly marked absence of evaluation impact . . . . These are the people who actively seek information to make judgments and reduce decision uncertainties. They want to increase their ability to predict the outcomes of programmatic activity and thereby enhance their own discretion as decision makers, policymaker, consumers, program participants, and funders, or whatever role they play. These are the primary users of evaluation (p.44). Patton discussed some practical implications of the personal factor for evaluators. Find and cultivate people who want to learn. Formal position and authority are only partial guides in identifying primary users. Find strategically located people who are enthusiastic, committed, competent, and interested. Quantity, quality, and timing of interactions with intended users are all important. Evaluators will typically have to work to build and sustain interest in evaluation use. Building effective relationships with intended users is part selection, part nurturance, and part training. Evaluators need people skills in how to build relationships, facilitate groups, manage conflict, walk political tightropes, and communicate effectively. 17 A particular evaluation may have multiple levels of stakeholders and therefore need multiple levels of stakeholder involvement (p. 54). Participation in Evaluation Closely related to evaluation utilization is participation in evaluation. In our strongly democratic culture, participation is nearly always viewed as positive. Whitmore (1998) asserted that, The idea of stakeholder participation in evaluation is now widely accepted within the evaluation community. Yet the meanings, even the purposes, of stakeholder participation in evaluation remain diverse, multiple, and thus clouded in many contexts (p. 1). She identified themes in participatory evaluation as: (a) practical participatory evaluation, a pragmatic approach to foster evaluation use; (b) transforrnative participatory evaluation, based in social justice and focused on empowering oppressed groups; and (c) participatory evaluation as a process of engagement rather than a fixed set of methods (p. 1). Cousins and Whitmore (1998) proposed that any application of participatory evaluation can be analyzed with regard to where it falls on three continua: (a) control of the evaluation process, from completely researcher controlled to completely practitioner controlled; (b) stakeholder selection, ranging from restriction to primary users to inclusion of all legitimate groups; and (c) depth of participation, ranging from consultation to deep participation (p. 10). Burke (1998) identified key elements of participatory evaluation processes: (a) key stakeholders actively involved in decision making; (b) inequities of power and voice among participating stakeholders acknowledged and addressed; (c) process is explicitly 18 political; (d) multiple and varied approaches used to codify data; (e) action component incorporated to be useful to the program’s end users; (t) explicit aim to build evaluation capacity so stakeholders can control future evaluation processes; and (g) process must be educational (pp. 45-46). King (1998) reflected on two decades of evaluation experience to distill the following practical lessons for participatory evaluation: (a) high levels of interpersonal and organizational trust are necessary; (b) people involved in participatory evaluation efforts must create shared meaning of their experiences over time, requiring communication and interaction; (c) participatory evaluation must address the power structure within which it is conducted; ((1) both volunteers and leaders are'needed; (e) participatory evaluation processes require adequate time and cannot be rushed; (f) tackling important issues and having appropriate resources are important incentives to foster participatory evaluation; and (g) outside facilitators play important roles (pp. 63- 65). In a review of public perception of program effectiveness and worth, Rocheleau (1986) acknowledged the existence of “very little empirical research concerning public or elite perceptions of specific domestic programs” (p. 35). Based on the limited research available, however, he proposed a framework to study public perceptions of public programs. The framework incorporated variables, that can have an important impact on perceptions: individual background characteristics, individual experiences, interpersonal experiences, elite perceptions, interest group activities, media impacts, and program characteristics (p. 36). In a case study that dealt with participation in stakeholder-based evaluation, Mercier (1997) observed that, 19 inequality among participants was the factor that most hindered the smooth operation of the stakeholder-based evaluation . . . . Certain participants were more familiar than others with the research and evaluation process. Although involving participants can give them the opportunity to discuss and appropriate the evaluation process, potential benefits are reduced if some participants feel out of their depth (p. 472). She went on to conclude that, stakeholder-based evaluation calls for reflection on the means to enhance and sustain the chances of equitable participation and to alleviate the risks of unforseen and undesirable effects . . . (p. 474). Brandon (1999) discussed some theoretical and practical issues related to stakeholder participation in reviewing evaluators’ recommendations. He suggested that this type of Stakeholder input can allow the evaluator to tap contextual knowledge and move more effectively from evaluation findings to recommendations. He indicated that, “the stakeholder groups that have the appropriate program expertise for reviewing evaluation recommendations are program staff or faculty and program beneficiaries . . (P. 364). Turnbull (1999) developed and tested a model to explain how participation can be expected to increase the use of evaluation information. He found that participatory evaluation is likely to result in increased use if participants perceive that: (a) their workplace goals are participative; (b) they are able to participate to a desired degree; (c) they perceive that they have an influence in the decision- making process; (d) they believe that the participatory process was efficacious in that it achieved its intended Outcomes (p. 140). Folkman and Rai (1997) used community dialogues as a vehicle for self- evaluation in a participatory, community-based evaluation effort. They selected this strategy as fitting their theoretical framework integrating constructivist theoretical assumptions, collaborative research principles and action science orientation (p. 456). This perspective assumes that, 20 an evaluation of program performance naturally occurs in the larger community through informal conversation among different groups or stakeholders . . . . From a constructivist perspective, the meaning of the program and its impact is being framed within these conversations independent of the more formal evaluations that are being undertaken by ‘independent’ evaluators (p. 458). There are multiple, contradictory realities or program impacts which represent socially constructed interpretations . . . . All stakeholders. including evaluators, collaborate in a value laden process of creating data and deriving interpretations about what outcomes are being produced . . . . From this perspective evaluation should focus on the making of meanings, i.e., the values, assumptions, data and lines of reasoning that are involved in framing different aspects of program performance . . . . The goal is not to predict and control outcomes but to reveal through dialogue and critical reflection the underlying values, assumptions and reasoning behind points of view that may be contradictory (pp 461 -462). Mathie and Greene (1997) examined the importance of stakeholder diversity in participatory evaluation. They asserted that, intended as a tool for transformation, the potential of participatory evaluation stems from its democratic base; it requires a sharing of power, and stimulates a strengthening of the analytical powers of all participating stakeholders. It encourages mutual understanding and appreciation of different perspectives, and that in turn can be the precursor for both intellectual transformation and social action (p. 279). Diversity within the Stakeholder participants is required both to develop “a holistic understanding of program meaning” and to “promote and enable . . . a ‘democratizing conversation’ so that action or change can take place beyond the evaluation’s boundaries, steered by engaged and committed participants” (p. 279). The ideal —— to have all perspectives engaged — may be difficult or impossible in reality. The authors argued that “although diverse and multiple perspectives are key to a holistic understanding of program experience, the same extent of diversity may not be as necessary for action in the larger context” (p. 282). They further suggested that, “diversity should not exceed the capacity of the group to embrace and work with it, given 21 the time constraints participatory evaluation practice typically has to work under" (p. 282). Mathie and Greene stressed that true participatory evaluation must go beyond the stage which just provides an opportunity for each voice to speak, each perspective to be expressed. It must continue to a deeper level of engagement in which participants learn to really listen and respond to one another. Applied to participatory evaluation, a democratic conversation requires that more powerful stakeholders acknowledge the legitimacy and practice of forms of knowledge other than their own . . . . We may . . . have to settle for less diversity if engagement, rather than representation, in the participatory process by the full range of stakeholders cannot be guaranteed (p. 283). According to Mathie and Greene, “the role of the evaluator is to enable a democratizing conversation to take place” (p. 283). Ashton (1998) examined facilitative evaluation approaches. She defined the role of an evaluator using collaborative models as a third-party intervener whose speciality is helping the parties frame realistic goals, measure progress towards operationalizing them, recognizing when a change of strategy may be required, and extract insights from their hard labors (p. 7). The Delphi Technique Definition The Delphi technique or process was developed in the mid-1950's as a way “to get a reliable consensus of opinion among people with exceptional knowledge about a particular subject area . . . . It uses repeated individual questioning and feedback to arrive at a consensus” (Johnson, Meiller, Miller and Summers, 1987, p. 110). Dalkey and Helmer (1963), who originally devised Delphi, said it was a technique 22 designed “to obtain the most reliable consensus of opinion of a group of experts. It attempts to achieve this by a series of intensive questionnaires interspersed with controlled opinion feedback” (p. 45 8). Delphi is a technique to use expert judgement. Its essence, according to Averch (1974), is “structured, indirect, iterative interaction among experts with Centralized control, tabulation, and feedback of information and judgments” (p. 300). Delp, Theisen. Motiwalla and Seshadri (1977) defined Delphi as “a group process technique for eliciting, collating, and generally directing informed (expert) judgment towards a consensus on a particular topic” (p. 168). Farmer and Richman (1963) defined Delphi as a method for obtaining a consensus of opinion about a matter not subject to precise quantification . . . . , where interactions of variables, difficult aggregations, and difficulties of quantification make it impossible to apply more common methodologies (p. 329). According to Campbell and Hitchin (1968), Delphi “was first developed as a method of integrating the opinions Of experts without sacrificing or compromising individuals’ suggestions and ideas . . .” (p. 37). This approach requires that a panel of experts on the subject under study be selected. These individuals are then asked to independently develop their best answers to the questions being asked . . . . They are required to make their underlying assumptions explicit and to identify any source material that they would find helpful in refining and improving their answers . . . . Each expert is given the composite replies of the group . . . . Successive revisions . . . are undertaken . . . . Finally, a composite forecast is compiled (p. 38). Linstone and Turoff (1975a) characterized Delphi as a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem. To accomplish this ‘structured communication’ there is provided: some feedback of individual contributions of information and knowledge; some assessment of the group judgment or view; some opportunity for individuals to 23 revise views; and some degree of anonymity for the individual responses (p. 3). The Delphi process has been frequently used to deal with policy issues and alternatives. Turoff (1975) defined a policy Delphi as “a tool for the analysis of policy issues and not a mechanism for making a decision. Generating consensus is not the prime objective” (p. 84). Rather than consensus, Turoff identified possible objectives for policy Delphis as: (a) to ensure that all possible options have been put on the table for consideration; (b) to estimate the impact and consequences of any particular option; and (c) to examine and estimate the acceptability of any particular option (p. 87). Process and Format Linstone and Turoff (1975a) identified four phases that generally occur when using Delphi: (a) exploring the subject, with each participant contributing information; (b) understanding how the group views the issue; (c) exploring any disagreement to examine underlying reasons; and (d) final evaluation (pp. 5-6). There is general agreement about the steps in the Delphi process (Linstone, 1978, pp. 274-5, cited in Merriam and Simpson, 1989, p. 122): 1. Formation of a team to undertake and monitor a study. 2. Selection of one or more panels to participate in the exercise — normally experts in the subject area. Development of the first round Delphi questionnaire. Testing of the questionnaire for proper wording (e. g. ambiguities, vagueness). Transmission of the first round questionnaire to the panelists. Analysis of the first round results. Preparation of the second round questionnaire (and possible testing). Transmission of the second round questionnaire to the panelists .Analysis of second round responses (Steps 7 to 9 repeated until desired or necessary data are collected). 10. Preparation of a report by the analysis team to present conclusions of the exercrse. ©¢HQMPP 24 Turoff (1975) identified six phases in a policy Delphi. Formulating the issues . . . . Exposing the policy options . . . . Determining initial positions on the issues . . . . Exploring and obtaining reasons for disagreement . . . . Evaluating underlying reasons . . . . Reevaluating options . . . (p. 88). P‘MPPPE“ He indicated that, a Policy Delphi deals largely with statements, arguments, comments, and discussion. To establish some means of evaluating the ideas expressed by the respondent group, rating scales must be established for such items as the relative importance, desirability, confidence and feasibility of various policies and issues (Turoff, 1975, p. 89). Weatherman and Swenson (1974) identified variables to be considered in order to assure a Delphi’s validity and reliability. Their participant variables include panel representativeness, panel appropriateness and competence, panel commitment, questionnaire clarity, response independence, panel personality differences and non- respondents. Their procedural variables include pertinent items, interval between rounds, method of reporting previous responses, number of questionnaires, round one questionnaire format and showing relationships among events (pp. 103-110). Panel Selection Deciding who to include on a Delphi panel is a key design factor. Linstone and Turoff (1975b) indicated that, “widening, or broadening, of the concept of ‘experts’ to that of ‘informed’ is becoming quite customary in the application of Delphi” (p. 80). Scheele (1975) indicated that a successful mix of Delphi panelists includes stakeholders, those who are or will be directly affected; experts, those who have an applicable specialty or relevant experience; and facilitators, those who have skills in clarifying, organizing, synthesizing, stimulating . . . plus, when it seems 25 appropriate, individuals who can supply alternative global views of the culture and society (p. 68). According to Averch (1994), there are many ways to select experts. “Technical experts may be mixed with political experts, or even program constituents and clients, to bring different perspectives to bear in the evaluation process” (p. 298). He recommended that, to maximize the amount of information from an evaluation . . . the expert cadre should comprise more than technical, substantive experts. In addition to substantive experts, the group might include general purpose policy analysts, philosophers of evaluation, or stakeholders. This mix reduces the dependence on experts, widens the base of experience in the group, and allows important nontechnical questions to be raised. Furthermore, the diversity in perspectives may force the group to resolve arguments and clarify recommendations, although consensus may be more difficult to achieve (p. 309). Averch also observed that “evaluation users generally find experts through direct or indirect reputational procedures” (p. 304). Limitations, Disadvantages and Problems Andranovich (1994) identified three important limitations of Delphi: (a) participants must communicate clearly in writing; (b) the process is labor-intensive and time-consurning; and (c) participants must be interested and motivated (pp. 2-4). Linstone (1975) identified potential pitfalls when using Delphi: (a) tendency to discount the future; (b) urge to predict; preference for simplicity over complexity; (c) illusory experience; ((1) Sloppy execution; (e) optimism-pessimism bias; (f) overselling; and (g) deception. 26 Advantages and Applications Andranovich (1994) characterized Delphi as designed for non-interacting groups. Non-interacting groups can include groups whose members are geographically distant, groups whose members tend to clash, or groups in which status differences might affect decision making (p. 1). He went on to say that, Delphi is used when it is important to have pooled judgment . . . . [it] allows the group to share responsibility. Shared responsibility is a tonic for developing consensus. Shared responsibility also promotes satisfaction through participation in and ownership of the resulting decision(s) (p. 2). Following its general process, Delphi allows considerable flexibility and many modifications have been introduced since its inception. Weatherman and Swenson (1974) indicated that the Delphi technique has been used in a variety of disciplines and for an increasing number of general purposes. Instead of open-ended questions, the format of the initial questionnaire may use specific items to be evaluated. The number of iterations may be adjusted. Additional questions may be inserted . . . . The purposes for which the Delphi study is conducted may be only tangentially related to future forecasting. The number and characteristic of panel members may be varied . . . (p. 99). They went on to list applications of Delphi including strategy probe, preference probe, forecasting probe, perceptions of a current situation, and others (pp. 99-102). Linstone and Turoff (1975a) also saw a wide range of potential Delphi applications. When viewed as a communication process, there are few areas of human endeavor which are not candidates for the application of Delphi. While many people label Delphi a forecasting procedure because of its significant use in that area, there is a surprising variety of other application areas (p. 4). One area they identified that is especially relevant to this study is exposing priorities of personal values and social goals. 27 It is not, however, the explicit nature of the application which determines the appropriateness of utilizing Delphi; rather, it is the particular circumstances surrounding the necessarily associated group communication process (Linstone and Turoff, 1975a, p. 4). Several of the seven properties that Linstone and Turoff mentioned as usually leading to use of Delphi applied to this study: 1. The problem does not lend itself to precise analytical techniques, but can benefit from collective subjective judgments on a collective basis . . . . 2. More individuals are needed than can effectively interact in a face to face exchange . . . . 3. The heterogeneity of the participants must be preserved in order to assure validity of the results, i.e., avoidance of domination by quantity or strength of personality (Linstone and Turoff, 1975, p. 4). Moore (1987) indicated that although a principal use of Delphi has been to forecast, it can also be used to “identify goals and objectives, array possible alternatives, establish priorities, reveal group values, gather information, and educate a respondent group. Delphi is useful whenever it is desirable to have pooled judgment” (p. 50). Linstone and Turoff (1975b) provided cases in which Delphi was used as a communication system for policy questions. They defined a policy question as one involving vital aspects, such as goal formation, for which there are no overall experts, only advocated and referees. Its resolution must take into consideration the conflicting goals and values espoused by various interest groups as well as the facts and staff analyses (p. 75). They went on to assert that translating “scientific knowledge into informed judgment on evaluating and analyzing decision options” is a potential area for using Delphi (p. 80) and they observed that Delphi can be effectively used by asking panelists to select the most important items from many more candidates, in essence “the process of getting a group to filter out the signal of real information from the multitude of communications or noise that may exist on a particular complex topic” (p. 83). 28 Weaver (1971) identified several examples of Delphi use in education. A Delphi study by Olaf Helmer in 1965 was part of a project to elicit preference judgments from a panel of education and related experts on their goals for possible federal funding. This study and two others that Weaver discussed, “differ in principle from the original use of Delphi. In the three studies, respondents were asked to focus on what they would like to see happen rather than what is likely to happen” (p. 268). Weaver concluded that, Although Delphi was originally intended as a forecasting tool, its more promising educational application seems to be in the following areas: (a) a method for studying the process of thinking about the future, (b) a pedagogical tool or teaching tool which forces people to think about the future in a more complex way than they ordinarily would, and (c) a planning tool which may aid in probing priorities held by members and constituencies of an organization (p. 271). Conditions recommended for Delphi use include adequate time for the process, participants with good writing skills and motivation for participants to respond. Delphi encourages individual thought and minimizes both direct confrontation of people with differing viewpoints as well as pressure to conform. A sample size may be as small as ten or as large as several hundred people (Johnson et al., 1987, p. 110). Use of Expert Judgment Averch (1994) suggested that although expert program evaluation is relatively rare in the traditional social sciences, it is the preferred method in assessing results in science, technology and higher education programs (p. 293). He recommended consideration of expert eValuation for use with programs subject to input and output uncertainty, and he characterized that state in the following manor: 1. Suppose that for t years some public agency has been operating a ‘program.’ 2. The agency cannot be certain about the effective quantity or quality of inputs it has bought during the t years, and there is no easy way to measure these. 29 3. The expected ‘benefits,’ ‘outputs,’ or ‘outcomes’ of the program are highly uncertain or occur in the future. 4. The agency does not know with precision whether decision-relevant outcomes can be attributed to the inputs and the design of the program (p. 294). Averch defined validity and reliability when using expert judgment in program evaluation. Validity of expert judgment in program evaluation means . . . a decision maker accepts some individual or collective expert judgment about a program and acts on its basis; as a result, social benefits are realized and social costs are avoided . . . . In an evaluation context, reliability means that other experts looking at the same information would come to approximately the same judgements (p. 295). Helmer and Rescher (1960) confronted what they termed “the mythology of exactness” (p. 25) and presented a strong case for the role of expert judgment in the “inexact sciences.” Epistomologically speaking, the use of an expert as an objective indicator . . . amounts to considering the expert’s predictive pronouncement as an integral, intrinsic part of the subject matter, and treating his reliability as a part of the theory about the subject matter . . . . Our ‘data’ are supplemented by the expert’s personal probability validations and by his judgements of relevance . . . and our ‘theory’ is supplemented by information regarding the performance of experts (p. 43). Sustainable Agriculture Background Sustainable agriculture has been a recognized area of research, education and activity in the United States since the early to middle 1980's. Both the terminology used and definitions have evolved through this time. Although there is no single definition of sustainable agriculture upon which everyone agrees, a general consensus exists for three dimensions of sustainability — environmental, economic and social. Examining progress along these three dimensions, plus interaction and balance among them provides a 30 framework within which to identify and situate indicators of sustainability that can help evaluate a program. In order to understand the relatively recent discussion and controversy about sustainable agriculture in the United States, it is important to be aware of earlier contributing factors. Harwood (1990) traced the history of sustainable agriculture and pointed out that the current debate has roots that include: (a) the prevailing Newtonian world view with its mechanical View of nature, dichotomies between nature and society. faith in progress, and consumer ethic; (b) Thomas Jefferson’s link between morality and agricultural practices; (c) the break between “systematic” and “scientific” agriculturists for developing and sharing technical knowledge; (d) Steiner’s development of biodynamic agriculture; (e) Howard’s contributions regarding humus farming; (t) Rodale’s extensive writing and popularization of organic agriculture; and (g) successes and unintended consequences of the Green Revolution. Robertson and Harwood (in press) traced development of the sustainability concept, especially as it is applied to agriculture. Table 2-1 summarizes the key ideas they identified and the authors who contributed them. According to Madden (1998), a key participant in the development of sustainable agriculture ideas and programs, The first seeds of what has grown into the SARE Program were sown in 1962 by the publication of Rachel Carson’s classic book, Silent Spring. . . . Until this time, it was generally believed that pesticides were harmless to the environment, and when used properly, posed no threat to human health and water quality (p. 5). A controversial and often-cited Report and Recommendations on Organic F arming was released by the United States Department of Agriculture in July 1980. Its introduction, from then Secretary of Agriculture Bob Bergland, stated, 31 Table 2-1: Sustainable Agriculture Concepts and Key Authors SUSTAINABLE AGRICULTURE CONCEPTS KEY AUTHORS Scale (relationships between farm size, community, region, globe) Robertson Sense of place (social & ecological), connections with the land Berry, Jackson Natural resource preservation Leopold, Bromfield, Faulkner Organic, biodynamic J .I. Rodale, R. Rodale Pesticide concerns Carson Transition to “biological” focus Rifltin Agroecology (holistic framework based on biology, biochemistry) Altieri, Gliessman Farmer and other stakeholder participation Harwood Community well-being Dahlburg, Heffeman, Lockertz Extemalities (factors outside current monetary process) Shuman, Harwood Carbon & nutrient recycling Harwood, Robertson Ecosystem services Shuman, Soule & Piper Production efficiency Harwood, Robertson Many large-scale producers as well as small farmers and gardeners are showing interest in alternative farming systems. Some of these producers have developed unique systems for soil and crop management, organic recycling, energy conservation, and pest control (United States Department of Agriculture, 1980, p. iii). Although that report focused specifically on organic production, it stimulated 32 considerable discussion and debate about a range of alternative agricultural systems. In his review of the report ten years later, Harwood (1993) noted, The feature . . . that most distinguishes it from other USDA reports of the time is that it dealt with an amorphous package of practices that constitute a truly indigenous knowledge system, as opposed to the commonly studied systems found in scientific literature . . . . this knowledge is not conceptualized the way a scientific knowledge system is. Its underlying principles are not stated in scientific terms, and biological relationships commonly are described with phrases that suggest human characteristics, such as the need to focus on ‘soil health’ (p. 150). In defining organic agriculture, the Bergland report included a section that Harwood observed later framed much of SARE'S biological portion. To the maximum extent feasible, organic farming systems rely upon crop rotations, crop residues, animal manures, legumes, green manures, off-farrn organic wastes, mechanical cultivation, mineral-bearing rocks, and aspects of biological control to maintain soil productivity and tilth, to supply plant nutrients, and to control insects, weeds and other pests (United States Department of Agriculture, 1980, p. xii). Shortly after the Bergland report was issued, a change in federal administrations led to its rejection (Madden, 1991). Youngberg, Schaller and Merrigan (1993) observed that this rejection occurred despite the fact that the report’s definition of organic farming did not totally rule out the use of synthetic chemicals . . . . The proponents of low-chemical production techniques had seriously underestimated the negative symbolism of organic farming, which had long since been dismissed by conventional agriculture as little more than a primitive, backward, nonproductive, unscientific technology . . . . In reaction to this newly-perceived reality, advocates of organic agriculture mad a conscious effort to identify and promulgate new language, new words, to describe the character and benefits of low-chemical agriculture (p. 198). Jackson is credited by many to have first used the term “sustainable agriculture” in 1978, to refer to a farming system based on resource conservation and quality of life in rural areas (ATTRA, 1999). Harwood (1990) observed that the term gained increased use beginning in the late 1980's. 33 Michigan State University hosted an international conference in 1984 titled “Sustainable Agriculture and Integrated Farming Systems.” In his conference paper, Douglass (1985) outlined three ways of thinking about sustainability. His “productivity” school included people most concerned about sustainability as providing adequate food; his “stewardship” school regarded sustainability primarily as an ecological phenomenon; and his “community” school was most concerned with effects of various agricultural systems on rural life (p. 10). The tensions among these three approaches have been threaded through the continuing discussions and debate for fifieen years about how to define and measure sustainability. Through the decade of the 1980's, concerns with various aspects of conventional agriculture intensified. The 1985 farm bill included support for research and education on farming practices to conserve resources and protect the environment through its Subtitle C, Title XIV, “Agricultural Productivity Research.” The program authorized by this provision was identified as Low-Input Sustainable Agriculture (LISA) (United States Congress, 1985). When Congress appropriated funds in 1988, the program was launched. At about the same time, the National Research Council’s (NRC) 1989 report, Alternative Agriculture, summarized scientific knowledge as of the mid to late 1980's, regarding topics including tillage, biological pest control, legumes as a nitrogen source, etc. It also included a series of case studies examining 14 farms across the US. One of the NRC’s conclusions was, Alternative farming practices are not a well-defined set of practices or management techniques. Rather, they are a range of technological and management options used on farms striving to reduce costs, protect health and environmental quality, and enhance beneficial biological interactions and natural processes (p. 8). 34 Following considerable debate about the definition and goals of sustainable agriculture (Youngberg, Schaller & Merrigan, 1993), the 1990 farm bill included a sustainable agriculture research section comprising three chapters that continued and expanded LISA and renamed it Sustainable Agriculture Research and Education (National Research Council, 1991). In 1991, Sustainable Agriculture Research and Education in the Field: A Proceedings, shared contents of an April 1990 workshop that had provided a forum for discussing field results from inception of the LISA program. The introduction indicated that, “beginning in 1989, a broad cross-section of people has grown comfortable with the term sustainable agriculture” (NRC, 1991, p. 2). Defining Sustainable Agriculture There is no shortage of definitions of sustainable agriculture nor of varying perspectives on the subject. Ikerd (1992) observed that, Sustainability is a goal or end that can be, and is being, pursued through a variety of strategies or means. However the concept of sustainable agriculture has become associated with a specific alternative to the industrial model or paradigm of farming. This alternative model is dynamic, integrated, site-specific and individualistic in nature. The sustainable model of a farm is analogous to that of a living organism, while the industrial model is more like a machine or a factory . . . . Sustainability is not a characteristic of farming practices, methods, or enterprises but, rather, is a characteristic of whole-farm systems (p. 44). Ikerd also pointed out some of the challenges inherent in trying to move from an industrial model to a sustainable model. The industrial mode assumes that the important problems of agriculture are common among large numbers of farmers and that such problems can be clearly identified, precisely defined, and universally solved. The sustainable agriculture model is holistic, dynamic, site-specific, and individualistic. The problems and solutions are complex and constantly changing (p. 44). 35 Madden (1998), in reflecting on the early years of SARE, recalled the definition debates. Since reductionist science requires clear and unambiguous definitions, it was both inevitable and appropriate that scientists require a definition of the subject at hand. And while attention to definitions can be healthy, I observed that many of those calling for a definition were demanding a bifurcation of all agricultural practices into two distinct categories, one called sustainable and the other non- sustainable or some other characterization (p. 18). He concluded that the demand for adherence to a rigorous definition characterized people with a philosophy that values reductionist research and that comfort in proceeding with “creative ambiguity” characterized those who focus more on solving problems in the real world. Ikerd commented that there is general consensus among proponents of sustainable agriculture that their goal is to sustain agriculture for the benefit of people in both current and all future generations (Ikerd 1996a, p. 1). He reminded us that, “we cannot prove through empirical studies that one approach to agriculture is sustainable or that another is not." Lockeretz (1990) suggested that the concept of sustainable agriculture has evolved at least in part because “a broader range of people have become interested in the same goals for diverse reasons” (p. 423). Batie and Taylor (1991) diStinguished between agricultural systems viewed as belief systems and agricultural systems viewed from a technological perspective. Alternative agriculture as a belief system involves a holistic philosophy, and it incorporates alternative values to those of conventional agriculture . . . . (p. 184). The dichotomy between conventional and alternative agriculture serves as a proxy for the dichotomy between a reductionist-science based, chemical-intensive agriculture and that of a holistic-science based on community and environment- enhancing agriculture (p. 185) 36 But, they maintained, “it is not necessary to embrace an alternative holistic belief system to adopt alternative agriculture practices” (p. 185). If one views alternative agriculture as an alternative set of technologies compared to the set used by conventional agriculture, then terms such as low-input, organic, sustainable and conventional each represent a particular set of technologies. Richardson (1994) reflected perspectives from his work in holistic resource management when he said that sustainability is the result of an endless series of decisions. It is not a state to reach, but a dynamic condition to maintain. Ikerd (1996a) wrote that though most agree that three key aspects of sustainability are economic, environmental and social, people view sustainability through different mental models or paradigms, so they can't agree on a single definition. He contrasted the paradigms traditionally associated with each of these three aspects. The traditional economic paradigm has clearly defined boundaries . . . . The natural environment and natural resource base are considered to be “external,” or out of bounds, by economists. Society likewise is considered to be an “external” factor . . . . Ecologic boundaries place economic and social issues outside of the domain of scientific inquiry . . . . The traditional paradigm of sociology deals primarily with relationships among people. Economic and natural environments represent the contexts within which people carry out social interactions and, thus, are “outside” the realm of specific social inquiry. Although there is substantial agreement that a sustainable agriculture must be ecologically sound, there may be less agreement about the other two dimensions (Ikerd 1996a). One rationale for the economic dimension is, enterprises that lack economic viability will lose control over use of ecologic resources to their economically viable competitors . . . . Human societies that lack economic equity and social justice are inherently unstable, and thus, are not sustainable over time . . . . We must have social incentives to create economic rewards for ecological protection" (Ikerd 1996, p. 2). Some practitioners have struggled to develop a new paradigm of sustainability 37 that is fundamentally different than the past because it clearly considers ecology, economics, and sociology, all three, to represent different dimensions of a single holistic, systems approach to scientific inquiry. Environmental, economic, and social impacts are all to be considered “within,” rather than “outside” of the boundaries of agroecosystems “managed” for sustainability. The constraints or boundaries within which sustainable agroecosystems fimction are the laws of nature, including human nature. The new paradigm considers economic, ecologic and social dimensions of sustainability to be inseparable aspects of the same whole . . . . The challenge is to comprehend the complexities of wholes rather than attempt to reduce wholes to more simple and easily understood elements (Ikerd 1996b). System Levels and Processes One essential aspect of sustainable agriculture is its systems perspective. According to Edwards (1987), Many farmers and even agricultural scientists view the various practices they use or develop as completely independent of one another . . . . A farming system is not just a simple sum of all of its components but rather a complex system with intricate interactions (p. 150). He went on to say that, in conventional “higher-input” farming, high yields can be obtained without appreciable attention to interactions . . . . However, as chemical inputs are lowered progressively, so the need for attention to the mechanism by which one input impacts upon another increases. Thus, the need for integrated farming systems increases (p. 150). Edwards viewed farm economics, cultivation, fertilization, crop protection and crop rotations as the major interactive components in farming systems, and he provided examples of interactions among these components. Thompson (1995) also advocated using a systems approach for sustainability. He proposed that a sustainable system is one with few, if any, internal threats. He went on to warn that, 38 The deep philosophical problem with sustainability is that we need two very different kinds of criteria for knowing how to use the concept. The first kind should tell us when the word has been used accurately in describing the object or system under analysis . . . . The second kind tells us when the ends in view have met the normative criteria that make sustainability an ethically significant goal (p. 153) He went on to warn of the confusion likely when the system describing or descriptive meaning of sustainability is inadvertently mixed with its goal prescribing or prescriptive meaning. Thompson pointed out that the systems perspective also prevents describing particular production technologies, social arrangements or other human practices as sustainable or unsustainable in isolation. “One examines a practice within a system context and then asks whether the total system is sustainable, presuming that what happens outside system borders remains stable” (Thompson, 1999). And of course, deciding upon the borders to be used for a particular system analysis always involves value judgements. Another issue is one of system levels. Lowrance, Hendrix and Odum (1986) proposed that “different constraints operate at different levels of organization and that management strategies for sustainability must be applied at the appropriate level” (p. 169). They also reminded us that changes at one level affect other levels of the hierarchy, and suggested that, within the hierarchy of agricultural systems, sustainability can best be addressed by recognizing the dominance of agronomic constraints at the field scale, microeconomic constraints at the farm scale, ecological constraints at the watershed or landscape scale, and macroeconomic constraints at the national or transnational level (p. 170). Waltner-Toews (1994) discussed the range of ways in which people conceptualize and understand sustainable agriculture. Some of the differences in understanding of agricultural sustainability are founded 39 on differences of values . . . . Other differences . . . are matters of scale. When agricultural sustainability is viewed at the field or even the farm level, the focus is on specific management techniques which may decrease the negative aspects of agricultural activity on natural resources while increasing or at least maintaining production and profits . . . . The tendency to focus on fields and farms is founded on a love affair with technical, and reductionist rather than social, holistic solutions to our problems . . . . Studies at the field level are scientifically much easier to design and carry out, since we rarely need true, interdisciplinary research . . . . If conceived and implemented in isolation, solutions at this level are simplistic and, in the long run, ineffectual . . . . They ignore the broader ecological, social and institutional context of sustainability . . . . It is at the scale of watersheds and larger ecological and socio-political regions that questions of food self-sufficiency, distribution of risks and benefits . . . and the resilience, or integrity, of the natural infrastructure, take on greater importance (pp. 10-11). Vickery and Lohr (1997) catalogued approaches to assessing agricultural sustainability. They developed a framework to classify assessment methods by whether they were designed for: (a) crop specific; (b) field-level; (c) farm-level; or (d) environmental impact. They further classified each method as qualitative or quantitative. They indicated that crop-specific tools emphasize productivity, often evaluate individual practices rather than systems, and are frequently associated with marketing programs. Field level assessments tend to be more generic, set similar criteria across crops, have more rigid criteria for sustainability, and are often employed to certify production practices for marketing. Farm level assessments incorporate components into a system assessment, may include crop-specific criteria, and set system goals that may allow flexibility in inputs or practices. Environmental impact assessments examine actual or potential damage from one or more practices. White, Braden and Hombaker (1994) saw sustainability “as a general direction in which to head, rather than a fully defined objective” (p. 242). They characterized progress within a framework of gradual evolution as occurring within three overlapping levels of progress. Efficiency changes are first and easiest, followed by substitution 4o changes and finally fundamental system redesign. Efforts to increase production efficiency are consistent with standard management techniques — with reduced environmental impacts and resource demands as coincidental benefits . . . . Maximizing efficiency can be a dead-end path in relation to sustainability, if the system being made more efficient is basically unsustainable . . . . Substitution changes . . . can produce improvements, but, as with efficiency changes, having too narrow a perspective when electing to make such a change can result in its being misdirected or short-sighted . . . . Compared to substitution, redesign changes are more fundamental and involve whole production systems. In terms of agriculture, the emphasis would be on production systems, perhaps farms, as a whole —— unique in time and space. The systems approach offers an opportunity to simultaneously pursue complex goals such as ecological and economic vitality, self-sufficiency, and diversity . . . . Redesign changes would likely also incorporate efficiency and substitution changes (p. 243-244). Stockle, Papendick, Saxton, Campbell and van Evert (1994) suggested that one approach to evaluating sustainability is to identify clearly unsustainable systems, and then develop alternatives that appear — based on our current knowledge — to be more sustainable (p. 46). To operationalize this approach, they developed a list of attributes of sustainability or assessment elements. For each attribute, measurable constraints were identified. A system is evaluated by assigning weights to each attribute, scoring the attributes of the proposed system based on specific constraints . . . , and then combining the weights and scores to produce a figure of merit . . . . This approach does not allow us to determine the sustainability of the farming system in absolute terms. Instead, it compares the relative sustainability of different systems (p. 46). Although the authors specified that, “any constraints chosen for the evaluation scheme must be amenable to numeric definition,” they conceded that direct measurement may not be feasible in many situations and they suggested simulation modeling and expert opinion as reasonable options (p. 47). 41 Dimensions of sustainability Padgitt and Petrelka (1994) suggested various dimensions that define and differentiate subsistence agriculture, commercial agriculture and sustainable agriculture and summarized their comparisons (p. 269) in Table 2-2. Table 2-2: Agricultural System Dimensions Defining Subsistence Commercial Sustainable Dimension Social Family Self Community Identity World of Past Present Future Reality Major Conflict Competition Cooperation Interpersonal Processes Nature of Uncontrolled & Planned & Uncontrollable Change controlled anticipated Relationship Vulnerable to Control over Harmony with to Nature Interpersonal Mutual distrust Individual rights Community needs Relations Natural Finite & consume Develop & consume Finite, conserve Resources & preserve Motivational Safety & security Self achievement Community Drive accomplishment Role of State Undeveloped, unstable; Coordinate, protect Regulate meeting needs of those rights; laissez faire in power Knowledge Tradition Science & Science technology Base technology mediated/indigenous Technological Borrowed or Supported, faith as Controlled for Development Wsererflitous solution to problems collective good 42 They also suggested that “if farmers are not knowledgeable of or do not adhere to best management practices within the commercial framework, they are unlikely to modify or integrate these practices into a more sustainable framework” (p. 277). Haapala (1995) presented a sustainability index model that included several helpful concepts. The model defined threshold as “the percentage of adoption of positive practices required of a producer or a community to qualify as ‘sustainable’”; positive practices as “cultural practices, materials and methods that are commonly held to favorably impact the given criteria”; and criteria as “the set of outcome-based indicators impacted by agricultural practices.” The sustainability index as a whole was defined as “the set of weighted positive practices available to the producers within the region . . . ” Neave, Kirkwood and Dumanski (1995) investigated indicators for assessing the stability of agricultural land management in Canada. They identified five objectives that support sustainable land management: (a) productivity, (b) security, (c) conservation, ((1) economic viability and (e) social acceptability, and they asserted that all five objectives must be reached for a system to be considered sustainable. They then identified and described indicators in physical, agronomic, economic and social categories. They suggested that indicators from across the categories need to be considered simultaneously and integrated at the farming system level and that determining and using thresholds for indicators may help assess degree of sustainability. Herdt and Steiner (1995) viewed sustainability as the result of the relationship between technologies, inputs and management, used on a particular resource base within a given socioeconomic context . . . . One may consider systems across an infinite range of space: global, regional, farm field, individual plants, and microscopic . . . . Sustainability can only be thought of in the context of a defined time period . . . . Consideration of the time dimension is further complicated by the dynamic nature of reality . . . . In the real world 43 agricultural production systems are constantly changing . . . . Measuring sustainability is further complicated by the dimensions in which people think about the human condition. Plant growth is a biological process that results in physical changes, but agriculture is an economic activity serving a social purpose . . . . The biological . . . dimension can be reflected in the quantity of output, which depends on the physical quantity of inputs and the biological grth processes . . . . The economic dimension . . . in the value of output, . . . the social dimension . . . in the capacity of systems to adequately support farming communities . . . Total social factor productivity, which measures total output relative to total input for both managed inputs and externalities, may be an appropriate approach . . . . And ecosystem health measures reflect the quality of the resource base . . . This must be measured for a particular cropping system on a particular field (pp. 5 - 11). Some efforts to define agricultural sustainability clearly focus on a single dimension. For instance, Crews, Mohler and Power (1991) argued that “sustainability is a measure of a system's potential to endure and is not the proper yardstick with which to measure the desirability of a particular set of social relations” (p. 148). They also asserted that, “the profitability of an agroecosystem is so tightly linked to the social structure of agriculture and ecological components of sustainability that it should not be considered a criterion in itself” (p. 148). Parr, Papendick, Homick and Mayer (1992) suggested that an index of soil quality be developed that could quantify various attributes of soil quality (soil properties, potential productivity, environmental factors, human and animal health, erodibihty, biological diversity, food quality/safety, management inputs) and then derive an index or indices usable for simulation and prediction. Further, the authors provided a conceptual diagram in which these attributes of soil quality serve as the link between strategies of alternative agriculture (skilled management, crop rotations, organic recycling, reduced chemical input, crop/livestock systems, and integrated pest management) and the goals of sustainable agriculture (productivity, profitability, energy conservation, environmental 44 soundness, economic viability, conservation of natural resources, improved health, food quality and food safety). Papendick (1993), further supported the importance of soil quality, noting that, The definition of soil quality implies that soil must perform three functions, which are concerned respectively with productivity, the environment and health. Notwithstanding the many other dimensions of sustainability — economic, cultural, social, and so forth — if the soil's quality cannot be sustained in a given production system, agriculture cannot be sustained and the other issues become academic (p. 155). Lockeretz (1989) looked at studies comparing economics of higher versus lower input crop systems and extended the analyses to estimate contributions to the local economy from each production system. He referred to this as “changing the boundary for calculating net return. . . . The boundary now becomes the local economy, rather than the farm” (p. 79). Lockeretz found that, “the total value retained locally was generally lower with the alternative system” (p. 80). However, he also speculated that, the results might have been different if calculated over a longer time, because he found that conventional systems contribute less to the local economy than alternative systems as a fraction of the value of production, although their contribution is higher in absolute terms ($/A). This means that while a larger amount of money remains in the local economy with conventional systems, a larger amount also leaves it, both absolutely and as a fraction of total production (p. 81). Strange (1984) suggested several economic characteristics of sustainable agriculture that might be considered indicators. 1. Farms are family centered. The farm is both a place of work and a home . . . . Learning to farm is a matter of apprenticeship. . . . Skills, values, and success are the human 'wealth' of the farmer and the inheritance of the farm child . . . . 2. In our society, a sustainable agriculture should also be owner operated . . . . Implicit in a system that is owner operated is a stricture against owning more than one can personally farm and be responsible for . . . . 3. If agriculture is owner operated without being hereditary, it is because farms 45 are internally financed. This means simply that farm production expenses, including the cost of land, which each generation must pay, are paid out of farm earnings . . . . The use of credit to finance the purchase of fixed assets, especially land, when the purchase is based on expected increases in their value, is speculation . . . . 4. Internally financed, owner operated farms can function in a market economy only if markets are open. In an open market, no farm, by reason of its size, can have a competitive advantage over other farms, either in the price it receives for products, the opportunity it has to sell those products, or the terms under which those products are sold (pp. 118-120). Levins (1996) proposed financial indicators for sustainable agriculture at the farm level. One indicator, the amount of government payments as a percent of gross income, “measures the extent to which a farm is indebted to the taxpayers for its survival. The indicator can also be useful in helping the public see what kind of farming they are choosing to favor with Special payments” (p. 4). A second, energy and machinery as percent of gross income, shows how willing a farm is to share its income with non-farm corporations. Expenses accounted for by chemicals, commercial fertilizers, and gas guzzling equipment are also a measure of how a farm is interacting with the environment. When measuring sustainable agriculture, the rule here must be ‘the less, the better’ (p. 5). Support for local families as a percent of gross income, feed production and use balance, and profit or loss from farming are the remaining indicators. Levins calculated each indicator from information already available on federal tax returns. Kriegl (1999) compared grazing and conventional dairying and suggested several points for comparison that can be considered benchmarks or indicators. He pointed out the importance of understanding the operations being compared well enough to determine why a particular benchmark value may deviate from what is expected. Financial or economic indicators he identified included rate of return on assets, rate of return on 46 equity, net farm income from operations per cow, investment per cow, debt per cow, basic cost per hundredweight of milk. Physical performance indicators were average herd size and pounds of milk sold per cow. Also noted were some important ways that these indicators interact. Salamon, F amsworth, Bullock and Yusuf (l 997) investigated family factors that may affect adoption of sustainable farming systems. The inductively identified social characteristics they identified as associated with adoption included family innovative traditions, an environmental or health trigger, systemic on-farm experimentation, and prudence with resources (p. 265). Herdt and Steiner (1995) asserted that sustainability is the result of the relationship between technologies, inputs and management, used on a particular resource base within a given socioeconomic context. Careful recognition of three aspects of systems — space, time, and . . . dimension—help make the discussion concrete (p. 5). Sustainable Agriculture Research and Education Program The Sustainable Agriculture Research and Education program (SARE) was first initiated as the Low-Input Sustainable Agriculture (LISA) program in 1988, after authorization in the Food Security Act of 1985. The Food, Agriculture, Conservation and Trade Act of 1990 changed the name to Sustainable Agriculture Research and Education. The program is administered through the USDA Cooperative State Research, Education and Extension Service (CSREES). In the 1990 Farm Bill, Congress defined sustainable agriculture as an integrated system of plant and animal production practices having a site- specific application that will, over the long tem, satisfy human food and fiber needs; enhance environmental quality and the natural resource base upon which 47 the agriculture economy depends; make the most efficient use of non-renewable resources and on-farm resources and integrate, where appropriate, natural biological cycles and controls; sustain the economic viability of farm operations; and enhance the quality of life for farmers and society as a whole (Food, Agriculture, Conservation, and Trade Act of 1990, Public Law 101-624, Title XVI, Subtitle A, Section 1603). Subtitle B of Title XVI authorized research and education programs in sustainable agriculture under three chapters, two of which have been funded. Chapter 1, Best Utilization of Biological Applications, authorized conduct of research and extension projects to obtain data, develop conclusions, demonstrate technologies, and conduct extension programs that . . . ‘ .... (1) facilitate and increase scientific investigation and education in order (to): (A) reduce . . . use of chemical pesticides, fertilizers, and toxic natural materials in agricultural production; (B) improve low-input farm management to enhance agricultural productivity, profitability, and competitiveness; and (C) promote crop, livestock, and enterprise diversification; and (2) facilitate the conduct of projects in order to: (A) study . . . agricultural production systems that are located in areas that possess various soil, climate, and physical characteristics. (B) study farms . . . managed using . . . low input and conservation practices; (C) take advantage of the experience and expertise of farmers and ranchers through their direct participation and leadership in projects; (D) transfer practical, reliable and timely information to farmers and ranchers concerning low-input sustainable farming practices and systems; and (E) promote a partnership between farmers, nonprofit organizations, agribusiness, and public and private research and extension institutions (Food, Agriculture, Conservation, and Trade Act of 1990, Public Law 101- 624, Title XVI, Subtitle A, Chapter 1, Section 1621). Subsequent sections dealt with program administration (Section 1622), establishment of a grant program (Section 1623), and authorization (Section 1624). Chapter 2, Integrated Management Systems, has not been funded to date. Chapter 3, the Sustainable Agriculture Technology Development and Transfer 48 Program, mandated development of specific training and education activities designed to facilitate adoption of sustainable agriculture practices and systems. Components were to include: (a) development and distribution of technical guides and handbooks on sustainable agriculture; (b) establishment of a national training program in sustainable agriculture; (c) designation of regional training centers for Extension and NRCS specialists and others with need for intensive sustainable agriculture training; ((1) establishment of a competitive grants program to award funding for basic training for all other agricultural agents; and (e) designation of state coordinators and specialists in sustainable agriculture to coordinate training program participation, outreach and local dissemination of sustainable agriculture information (Food, Agriculture, Conservation, and Trade Act of 1990, Public Law 101-624, Title XVI, Subtitle A, Chapter 3, Sections 1628-9). In addition to the Congressional legislation that authorized and appropriated funding for SARE, there has been one review of the program by the federal government. At the request of Congressman Fred Grandy, the Government Accounting Office undertook a study to 1) describe the US. Department of Agriculture’s (USDA) efforts to encourage sustainable agriculture, 2) specifically report on the Department’s Sustainable Agriculture Research and Education (SARE) Program, and, if appropriate, 3) recommend actions to improve both (GAO, 1992, p. 2). One of the report’s recommendations was, To increase the impact of the SARE Program, improve its effectiveness, and help ensure its integrity, we recommend that the Secretary of Agriculture direct SARE Program management to establish 1) guidance and systems to collect, evaluate, synthesize, and report the results of SARE research projects at regional and national levels and 2) national standards for regional monitoring of SARE projects (GAO, 1992, p. 42). 49 Another review of SARE and its research results was undertaken by Holt (1992). with a different focus. He presented Congressional testimony that contrasted the linear model of research conducted with support from the National Research Initiative with the parallel model of research conducted with support from SARE. One key difference he observed is that traditional agriculture competitive grant programs typically address strategic questions by specifying the subject matter or disciplines that are to be included, not by describing expected outcomes. SARE, however states desired outcomes more in terms of desired changes in agricultural systems (p. 7). Although Holt argued strongly for the value of the parallel mode, he also Observed that the North Central Region (N CR) SARE could improve its process. SARE program administrators have not gone so far as to insist that investigators identify a quantitative practical goal and a variable by which progress toward achieving the goal can be measured . . . . Until program managers and participants can take that step and make that substantial commitment, the linkage between publicly supported research and education programs and the mission and objectives of agriculture will remain somewhat tenuous (Holt, 1992, p. 12). 50 CHAPTER 3 METHODS AND PROCEDURES Introduction This research was qualitative and holistic. It sought to clarify NCR SARE stakeholder perspectives regarding evaluation issues and give voice to the stakeholders. It did this through a process of documenting and trying to understand and interpret the meaning of their responses to a series of three Delphi surveys. The research was designed to maintain participants’ individuality and to allow their ideas and concerns to inform ongoing discourse and decisions within NCR SARE specifically and the sustainable agriculture community in general. Research Questions The research was designed to help answer the question of whether it is possible to construct a meaningful participatory framework to allow stakeholders to be actively involved in defining evaluation issues and the problems associated with those issues. Within this overall question, several subsidiary questions were also addressed: A. Are there clearly recognizable issues regarding evaluation in sustainable agriculture? Al. If so, what are the issues? A2. Do evaluation issues in sustainable agriculture clearly group in major themes? B. Are there clearly definable problems associated with sustainable agriculture 51 evaluation issues? Bl. What are the problems? B2. Is there consensus regarding the problems associated with sustainable agriculture evaluation issues? B3. Are there differences in the types of problems associated with each sustainable agriculture evaluation issue? C. What are the suggestions and recommendations? Cl. Is there consensus regarding the suggestions and recommendations associated with sustainable agriculture evaluation issues? C2. Are there differences in the types of suggestions and recommendations associated with sustainable agriculture issues? Population and Sample Population Description The population for this research was North Central Region Sustainable Agriculture Research & Extension program stakeholders. The operational definition for stakeholders was all persons from the following groups: (a) current and past members of the NCR SARE Administrative Council; (b) current and past members of the NCR SARE Technical Committee; (c) current NCR SARE paid staff; ((1) current NCR SARE state PDP Coordinators; (e) current and past recipients of NCR SARE Research & Education grants; (1) current and past recipients of NCR SARE Professional Development Program grants; and (g) current and past recipients of NCR SARE Producer Grant Program grants Member lists for each of these groups were provided by the NCR SARE office in 52 Lincoln, Nebraska. These lists totaled 431 names, but that number included many duplicates, resulting from instances in which someone served on the Technical Committee and later on the Administrative Council, someone received both a research and education grant and a professional development program grant, and similar situations. Sampling Procedure These lists of NCR stakeholders were organized into one document. The individual lists were maintained within that document. A decision was made not to combine lists or eliminate duplicate listing of names, but to keep the lists as provided, so that nominators could see the way or ways in which each stakeholder was involved with NCR SARE. Copies of the stakeholder lists were provided to a group ofNCR SARE leaders assembled at a May 1999 meeting to guide this research. Members of that group were asked to circle names on the lists of people whom they believed would provide valuable perspectives on NCR SARE evaluation efforts. No limit was placed on the number of people they could nominate. Their lists were collected and their nominations compiled. Table 3-1 Shows the total number of individuals in each stakeholder group as well as the number from each stakeholder group nominated to participate in the research. When the lists of nominees from each stakeholder group were combined and duplicates removed, 232 names remained. That number was still too large for the planned research methodology. To reduce the number, nominees were sorted by state. The stakeholder group(s) that each individual represented was recorded in this sort, as was the number of times each individual had been nominated. All nominees with addresses outside the North Central Region were removed. Nominees who had received 53 Table 3—1: NCR SARE Stakeholder Groups and Nominations Stakeholder Groups Number in Number Group Nominated Current and past members of NCR SARE 88 64 Administrative Council Current and past members of NCR SARE 100 67 Technical Committee Current NCR SARE paid staff 5 2 Current NCR SARE state Professional 19 12 Development Program Coordinators Current and past recipients of NCR SARE 152 60 Research & Education grants Current and past recipients of NCR SARE 51 31 Professional Development Program grants Current and past recipients of NCR SARE 16 54 Producer Grant Program grants Totals 431 290 results of their projects. The list was reduced to 144 names. Table 3-2 shows how these individuals were 54 only one nomination were removed, except in situations where the removal would unduly minimize state stakeholder group representation among invitees. Efforts were made to balance the overall list, both with respect to the number of people to be invited to participate from each state and with respect to the various stakeholder sub-groups. Another factor in the sorting was that just one year earlier, all of the Producer Grant Program recipients had been surveyed about their experiences with the program and the distributed in terms of their state and their NCR SARE stakeholder roles. The top number in each cell represents the number of people from the state and role invited to participate. Although these numbers represent just 144 individuals, they sum to 193 because some of the individuals belonged to multiple stakeholder groups and were counted in each cell representing a group to which they belong. Invitations were mailed to these 144 NCR SARE stakeholders. Appendix A contains a copy of the letter inviting people to participate in the study plus the response card they were requested to complete and return. Eighty-five people responded positively to the invitation to participate. The bottom number in each cell of Table 3-2 shows their distribution by state and NCR SARE stakeholder groups. The total of 119 actually represents 85 individuals, again due to some individuals having played multiple roles within NCR SARE. Table 3-3 shows the number of individuals from each State who were invited to participate in this study and the number who agreed to participate. The numbers in this table represent people rather than stakeholder roles relative to NCR SARE. 55 Table 3-2: NCR SARE Stakeholders Invited/Agreeing to Participate: Distribution by State & Stakeholder Groups Group IA IL IN KS MI MN MO ND NE OH SD M Tot AC 3 5 1 7 3 3 2 6 3 3 4 44 2 3 3 1 6 2 1 2 4 2 2 3 31 TC 5 4 4 2 4 3 2 2 5 4 2 4 41 4 1 4 1 2 2 1 o 3 2 2 3 25 Staff 0 0 0 0 o o o 4 0 o o 5 o o o o o o o 3 o o o 4 State 2 1 2 1 2 1 1 1 15 1 0 1 1 1 1 o o 9 R&E 3 3 3 4 5 4 1 2 5 2 2 6 40 2 1 3 1 4 1 1 1 5 2 2 2 25 PDP 2 2 2 1 2 1 2 3 3 2 2 2 24 1 1 1 g 1 2 1 1 2 2 2 2 2 13 PGP 3 3 2 2 5 2 3 o 1 1 1 24 1 o 1 2 1 o 0 o o 1 o 7 Note: AC=current & past Administrative Council members; TC= current and past Technical Committee members; Staff=current staff members; State=current NCR SARE state PDP coordinators; R&E=current & past research and education grant recipients; PDP=current and past professional development program grant recipients; PGP=current and past producer grant program grant recipients. In this table, one individual may be counted in more than one stakeholder group. The top number in each cell is the number invited to participate; the bottom number is the number who agreed to participate. 56 Table 3-3: Individuals Invited to and Agreeing to Participate in Study State Number Number Invited Agreeing to Participate IA 1 l 7 IL 1 6 7 IN 14 1 0 KS 9 5 MI 1 9 14 MN 10 3 MO 1 l 3 ND 8 5 NE 14 1 0 OH 1 0 7 SD 8 7 WI 14 7 Total 144 85 Instrumentation & Data Collection The Delphi method was used as the research methodology. Three survey questionnaires in three separate rounds were mailed to each participant in the study. Potential participants were nominated by key NCR SARE leaders from a list of all current and past NCR SARE stakeholders. Adjustments were made to achieve reasonable representation across the 12 NCR states and among groups of farmers, university faculty and staff, and others. Each of 144 nominees was invited by letter to participate. A return postcard was included for their response. Eighty-five people agreed to participate and 57 Instrumentation and Data Collection The Delphi method was used as the research methodology. Three survey questionnaires in three separate rounds were mailed to each participant in the study. Potential participants were nominated by key NCR SARE leaders from a list of all current and past NCR SARE stakeholders. Adjustments were made to achieve reasonable representation across the 12 North Central Region states and among groups of farmers, university faculty and staff, and others. Each of the 144 nominees was invited by letter to participate. A return postcard was included for their response. Eighty-five people agreed to participate and received the three survey rounds. Respondents were provided with a stamped, self-addressed envelope with each round to return their responses. They were also provided information to allow fax or e-mail responses. The questions for this research comprised the first portion of each survey round instrument. The second part of each instrument comprised questions about SARE evaluation criteria, which were designed for a companion NCR SARE evaluation framework project. Appendices B, D, and F provide copies of the survey forms in their entirety. Only results from the “Evaluation Issues” portions of the surveys are reported here. Round One Round One questions included two open-ended questions requesting participants’ descriptions of sustainable agriculture evaluation issues and potential evaluation criteria. The survey was mailed to research participants. A self-addressed, stamped envelope was included and a fax number provided for returning surveys. 58 All responses from the round were recorded. Responses were then reviewed to identify key ideas, regularities and patterns. Then responses were sorted to form Issue/Question Clusters. These Issue/Question Clusters, without ranking or weighting, formed the basis of the Round Two questionnaire. Round Two The Round Two survey incorporated Round One responses. It asked respondents to select the two issue/question clusters they believed would be most important for NCR SARE evaluation in the next five years. It also asked respondents to identify and describe problems that might be associated with each of the issue/question clusters that they selected. The Round Two survey was mailed to research participants with self-addressed stamped envelopes included and a fax number provided for returning surveys. In Round Two data analysis, the number of times each issue/question cluster was selected was tallied. The potential problems identified for each issue were recorded. The lists of problems were reviewed to identify key ideas. The lists were then summarized for inclusion in the Round Three questionnaire. Round Three The Round Three survey incorporated responses from Rounds One and Two. It presented the list of issue/question clusters plus a summary of the problems identified for each cluster. Respondents were asked to recommend ways that NCR SARE could deal with the evaluation problems identified. The Round Three survey was mailed to research participants with self-addressed stamped envelopes included and a fax number provided 59 In R: It’f SU 551 D: ‘ Ill! for returning surveys. All Round Three responses were recorded, then reviewed to identify key ideas. regularities and patterns. Responses were then sorted into recommendation themes. Response Rates and Patterns An identification code was assigned to each individual invited to participate in the research project. This allowed tracking responses by state and by stakeholder group. Table 3-3 shows which respondents responded to each Delphi Round. Each subsequent Round generated fewer responses; there were 53 first round responses, 43 second round responses and 32 third round responses. That pattern is fairly typical for Delphi surveys. Sixty-seven people responded to at least one of the three rounds, 21 of them to all three rounds. 60 Table 3-4: Stakeholder Participation in Delphi Survey Rounds Respondent Round 1 Round 2 Round 3 All Any Rounds Round IAOI x x x x x IA02 x x IA03 x x IA05 x x x x x IAO9 x x IL02 x x IL08 x x IL12 x x x IL13 x x x IL14 x x x IN03 x x IN07 x x x x x IN08 x x IN09 x x x [N] 0 x x IN 11 x x x KS? x x KSOl x x x x x KSO4 x x KSO8 x x x KSlO x x x Table 3-4 continued on next page Table 3-4 continued 61 Respondent Round] Round 2 Round 3 All Any Rounds Round MIOl x M102 x M103 M105 M106 M107 M110 M112 M113 M115 M117 M119 MNOS MN06 MN 10 M007 M008 ND? ND02 ND03 NDOS ND07 x Table 3-4 continued on next page 62 Table 3-4 continued Respondent Round 1 Round 2 Round 3 All Rounds NE01 NE04 NE05 NE07 NE09 NE10 NEll NE12 OH01 OHOS OH06 OH09 OHIO SD01 SD02 SD04 SD06 SD07 SD08 W106 W107 W108 W112 W114 Totals 53 43 32 21 67 63 Data Analysis The approach for data analysis was inductive. The qualitative and iterative nature of this research meant that data collection and data analysis were largely simultaneous and interactive. Delphi Rounds Two and Three both required analysis of the earlier Rounds for their completion. With each Round, the investigator sified, examined, combined, reduced and interpreted data as it was collected. A form of content analysis was used to analyze survey responses. The data were and reviewed repeatedly to look for regularities, patterns and major ideas within particular categories and across all categories. As the investigator reviewed the data, She began to cluster the responses -- to sort and group together responses that appeared similar and to assign a name or code to each group. Round One All Round One responses from the round were recorded. Responses were then reviewed to identify key ideas, regularities and patterns. Then responses were sorted into Issue/Question Clusters -- groupings that combined responses with similar ideas. Statements within each group were edited to retain ideas but reduce text length. The Issue/Question Clusters were not ranked, ordered or weighted. They formed the basis of the Round Two questionnaire. Round Two All Round Two responses were recorded. The number of times each issue/question cluster was selected was tallied and the Issue/Question Clusters were 64 ranked based on the tally. The potential problems identified for each issue were grouped according to the issue to which they responded. The problem lists were reviewed to identify key ideas. Statements within each list were edited to retain major ideas but reduce text length and were included in the Round Three questionnaire. Round Three All Round Three responses were recorded. Responses were reviewed to look for patterns and regularities. Then, tentative codes were assigned to indicate the perspectives that respondents expressed. Each response could have no code assigned, one code assigned or more than one code assigned, depending on the ideas and concepts within that response. Codes represented a combination of terms found within the written responses that seemed to capture a theme and terms provided by the researcher to capture an observed pattern. Within each issue area, responses were then grouped by code. In cases where one response carried more than one code, that response was listed within each coding group that applied. Only the portion of the response that related to a particular code was included. 65 CHAPTER 4 FINDINGS Introduction This chapter presents the findings from each of the three Delphi Rounds conducted. The pressing Issue/Question Clusters identified in Round One are listed and described. The order of importance of the Issue/Question Clusters established in Round Two is presented, along with a summary of the problems identified with each Issue/Question Cluster. Recommendation Themes, which were generated in Round Three, are displayed. Because an important focus of this research was to give voice to NCR SARE stakeholders, each description includes one or more direct quotes from selected respondents. Round One: Identifying Issues/Questions The Round One survey instrument was sent to 85 people. Appendix B contains a copy of the survey instrument. Its purpose was to ask each respondent to identify two pressing issues or questions they considered important when evaluating sustainable agriculture. After six weeks, responses were received from 53 individuals, with each respondent identifying two pressing issues or questions. Round One responses are shown in their entirety in Appendix C. After analysis, Round One responses were sorted into clusters based on similarity of the issues and questions they included. Table 4-1 displays the resulting 14 pressing Issue/Question Clusters with their identification letters. No attempt was made at this 66 point to quantify or prioritize the issues or to place them in any particular order. Table 4-1: Pressing Issues/Questions Identified in Round One Identifier Issue/Question Cluster A Who is currently involved? Diversity of approaches, practices and systems Long-term sustainability Empower people Whole systems questions Useful information Greatest barriers and critical components mammcnw Specific, measurable impacts I Sustainable agriculture practices Measuring sustainability Practical Grounded in research New way of producing, consuming and being ZZF‘W" Outside evaluator The Issue/Question Clusters identified from Round One responses follow. Issue/Question Cluster A: Who is Currently Involved? Issue/Question Cluster A involved examining who is currently involved in NCR SARE programs and whether they are the “right” people. Does/should SARE focus on small to moderate-size family-operated farms? To what extent is SARE relevant to mainstream agriculture? To what extent is SARE helping to develop people who can lead change? To what extent is SARE helping to link farmers, researchers and 67 extensionists in meaningful ways? Who have been the primary audiences for the research and education efforts? Who benefits most/ Who loses? How have these audiences been involved in each stage of the research and education effort? (WIO8) Issue/Question Cluster B: Diversity of Approaches, Practices and Systems. Issue/Question Cluster B involved the extent to which NCR SARE included adequate diversity of approaches, practices and systems. Concerns were expressed about a narrow, limited approach. Whether a diversity of approaches, practices and systems are being included . . . . My concern is that we not get caught in a too-narrow definition of “sustainable” agriculture, such that we end up with only one way to do the “alternative” to conventional agriculture. . . . The inclusion of a diversity of approaches would also take into account the diversity of agriculture, climate and social structure across the region. (IN 1 1) Issue/Question Cluster C: Long-Term Sustainability. Issue/Question Cluster C involved ideas about the need to address sustainability over the long term. Respondents noted the discrepancy of SARE’S one or two year projects compared to the time needed to determine project effects on long-term sustainability. Sustainable agriculture means an agriculture that endures over a long period of time. SARE research grants are for a period of 1-2 years. There is a discrepancy here, and Short term research results can logically be questioned as to their validity over the long term. (OH09) There is a need for longer term research projects that will lead to greater adoption and proven results. (W107) Issue/Question Cluster D: Empower People. 68 Issue/Question Cluster D involved SARE’s role in empowering people. Is SARE more about technology transfer or about empowering people? Evaluation of a program focused mostly on empowerment cannot be done in terms of actions and outcomes, but must be accomplished in terms of changes in understandings and abilities. Only the people SARE programs are designed to serve can tell us how well the programs are working. Does the program empower individuals who will model the desired change? (K801) Are farmers demanding alternatives? Are farmers asking for more than one opinion when seeking advice on an emerging production question? Are farmers requesting a “low input” alternative for pest control, fertility requirements, disease control? Have farmers begun to routinely question the rationale behind consultant recommendations? Are farmers starting to ask, “How can I get long-term control of this situation?” Are farmers developing farm plans that address their quality of life questions? (M117) The objective of SA programming is to empower people to develop and access the information they need to understand and solve their own problems. (M007) Issue/Question Cluster B: Whole Systems Questions. Issue/Question Cluster E involved holistic or whole systems approaches within SARE. Respondents mentioned the need for a range of research approaches, the importance of placing work in the appropriate context, and the importance of keeping all dimensions of sustainability in mind. Are we addressing a series of whole systems questions in projects we fund? (MNOS) Basic research is needed, but it is important that it be done in the context of the system in which the technology will be used. Basic and applied research and education on everything from cell-level research and genomics, to field level 69 questions need to be addressed. (MNOS) To what extent the issue takes a holistic approach. The research or education project should address more than economics or environment or human factors. but should include components of inquiry and evaluation of each . . . . Research and education must address root causes of problems, not symptoms. (N E12) Projects should emphasize holistic approach and include BOTH social and natural sciences at initiation of study. (SD01). Sustainable agriculture intertwines the economic, environmental and social aspects. While a sustainable research or education project may focus more on one aspect (it’s unrealistic to expect to see all aspects equally addressed in a given project), there should not be a total disconnect from the other areas. (ILl3) Issue/Question Cluster F: Useful Information. Issue/Question Cluster F involved the extent to which NCR SARE projects are analyzed for findings and lessons learned. How well is data from SARE projects translated into useful information that is specifically targeted for and disseminated to farmers, lenders, consumers and policy makers? To what extent has NCR SARE developed peer-to-peer structures for information distribution? All research must be explained to the four major societal groups — farmers, lenders, consumers, institutions — to educate them. (M008) The age-old Extension philosophy of “here is a publication or two; read it and decide for yourself” is NOT effective. We need new techniques with the personal investment of extension, researchers and producers. Some kitchen table mentoring exists; we need more of that. Producers tell me, “we’re drowning in data, but we want useful information.” (NDO3) It seems as if we’ve made great strides in researching, demonstrating, and identifying “sustainable” practices and systems. However we have not transferred our knowledge sufficiently to our “end users” — farmers, ranchers, educators and other researchers. (N E04) 70 Issue/Question Cluster G: Greatest Barriers, Critical Components Issue/Question Cluster G involved the barriers that prevent desired changes from occurring and the critical components that support those changes. How well does SARE identify and understand these matters? What are the greatest barriers preventing farmers from adopting sustainable production practices? What are the most critical components that cause farmers to change production practices to more sustainable approaches? (M110) Issue/Question Cluster H: Specific, Measurable Impacts Issue/Question Cluster H involved specific, measurable impacts of SARE programs. To what extent has NCR SARE really made agriculture more sustainable — farmers more profitable, the environment cleaner, quality of life/rural community viability improved? What are SARE’S specific, measurable impacts in these areas? Separating activities from outcomes. Is research getting to producers and having impact? (M119) Does the research result in education and technology transfer that is adopted? (M106) The impact of sustainable agriculture education and research cannot be evaluated in terms of conventional quantifiable outcomes. (M007) A pressing issue in my mind is whether our public investment in sustainable agriculture has made a difference in any number of ways: 1) Awareness/ knowledge of how alternative production and marketing practices can influence agricultural sustainability; 2) Output, in the form of publications, educational materials, formation of learning groups, etc.; 3) Impacts on the acceptance of the outputs produced in these projects by university faculty and administrators as well as journal editors; 4) Impacts on the research/extension/instruction programs of university personnel who have been involved in research/education efforts; 5) Impacts in the form of changed practices by farmers involved in research/ education efforts or by farmers influenced by these efforts. (W106) 71 Issue/Question Cluster 1: Sustainable Agriculture Practices Issue/Question Cluster 1 involved practices that are viewed as falling within sustainable agriculture. Agricultural extension programs are often evaluated according to their ability to cause or at least influence production practice changes. Respondents raised questions about defining “sustainable” agriculture practices and about differentiating them from “conventional” and “organic” agricultural practices. What is considered to be a sustainable agriculture practice? If a practice is done in association with organic farming it’s ok in some circles. However if a traditional farmer does it, it’s not sustainable. (M112) Issue/Question Cluster J: Measuring Issue/Question Cluster J involved challenges that exist in measuring sustainability. To what extent does NCR SARE measure and evaluate incremental steps that show progress in the direction of greater sustainability? How much can we attribute measurable/measured changes to SARE? The question for me is how can we measure if the research and education that has been firnded will lead us to an agriculture that is sustainable? (IL14) Sustainable agriculture research must be broad enough to assess the positive and negative implications of any innovative or new idea then measure the results against a set of criteria that has been developed from the most inclusive sources of knowledge. (M112) We need to evaluate small, incremental steps that show progress in the direction of sustainability . . . . Sustainable ag folks . . . want an all or nothing — i.e. you are sustainable or you’re not. This isn’t how S.A. happens. (M117) 72 Issue/Question Cluster K: Practical Issue/Question Cluster K involved the need for NCR SARE projects and information to be practical. To what extent are NCR SARE projects designed to work on real farms, taking account of economic, time and scale questions? How quickly does NCR SARE move research to practical application on farms? To what extent do projects operate at reasonable and competitive levels of productivity, profitability and net income? The research must be moved to the practical application on the farm phase as quickly as possible. We lose about 100 farmers every day. Agripreneurs can’t wait three years for university data. (M008) Does this production system create household income? If not, production innovations have little value. (NE09) Issue/Question Cluster L: Grounded in Research Issue/Question Cluster L involved the current and the appropriate role of research within SARE. One aspect of this dealt with “farmer driven” as opposed to “researcher driven” approaches. Should SARE be “farmer driven” or “research driven” or what are the pluses and minuses of each approach? (M106) SA research and extension, particularly within the SARE program, appear to be “science phobic” and very much focused on applied, “farmer-driven” and “farmer friendly” research — both useful approaches, but not in isolation — these approaches seem no better to me than “researcher-driven” projects. (MN 05) Issue/Question Cluster M: New Way of Producing, Consuming and Being Issue/Question Cluster M involved new ways of producing, consuming and being and the depth of SARE’s agenda. Is SARE really at the forefront of a sea change in our 73 food and agriculture system or is it just window dressing for business as usual? Are we contributing to a new agriculture and a new way of producing, consuming and being or are we making a growth-oriented industry more socially and environmentally acceptable? How DEEP is the sustainable agriculture research agenda? (M105) Issue/Question Cluster N: Outside Evaluator Issue/Question Cluster N involved internal versus external project evaluation. To what extent should NCR SARE assure that an outside evaluator visit or otherwise review each funded project? I felt that members of the Council needed to visit each of the projects firnded during the course of the research . . . . This would be an evaluation of the effectiveness of the P.I.’s in accomplishing their goals as set out in their application. (SD07) Round Two: Prioritizing Issues/Questions and Identifying Associated Problems A second instrument was created following the analysis of Round One, building on its results. See Appendix D for a copy of the survey instrument. This instrument asked respondents to select from the list of pressing issues/questions the two evaluation issues that they believed would be most important for NCR SARE over the next five years. It also asked them to identify any problems that they might perceive to be associated with the two pressing issues/questions which they selected. The survey was sent to 85 people, and 43 responses were received. Round Two responses are shown in their entirety in Appendix E. Responses were used to establish an order of importance for the evaluation 74 Issue/Question Clusters that were identified in Round One. Table 4-2 shows the order of importance. Table 4-2: Order of Importance of Evaluation Issue/Question Clusters Rank Percentai& N Identifier Issue/Question Cluster 1 30% (13) F Useful information 2 28% (12) E Whole systems questions 3 21% (9) M New way of producing, consmning and being 4 19% (8) C Long-term sustainability 5 16% (7) H Specific, measurable impacts 5 16% (7) D Empower people 5 16% (7) A Who is currently involved 8 14% (6) B Diversity of approaches, practices and systems 9 12% (5) K Practical 9 12% (5) G Greatest barriers; critical components 11 7% (3) L Grounded in research 12 5% (2) J Measuring 13 2% (1) I Sustainable agriculture practice 14 0 0 N Outside evaluator Write in O Summarize and share information Total respondents =43 Following are the problems respondents identified with each Issue/Question Cluster. 75 Issue/question Cluster F: Useful Information. Issue/question F, generating and disseminating useful information from SARE projects, was selected by 30% (13) of the respondents. They identified problems including the difficult decisions between funding more research versus disseminating existing research results and finding the necessary support from already over-extended staff members. They also commented on the logistical challenges of gathering information from a plethora of projects and then translating and disseminating it. Problems faced will include tracking down products resulting from SARE grants. Is it a SARE product if it was developed years later? (N E04) Getting information providers to accurately summarize how the information was disseminated. (OH06) Greater emphasis on dissemination could mean PR competes with research in every proposal. (IA01) Obtaining qualified and unbiased analysis of grants and dissemination of information. (N E05) Information is not handled in mainstream ag extension but “labeled” as sustainable and often given less attention. (NDO3) Staff is already over-extended. (IL14) Issue/question Cluster E: Whole Systems Questions Issue/question E, NCR SARE’s involvement in whole systems questions, was selected by 28% (12) of the respondents. The problems they identified included the need for multi-disciplinary team approaches, the long time required and the high costs of this approach. They questioned whether SARE really has the resources to use this approach effectively. 76 Answering whole systems questions will probably takes large interdisciplinary teams that need and several years to do their work. Also, people might need help in figuring out how to do this kind of research. (MN06) Conventional approach to technology development and transfer and reluctance to confront the challenge with “systems” or “wholes.” (M007) Small farm systems do not receive the same attention throughout agriculture as do large operations. (IN09) Difficult to measure. Doesn’t fit into tvvo-year funding cycles. Requires new ways of doing evaluation. There is never a “neat” or “complete” answer or end point; doesn’t fit well with media or people who want dramatic numbers. (KSOl) Difficult in quantifying environmental and social impacts in the one or two year life of a grant. Symptoms much easier to identify than root causes. (NE05) Need for team approach. Much of this work is conducted in land grant institutions, most of which do not adequately reward or even recognize the importance of teamwork. We are still promoted and tenured primarily on the basis of individual accomplishments. (SD01) Issue/question Cluster M: New Way of Producing, Consuming and Being Issue/question M, a new way of producing, consuming and being, was selected by 21% (9) of the respondents. They identified problems including our limited knowledge about what causes people to make significant changes as well as the inherent conflict between a demand for quick adoption and accountability and the time needed to redesign entire systems. Another problem identified dealt with the need to focus more on qualitative aspects of development than on quantitative aspects of growth. Deciding with any accuracy which problems/ studies lead into a deeper understanding. (MNIO) This is a broad question and may be difficult to focus. (N E04) Redesign of the ag system may be needed and this type of work is hard to evaluate 77 and high risk. (IA05) It is difficult to design a new way or approach to national practices or to change the overall way people do things (SD07) What is it that causes people to shift from linear thinking patterns to a more holistic approach? (NE 1 0) Issue/question Cluster C: Long-Term Sustainability Issue/question C, long-term sustainability, was selected by 19% (8) of the respondents. They identified problems including the mismatch between SARE’S one to two year funding cycle and the time needed for many changes in agricultural system sustainability plus effective evaluation of many research and education efforts. Evaluating sustainable agricultural practices certainly requires looking at their impacts over more than two years. However long term research ties up funds and limits the number and diversity of funded projects. (OH09) Encourage by funding requirements long term projects. (IA09) Funding over the long-term, but also the difficulty in keeping a multi-disciplinary team together long enough to measure the various facets of sustainable ag, particularly when faced with hostile peers. (SD01) Issue/question Cluster H: Specific, Measurable Impacts Issue/question H, specific, measurable impacts, was selected by 16% (7) of the respondents. They identified problems including how to determine what should be measured and then getting the data needed. They also questioned whether issues of sustainability can be meaningfully discussed in a growth economy. Getting hard data on changes made, practices adopted, etc. (OH06) 78 There is little room for discussion of these issues in a grth economy . . . . Current agriculture fits into the extractive economy and values only profit. . . . (W114) The problem is what are the specific measurable impacts. (IL14) Issue/question Cluster D: Empower People Issue/question D, empowering people, was selected by 16% (7) of the respondents. They identified problems including its measurement and evaluation, its fit with the current SARE funding cycle, its challenge to convention, and the limited support for an empowerment approach. How can we overcome a conventional evaluation process mind set and reluctance to change convention to address real issues? How can we move away from technology transfer? (M007) Difficult to measure. Doesn’t fit into two-year funding cycles. Requires new ways of doing evaluation. Likely won’t appeal to government funders; may threaten them. Appears as “soft science.” (K801) Changes in understanding cannot be measured easily or with complete accuracy. Evaluation time line will be long. The vast majority of farmers/people are not at this level. Constituency is very small. (MNlO) Issue/question Cluster A: Who is Currently Involved? Issue/question A, who is currently involved in NCR SARE, was selected by 16% (7) of the respondents. They identified problems including how to decide who should be targeted for involvement and the related problem of how that decision is implemented. They also listed the problem of mainstream versus insider evaluation and impacts. 1 don’t see NCR SARE having mainline ag impacts with present players. (M107) 79 How does one decide who the “right people” are? I think SARE should be focusing on the needs of family farms, but not by some litmus test of size or structure. (IAOI) Issue/question Cluster B: Diversity of Approaches, Practices and Systems Issue/question B, diversity of approaches, practices and systems, was selected by 14% (6) of the respondents. They identified problems including a tendency for some to view sustainable agriculture as synonymous with organic agriculture and to reject all aspects of conventional agriculture. Some have a rather narrow view of sustainability in agriculture and dismiss anything “conventional.” A broad definition risks alienating some strong SARE supporters. (OH09) The biggest problem or challenge in dealing with this issue is a “mind set” by “organic” promoters that this is the only type of production that is sustainable. (SD08) Issue/question Cluster K: Practicality Issue/question K, practicality of SARE projects, was selected by 12% (5) of the respondents. They identified problems including how to get results used on farms. How do we take the project results and get them to be practiced on real farms? (ND02) SARE must fund projects that are practical, profitable and productive and can be adopted to larger scale use. (W114) Issue/question Cluster G: Greatest Barriers; Critical Components Issue/question G, greatest barriers and critical components, was selected by 12% 80 (5) of the respondents. They identified problems including the difficulty of measuring the impacts of such an approach, at least in the short term. Another problem identified was that the greatest barriers may lie largely outside the areas that SARE can most effectively address. The greatest barriers may not be able to be addressed through educational programs as we know them. (IA05) This issue has to do with diagnosing the problem and it’s hard to show action results when you put emphasis on doing the right thing instead of doing things right. (M119) Issue/question Cluster L: Grounded in Research Issue/question L, SARE project research grounding, was selected by 7% (3) of the respondents. They identified problems including a perception that SARE avoids basic research and a concern that innovative research approaches may not be provided a fair evaluation. NCR SARE seems phobic about funding or supporting any basic research. Methods are poorly understood, resulting in an almost paranoid reaction to any type of systems research that even suggests using basic research approaches. (MN 05) I usually find this a loaded question. In my experience, individuals use this to quickly dismiss the different — that which hasn’t been researched by replicated, reductionist methods by a land grant university. This immediately shrinks options into the small world of what fits the prevalent paradigm and which is acceptable to land grant funders. (KSOl) Issue/question Cluster J: Measuring Issue/question J, measuring, was selected by 5% (2) of the respondents. They did 81 not identify any problems associated with this Issue/Question Cluster. No problem once sustainability is defined in broad terms. SARE’s two year “projects” can only be incremental steps and should not be difficult to evaluate. . . (SD06) Issue/question Cluster 1: Sustainable Agriculture Practices Issue/question I, sustainable agriculture practices, was selected by 2% (l) of the respondents. This respondent identified the problem of overlap between practices that might be considered organic, conventional, or sustainable. Must recognize that if organic is defined and conventional is defined, that a sustainable system could include both. (M112) Issue/question Cluster N: Outside Evaluator Issue/question N, outside evaluator, was not selected as one of the top two issue/questions by any of the respondents and no problems were identified. Issue/question Cluster 0: Summarize and Share Information Issue/question O, summarize and share information, was written in as an addition to the issue/question list in Round Two. We now have 11 years of research under the funding of SARE. We need someone or group to summarize this wealth of information and condense or act on the outcome. (SD07) 82 Round Three: Recommendations to Address Problems Following the analysis of Round Two, a third instrument was created. The focus of this instrument was to ask participants to make recommendations about ways to address the problems that were identified in Round Two related to the Issue/Question Clusters identified in Round One. Appendix F contains a copy of the survey. The Round Three survey was mailed to 85 people, and 32 responses were received. Round Three responses are shown in their entirety in Appendix G. Round Three responses were sorted and grouped into Recommendation Themes. Tables 4-3 through 4-14 display the Recommendation Themes for the problems associated with each Issue/Question Cluster. The tables indicate how many respondents commented about problems related to each Issue/Questions Cluster, how many respondents’ recommendations were included to comprise each Recommendation Theme, and what percentage of respondents to a particular Issue/Question Cluster each Recommendation Theme represents. In cases where one respondent’s recommendation included more than one idea, parts of that recommendation were included in each appropriate Recommendation Theme. All Recommendation Themes that were identified by two or more respondents are discussed. Recommendation Themes for Useful Information (F) Twenty-nine respondents made 60 codable responses that dealt with problems related to useful information. These codable responses were organized into 12 Recommendation Themes. Table 4-3 displays Recommendation Themes for useful 83 information and the number of respondents whose responses are included in each theme. Table 4-3: Recommendation Themes for Useful Information (F) Recommendation Themes N Percentage_ Clarify project expectations 13 45% Target funding 12 41% Work with Extension/Land Grants 9 31% Do more follow up 8 28% Utilize media 6 21% Don’t depend on Extension 3 10% Expand PDP coordinator role 3 10% Partner 1 3% Facilitate networking 2 7% Community focus 1 3% Show people how 1 3% Not a problem 1 3% Total 60 Total respondents = 29 Thirteen (45%) of the respondents spoke about various ways in which SARE might clarify expectations associated with its project funding to get more useful information from those projects. Each proposal should contain an explicit, feasible plan for dissemination of results. The RFP might address this as follows, “Describe the deliverables of this project, in what form they will be delivered, and to whom they will be delivered.” (IL08) Outreach should have been one of the requirements for funding and should be part of the final report. (W114) PR should be part of every proposal. Perhaps withhold some of grant until 84 information dissemination is assured. (IN09) Make sure that we ask report writers to tell us how they expect to use information after grant ends. Improve integration between Chapter 1 and Chapter 3 programs. Ask PDP proposals to use Chapter 1 work. (MNO6) Twelve (41%) of the respondents offered ideas about targeting SARE funding to assure dissemination of useful information. Perhaps divert funds to hire someone to analyze and summarize pertinent material. (IN09) Issue specific calls for proposals to analyze and disseminate existing information. (MNO6) This evaluation must tell the funder things they don’t want to hear as well as happy messages. I suggest contracting this evaluation out maybe even as a RFP. (KSOl) Contract with retired or other professionals in the field to synthesize and publish the salient points for use by all. (N D03) Perhaps use an outside evaluator to obtain critical data and information. Can some of the SARE dollars be used for selected intensive evaluation? If so, focus on some of the most effective grants and don’t worry about the others. (M110) Every proposal/funded project could have monies built into it to a) allow the researcher to attend a conference where SARE projects will be profiled; and b) participate in a conference — perhaps conduct a workshop — in which SARE projects are profiled. (M105) SARE could hire a consultant to work in concert with the communications specialist to review results and information products from all grants previously funded. (N E04) Nine (31%) of the respondents suggested that SARE work more closely with Extension and land grant institutions to disseminate information. SARE can’t take it upon itself to create a whole new Extension service. Just do good research, make every effort to get it to users, and work to build a strong constituency for sustainable agriculture within Extension. (W112) 85 Partner with Land Grant Universities — grad students have to review the literature for their research projects. University communications departments can lead in writing/telling about projects from their schools. May need feeds from SARE staff to initiate dissemination. Use interns, volunteers, retirees to do PR/media work. Web-based story/activity archives - use students to build the site for experience. (NEl 1) Very few farmers have the scientific training to properly summarize their projects and demonstrations. Is the Dale Mutch model worth exploring on a broader basis? Perhaps the role of dissemination needs to be a somewhat separate effort led by Extension. This would take some of the pressure off the proposal development and implementation process. (M110) But three (10%) of the respondents recommended that SARE not depend on Extension. Not Extension (for distributing SARE information). They’re on a different track (commercial, chemical, Monsanto) and SARE won’t change them. (SD06) Eight (28%) of the respondents suggested strategies for post-project follow-up to find out about longer-term results. Ask in the final report how information was disseminated. If tracking is really important on any issue with the SARE program, interview grantees by phone. Also, grantees will be more responsive if they believe their reporting/tracking to the SARE program improves (or hurts) their future funding. One way to get that message home would be to reject a pre-proposal based on unsatisfactory reporting on previous grants. The caution, of course, is to be firmly consistent and have agreement among decision makers within SARE. (KSOl) It might be useful to do a 5-year follow-up on selected projects to see what else “new” has happened. (M105) Keep all grant recipients involved in SARE and do follow-ups to grants, especially producer grants, out several years to ask if practices are still being used. (W114) Closer work with the media was recommended by five (19%) of the respondents. Team up with the popular farm press at every opportunity. Farmers read . . . , and the farm press is looking for stories of hope. (N E10) 86 Three (10%) of the respondents made recommendations about expanding the role - of PDP coordinators. Emphasize use of Professional Development Program (PDP) to disseminate SARE results. To do this, the PDP’S in each state and researchers must become more aware of each other and communicate. Regional PDP coordinator could help facilitate this. Researchers can be encouraged to use PDP in their outreach. (OH09) Two (7%) of the respondents made recommendations that SARE act to facilitate networking. Every proposal/funded project could have monies built into it to a) allow the researcher to attend a conference where SARE projects will be profiled; and b) participate in a conference — perhaps conduct a workshop — in which SARE projects are profiled. This would further the networking and the educational process for everyone. (M105) Recommendation Themes for Whole Systems (E) Twenty-eight respondents made 45 codable responses that dealt with problems related to whole systems. These codable responses were organized into 18 Recommendation Themes. Table 4-4 displays Recommendation Themes for whole systems questions and the number and percentage of respondents whose responses are included in each theme. Five (18%) of the respondents recommended that NCR SARE emphasize whole systems research. Put out a call for whole farm research and economic and social and environmental! People will respond. (N E1 1) 87 Table 4-4: Recommendation Themes for Whole Systems (E) Recommendation Themes N Percentag e Emphasize whole systems research 5 18% Partner 5 18% Clarify expectations 4 14% Exclude basic research 4 14% Options to systems research 3 11% Include basic research 3 11% Emphasize community focus 3 11% Facilitate networking 2 7% Show people how 2 7% Emphasize team approaches 2 7% Target funding 2 7% Emphasize farm/farmer focus 2 7% Develop people skills 2 7% Don’t depend on Extension/Land Grant 1 4% Fund longer projects 1 4% Don’t fund longer projects 1 4% Not a problem 1 4% Provide recognition 1 4% Total 45 Total Responses = 28 The problem is worse than envisioned by many with interests in SARE. The SARE program tends to view farms as complete systems and fails to recognize that farms are just part of a much larger food and agriculture sector. Some SARE advocates view 88 processors, distributors, retailers, and input suppliers as the enemy, when, in fact, they are part of the system required to make agriculture work in modern times. Except in situations where farmers market their own products directly to end-users, farm enterprises are just stages in complex multi-stage value chains. Very few people in traditional agriculture are trained or inclined to study these very complex systems, where some of the most important messages are generated. Often, they don’t want to hear those messages. SARE should encourage industrial engineers, operations researchers, and other non- traditional scientists to analyze these value chains, using their sophisticated systems research approaches. (IL08) Five (18%) of the respondents recommended that NCR SARE partner in this whole systems work. Combine forces with ARS IFS. The must develop procedures for this too. (NEll) Whole systems research is very costly. SARE doesn’t have the resources or expertise. Again, partner with appropriate entities. (IN07) Seek external partners to assist in funding. (M110) Four (14%) of the respondents recommended that NCR SARE work to clarify expectations. Four (14%) of the respondents recommended that basic research be excluded from NCR SARE work. Basic research related to the mechanistic world view is inappropriate for SARE. (M103) SARE should not fund basic research period! 1!! (M113) But three (11%) of the respondents recommended that SARE include basic 89 research in its portfolio. Get over it [being phobic about funding basic research]! Basic research will happen —- it is part of NCR SARE’s job to support research that enhances SA and basic research is needed to do so. (MNOS) Three (11%) of the respondents recommended that NCR SARE emphasize a community approach and focus. An effort should be conducted on a community basis. Developing indicators of sustainability and applying them to a community would be a better method of determining these dimensions. (W107) Need to engage communications experts to assist in communicating the importance of agriculture to rural communities and areas. (M110) Two (7%) of the respondents suggested that NCR SARE facilitate networking. Encourage multi-disciplinary, multi-professional projects. Co-fund projects with PDP and producer funds. (N E04) Two (7%) of the respondents recommended that NC R SARE develop training opportunities to show people what is desired. Sponsor projects or workshops that focus on how to do whole systems research. (MNO6) Two (7%) of the respondents made suggestions that dealt with ways to emphasize and support team approaches. Multi disciplinary teams are never neat and clean. This is always a problem with a team approach, but it is vital to obtain the holistic information SARE requires. In short, don’t fret about this -— press on. (SD01) Two (7%) of the respondents made suggestions about ways to target funding for whole systems work. A key strategic question is whether you go with those you know who carry through programmatically or choose new applicants. If course, if you want to be on the learning edge, you need new ideas, skills, and practices. So the programmatic goal may well be how to balance the familiar with the new. (KSOl) 90 focus. Two (7%) of the respondents made suggestions about emphasizing a famr/farmer I have experienced a wide variety of commitment from multiple disciplines into a project. I fear too often partners in a grant application were not very involved in project design. Sometimes a partner will not even be asked and then appear on an application. Sometimes only asked by phone and never see the application. From the funder’s perspective, this is very difficult to tell. One response from SARE (and throughout this response I am only responding to the North Central Region) that is very wise is to at least focus on the quality of farmer letters of support. That is excellent. I would look for originality in writing and expressions of passion and involvement. From my perspective as a grantee, the more work I put into development of the grant, the better it becomes implemented. Sometimes that pre-application work is a luxury. But if funded, it pays off later. (K801) Including an on-farm component in ARE projects helps considerably in this area (educate public on importance of sustainability). (SD01) Two (7%) of the respondents made suggestions about developing people skills. SARE should look at providing support to encourage leadership and community building skills among land grant faculty, non-profits, grassroots workers and farmers. (IA05) People create most of their problems. Solutions require changing values, paradigms, reward systems in the marketplace, and changes in management skills. (K801) Recommendation Themes for New Ways of Thinking, Producing, Consuming (M) Twenty-eight respondents made 34 codable responses that dealt with problems related to new ways of thinking, producing, consuming and being. These codable responses were organized into 10 Recommendation Themes. Table 4-5 displays Recommendation Themes for a new way of thinking, producing, consuming and being and the number of respondents whose responses are included in each theme. 91 Table 4-5: Recommendation Themes for New Ways of Thinking, Producing, Consuming (M) ll Recommendation Themes N Percentage" Deepen efforts, redesign 15 54% Fund longer projects 3 1 1% Facilitate networking 3 11% Not a problem 3 1 1% Clarify expectations 2 7% Improve reviewer selection 2 7% Target funding 2 7% Do more follow up 1 4% Don’t depend on Extension 1 4% Partner 1 4% Total 34 Total Responses = 28 Fifteen (54%) of the respondents made recommendations about ways NCR SARE might support a deeper agenda for sustainability and help redesign food and agriculture systems. SARE needs to have a portfolio approach in which there is a commitment to fund fundamental system-redesign projects in order to develop the knowledge base for sustainable agriculture in the future. (W112) Develop a network of people who have made the paradigm shift and use them as a sounding board for developing a “deeper” sustainable ag research agenda. (NDOS) This (Development without growth) requires a change in mind set and few people in land grants have made that change. Place strong emphasis on innovation and design, not just fine tuning current systems. Reward risk takers! (N El 1) 92 Great question. I believe there needs to be a balance between development and growth. And a strength of SARE is that it changes its emphasis and follows the learning curve in sustainable ag. For example, emphasis shifts from production research, to holistic planning, to marketing. I think this is very healthy. Constituencies get ruffled. This shifts dictate new expertise and audiences. Moving from linear to holistic thinking required being moved into areas of mental discomfort. Farmers tend to be practical, hands-on folks. Seeing, reflecting, being mentored are important processes. (K801) Three (11%) of the respondents suggested that deeper changes often take a long time. Unless SARE is willing to invest in longer term studies and provide insights into what makes these studies successful, neither (system redesign or shift to holistic approach) is likely to happen. This (development versus growth) is the difference between generating wealth and generating money. Focus on the ability to build equity. (N El 0) Three (11%) of the respondents made recommendations for ways in which NCR SARE might facilitate increased networking. Possibly a workshop/conference on holistic approach. (W106) Three (11%) of the respondents made comments indicating that this is not a problem, that SARE is already providing an opportunity for a deeper agenda. Not a problem. SARE provides a space in which people MAY “think, consume, be” different. SARE can’t mandate that, it’s going to have to evolve. (IA01) Two (7%) of the respondents suggested the need to clarify expectations for NCR SARE projects. Emphasize research on management systems, agroecosystems, whole-farm plans, etc. in calls for proposals. (OH09) Two ( 7%) of the respondents suggested that SARE improve its reviewer selection processes. SARE needs to actively involve consumers and non-ag people. (W114) 93 Two (7%) of the respondents made recommendations about targeting funding to support these new ways. SARE should allocate some funds for redesign work and track the impact of the work as it filters in the larger community. SARE should fund some type of human systems team that follows all research/education projects. (IA05) Can this analysis be undertaken through some university research? This is critical information and would help to make Extension more effective in working with farmers. (M110) Recommendation Themes for Long Term Sustainability (C) Twenty-six respondents made 30 codable responses that dealt with problems related to long term sustainability. These codable responses were organized into eight Recommendation Themes. Table 4-6 displays Recommendation Themes for long-term sustainability and the number of respondents whose responses are included in each theme. Thirteen (50%) of the respondents recommended that NCR SARE fund some longer projects. Why not invite collaboration on at least one, carefully thought out, long-term research project? (ND05) Fund some projects for longer than 2 years. (IAOI) “Sustainable ends” cannot be served with two year discontinuous firnding. Partner for longer term activity. (IN07) 94 Table 4-6: Recommendation Themes for Long-Terrn Sustainability (C) “Recommendation Themes N Percentag Fund longer projects 13 50% Use project renewal mechanism 4 15% Partner 3 11% Broad definition; sustainability a process 3 11% Follow up more 2 8% Do not fund longer projects 2 8% Encourage basic research 2 8% Increase central control 1 4% Total 30 Total Responses = 26 Four (15%) of the respondents suggested NCR SARE use its current guidelines, which provide an option for renewing worthy projects. I think that two-year projects are ok provided that SARE is not biased against continuation proposals for promising, productive projects. (W112) Make clear guidelines for the extension of time lines on longer projects. (NE10) Three (11%) of the respondents recommended that SARE try to develop partnerships that could help with longer-term funding. Look to support projects where other funds are available. (W106) Three (11%) of the respondents recommended keeping in mind that sustainability is a process and must be broadly defined. First, we need to recognize that sustainability is a process with a beginning but not end. What we may think is ‘sustainable’ today will shift tomorrow, partly because nature is always changing. (ND05) 95 Trying to identify practices that will make agriculture sustainable in the long term is very difficult when technological and economic changes are so rapid and unpredictable. Farmers need ways to respond rapidly, effectively, and sustainably to change. In the long term, flexibility is the only sustainable approach. They cannot be restricted by long-term strategies. such as long-term rotations. (IL08) Two (8%) of the respondents made suggestions about improved follow-up for NCR SARE efforts. Require P1 to agree to a follow-up after proposal is over at 1,3, 5 years. (K810) Two (8%) of the respondents suggested not funding longer projects. Stay with short-term, seed money grants; have one requirement be grant proposals for long-term funding. (N El 1) Two (8%) of the respondents suggested encouraging basic research. Politicize SARE to get a share of NSF or NR1 funding. (ND03) Recommendation Themes for Specific, Measurable Impacts (H) Twenty-seven respondents made 31 codable responses that dealt with problems related to specific, measurable impacts. These codable responses were organized into 13 Recommendation Themes. Table 4-7 displays Recommendation Themes for specific, measurable impacts and the number of respondents whose responses are included in each theme. Six (33%) of the respondents recommended strategies to target some of the project funding for a diversity focus. We may need to fund projects that are designed to develop alternative ways of measuring impacts. (W106) Maybe we should set aside a certain percentage of our budget for evaluation and measurement of impacts. (MN 06) 96 Really tough — maybe ask for a couple of projects to develop evaluation tools? (NEH) You cannot escape making choices. Someone has and will do it. And those choices should be based on good evaluation. This then means contracting for meaningful evaluation. (K801) Part of the role of research and Extension is to help refine the questions and analyze data and information. There is a need to continually raise questions about the social implications, environmental consequences, and economic parameters of production practices. This is difficult work (research) however it has to be led by university researchers to gain greater insights to benefit rural families and their communities. (M110) Table 4-7: Recommendation Themes: Specific, Measurable Impacts (H) Recommendation Themes N Percerfig Target funding 6 33% Quality of Life 5 19% Forces beyond SARE 4 15% Clarify expectations 2 7% Partner 2 7% Show how 2 7% Specific indicators 2 7% Immeasurable 2 7% Participant evaluation 2 7% Follow up better 1 4% Emphasize farm/farmer focus 1 4% Not a problem 1 4% Emphasize reviewer selection 1 4% Total 3 1 Total Respondents = 27 97 Only case studies or large databases can give a confident answer here. The concern is externally great for work in this area with GMO dominance in the bushes. (ND03) Five (19%) of the respondents made suggestions related to quality of life. SARE should look at a new array of projects that delve into the psycho-social or mental health values of a more sustainable agriculture. Does a higher quality of life have a monetary value in terms of less resource degradation, less medical bills, etc? (IA05) Quality of life if the most difficult concept to capture for obvious reasons, but in some ways it’s also the most important aspect of sustainable ag that we should measure. SARE should work with rural sociologists and others to try to gauge QOL impacts. (N E04) Four (15%) of the respondents pointed out SARE’s relatively small potential for impacts within much greater outside forces. SARE participants need to be concerned about impact, but we can’t realistically expect huge impacts when farmers have to operate in an adverse policy environment. (W112) The population served by modern agriculture is almost entirely urban. What about their community and quality of life? It is their desire for abundant, uniform, safe, high quality, affordable food, as expressed when they buy groceries, that drives agriculture toward the industrial model and toward highly coordinated markets. Certain restrictions in the subject matter of SARE research cause a Significant part of the program to be irrelevant to a large segment of modern agriculture, namely the segment that serves those big grocery stores where most people buy their groceries and lots of other things. On the other hand, removing those restrictions would make it difficult to differentiate the SARE program from other federally sponsored agricultural research programs. The SARE program is different, however, in being more mission-linked (not just mission-related, like the NR1). SARE should remove the ideological restrictions and emphasize its mission-linked, goal-focused, systems-oriented nature. (IL08) Two (7%) of the respondents recommended that NCR SARE clarify expectations. All projects must have measurable outputs -— quantitative or qualitative. We must be able to answer how many objectives we achieve and how well. Consumer interest is bringing appreciation of quality of life into the marketplace. 98 We can be more explicit about what it means — in ways that are hard to co-opt. (N El 1) Two (7%) of the respondents recommended that NCR SARE partner more effectively with other entities. Beware of “taking credit.” There are likely many entities that have contributed to an outcome. Share credits. SARE in some instances has publicly claimed sole credit when they were “a participant.” (IN07) Two (7%) of the respondents made suggestions related to showing people how to measure impacts. We may need to fund projects that are designed to develop alternative ways of measuring impacts. These results then need to be shared with firture project holders. (W106) Two (7%) of the respondents suggested that NCR SARE identify specific indicators. Shine the light of public opinion on the various systems and the truth will be evident. How many smiles/season does each system provide to the farmer? What is done with the profits extracted? What are the underlying values that determine happiness? Determine the difference between optimum production versus maximum production. (NE 1 0) Two (7%) of the respondents suggested that SARE’s results are immeasurable. This is an item of faith. You can’t measure this any more than an individual can measure the goodness of a life. Do it because it needs doing. (MNlO) Two (7%) of the respondents suggested that NCR SARE emphasize participant evaluation. Those who are actively engaged in the project, or adopt aspects of it, or who reject it should have a voice in its evaluation. (M105) 99 Recommendation Themes for Empowering People (D) Twenty-three respondents made 24 codable responses that dealt with problems related to empowering people. These codable responses were organized into 10 Recommendation Themes. Table 4-8 displays Recommendation Themes for empower people and the number of respondents whose responses are included in each theme. Table 4-8: Recommendation Themes: Empowering People (D) Recommendation Themes N Percentag" Facilitation focus 7 30% Qualitative focus 6 26% Re-invent Extension 4 17% Show people how 2 4% Clarify expectations 1 4% Community focus 1 4% Follow up _ l 4% Fund longer projects 1 4% Partner 1 4% Not a problem 1 4% Total 24 Total responses = 23 Seven (3 0%) of the respondents recommended that NCR SARE pursue an emphasis on facilitation. We need to help extension personnel expect to be questioned and we need to help everyone learn to ask questions in the form of respectful dialogue so they don’t immediately alienate others. (MNO6) The empowerment of people must be the guiding principle behind SARE. Granted, the constituency looking for these outcomes may be small in circles of 100 power; ask any caring parent about their dreams for their kids and you will get aspirations of empowerment. So I argue this is a widely held and understood goal. This is what makes SARE distinct and, I argue, effective. This is getting at root causes. This is what gets SARE visibility and distinguishes it from so many USDA programs. Celebrate this distinction! The challenge is evaluation. What works? It’s a combination of science, art, and spirituality. This is what makes it so baffling, a large target for criticism, and interesting. Simply put, it just has to be done as best it can. Commit to long term evaluation. Maybe bring in grantees to learn and figure how to better evaluate. Learn from other funders on what they have done. Learn from their MISTAKES. I hear good things from the University of Kansas with their community health evaluation. I hope to learn from them. (K801) Absolutely. Facilitate the discovery and learning process. We need a complete mind set change here. But it goes against all our experience and also does not meet the expectations of most audiences. (N E1 1) Six (26%) of the respondents recommended that NCR SARE strengthen its qualitative work. The term technology transfer is totally inappropriate for most alternative ag systems. All of Rogers and Shoemaker and Zaltrnan and Duncan’s (work) still is good and appropriate. Farmers are outstanding at changel! Since it is a qualitative issue, the techniques of Jules Henri Poincare, Benuit Mandelbrot, etc. should be used. With this, the basic research argument is no longer an issue!! (M103) Some good case studies could answer some of these questions. (W107) Four (17%) of the respondents made recommendations about Extension’s role in this area. Extension is becoming increasingly irrelevant to most farmers. But this presents an opportunity for a small group of Extension people to re-invent Extension. Why not use some of the Extension training money to invite the development of an “empowerment” rather than a “technology transfer” model of extension within NCR SARE? Their objective would be to reinvent Extension on a small scale and demonstrate how it could be done on a larger scale. (N D05) We need to help extension personnel expect to be questioned and we need to help everyone learn to ask questions in the form of respectful dialogue so they don’t immediately alienate others. (MNO6) 101 Two (9%) of the respondents recommended that NCR SARE emphasize showing people how to change. Facilitate the discovery and learning process. We still need to attempt to do this evaluation and maybe have a project on how to do it. (NEl 1) Real world problem facing all of us public servants. How do we get 95% of producers to change? If SARE has good answers, others will follow. Either table topics, free food, incorporate into other community events. One-upmanship. This is a real tough issue. (ND03) Recommendation Themes for Who is Currently Involved? (A) Twenty-eight respondents made 28 codable responses that dealt with problems related to who is currently involved in NCR SARE. These codable responses were organized into nine Recommendation Themes. Table 4-9 displays Recommendation Themes for who is currently involved and the number of respondents whose responses are incl uded in each theme. Thirteen (46%) of the respondents recommended that SARE keep its efforts focused. Who cares about mainline agriculture? Mainline agriculture has resulted in middle-sized farmers leaving farming. (M113) Get meaningful outside evaluation. I question whether mainline ag impact is a realistic intermediate goal. The trends toward consolidation and industrialization are so powerful we may not be able to stop them in the near firture. Our mission may well be creating a parallel development option of local food systems. (K801) Focus should be on small and moderate-size farms. Emphasize public rather than corporate benefits. (OH09) We need to spend more time talking about who “mainstream agriculture” is. To some, it means large farms, and I don’t think they need our help as much as smaller farms do. I think SARE works more with small and moderate sized farms and I’m comfortable with that. (MNO6) 102 Table 4-9: Recommendation Themes: Who is Currently Involved? (A) ll Recommendation Themes N Percentag; fStay focused 13 46% Reach mainstream 4 14% Clarify expectations 3 11% Involve good communicators 2 7% Involve people committed to change 2 7% Diagnostic focus 1 4% Increase central control 1 4% Emphasize reviewer selection 1 4% Not a problem 1 4% Total 28 Total Responses = 28 I think that SARE would be best off if it is stated that its core audiences are smaller and moderate-size farms but that it welcomes its technologies being useful to larger farms. It’s contradictory but realistic. (W112) Four (14%) of the respondents recommended broadening NCR SARE’S audience and involvement to better reach the mainstream. “Mainline” involvement is critical to having impact across agriculture. Don’t exclude representatives from the “mainline”. Quit being “standoffish” and invite “mainliners” to the table as participants. (IN07) Get state Extension directors and NRCS state conservationists involved. (OH06) Identify the number of new people coming into sustainable agriculture. SARE should encourage new faces and new leadership instead of keeping the same people on the lecture circuit and at the conferences. These events are almost always the “choir.” (IA05) Three (1 1%) of the respondents suggested clarifying program goals as necessary to determine who should be involved. 103 The big question is whether you want to support those interested in a local, value- added sustainable agriculture or move those in mainstream ag towards sustainable ag. The answer here tells you a lot about what you fund and how it gets communicated. (W106) Depends on program objectives. If you want to impact ag on the larger scale, then move from being a “club” to include “mainliners”. (IN 07) Two (7%) of the respondents recommended that NCR SARE involve good communicators. Get the “right” people as spokespersons. People who are creditable (not movie actors) and who can explain sustainable agriculture in positive terms that are readily understood. (IN09) ' Do theories of diffusion of innovation still apply to today’s agriculture? Perhaps this needs to be revisited in light of today’s farmers, agriculture. Sustainable farmers aren’t often viewed as the “leaders.” (M110) Two (7%) of the respondents recommended that SARE involve people committed to change. Anyone who identifies a need for change then commits to helping make change happen would be the “right” person. However, the individual must acknowledge that any potential change may require changes from within (ourselves or our institutions) as well as external changes. (M112) Recommendation Themes for Diversity of Approaches, Practices, Systems (B) Thirty respondents made 40 codable responses that dealt with problems related to diversity of approaches, practices and systems. These codable responses were organized into nine Recommendation Themes. Table 4-10 displays Recommendation Themes for useful information and the number of respondents whose responses are included in each theme. Twelve (40%) of the respondents recommended that NCR SARE act in ways to 104 enhance reviewer selection. Be more strategic in selecting AC and TC members to make sure that we do represent a diversity of approaches. (MNO6) Carefully choose evaluators and/or be sure they understand SARE’S definition. (SD06). Broaden pool of educators; go outside traditional pool into general consumers (for evaluators). (W114) Diversity will only be ensured in the SARE program if the decision makers in SARE (the AC and TC) represent diversity. In my perspective, there needs, especially on the AC, to correct the under-representation by farmers and NGO’s. And I wouldn’t recommend a larger AC! If marketing and local food systems becomes a more important strategic goal, then consumer, marketing, and food system folks need to get on these decision making bodies. There also should be sensitivity to gender and race. But most importantly, representatives should be people who think outside of the industrial ag box. (K801) Table 4-10: Recommendation Themes: Diversity of Approaches, Practices, Systems (B) Recommendation Themes N Percentag Improve reviewer selection 12 40% Organic does not equal sustainable 9 30% Importance of diversity 9 30% Goals 3 10% Organic equals sustainable 2 7% Clarify expectations 1 3% Increase central control 1 3% Facilitate networking 1 3% Reach mainstream agriculture 1 3% Total 40 Total Respondents = 30 Nine (30%) of the respondents recommended maintaining the concept of 105 sustainable agriculture as distinct from that of organic agriculture. SA is M than organic, as determined by the SARE program’s goals. We’ve erred on the organic side more than not, which I think has been at the expense of good projects — the mention of a herbicide in proposals has been the death of great proposals in the past. 1 think we need more reviewers that can explain the real impacts of proposals so better thought can be given on the merit of proposals — not just a knee jerk “all pesticides are bad” rxn. (MN05) Stick with “sustainable agriculture”. . . . Avoid pigeonholing into conventional, biodynamic, organic, etc. (M112) Shades of gray! Less tolerance in some organic growers than in any others. Any progress towards sustainability versus corporate minion farming is good. Preach the gospel, so to speak. (ND03) ' Broaden to ag and food systems, not just organic. (N E1 1) Organic and sustainable overlap but are not the same. (M119) Many paths to sustainability should be chosen. Which paths to support programmatically with SARE should be partly based on which of those paths are being funded elsewhere. For example, no-till gets lots of funding outside of SARE. So I would not make it a focus in SARE. I would only fund no-till on the learning edge such as designing crop rotations that innovatively integrate crop rotations or dramatically reduce pesticides or integrate livestock. (K801) Recognition that there fl many solutions to similar problems is critical for university experts to recognize and accept. We must accept that we are dealing with a living, dynamic system. Consequently, methods and practices will continually change as new knowledge is developed. Use teams of multi- disciplinary evaluators to avoid this situation (favoring a “pet” approach). (M110) Nine (30%) of the respondents recommended enhancing the value and importance of diversity to SARE. Include diverse groups and viewpoints on Administrative Council and Technical Committee. (OH09) Encourage innovative and unusual methods; too much research is the same stuff again and again. Cannot be too diverse in thinking — we need that! (N El 1) Diversity of approaches could stem from diversity of project investigators from a 106 goals. diversity of organizations and institutions. SARE should broaden horizons beyond the land grant system in funding decisions. (N E04) Three (10%) of the respondents made suggestions about sustainable agriculture Methods and approaches are secondary to the purpose of the project and how well the project advances the objective of the SARE program. It is important that the methods/approaches be appropriate for the issues being addressed. (W106) Two (7%) of the respondents made suggestions about dealing with sustainable and organic as the same. Let’s call it the organic way and the natural way. (K810) Recommendation Themes for Practicality (K) Twenty-four respondents made 25 codable responses that dealt with problems related to practicality. These codable responses were organized into eight Recommendation Themes. Table 4-11 displays Recommendation Themes for practicality and the number of respondents whose responses are included in each theme. Twelve (50%) of the respondents recommended that NCR SARE efforts dealt with maintain and enhance its farm and farmer focus. Projects must be firnded that move results on to farms as part of the project. (W106) Reviewers need to fund only projects that can be applied to farms. To some extent research projects must mirror ideas from producer grants. (W114) By having the work done on real farms and having farmers be the primary educators. (M103) Profit incentives; lifestyle incentives. (ND03) 107 Measuring profitability needs to be a program priority. In my narrow experience, this is a very under-utilized management tool. I am less interested in research projects that measure general profitability of one system compared to another under a controlled research context, but how we help farmers measure profitability themselves as they compare changes and innovations. I suggest as these farmers with outside financial review become case studies/media stories to promote promising options. (K801) Table 4-11: Recommendation Themes for Practicality (K) Recommendation Themes N Percentag; Emphasize farm/farmer focus 12 50% Use media 4 17% Not a problem 2 8% Basic research 1 4% Reach mainstream l 4% Stay focused 1 4% Integrate work 1 4% Increase central control 1 4% Redesign 1 4% Total 25 Total Responses = 24 Four (17%) of the respondents suggested focusing on media use. Better exposure in the farm periodicals. Attendance at trade shows, etc. (N E10) Two (8%) of the respondents indicated that this was not a problem. Recommendation Themes for Greatest Barriers and Critical Components (G) Twenty-five respondents made 27 codable responses that dealt with problems related to greatest barriers and critical components. These codable responses were 108 organized into 13 Recommendation Themes. Table 4-12 displays Recommendation Themes for greatest barriers and critical components and the number of respondents whose responses are included in each theme. Table 4-12: Recommendation Themes for Greatest Barriers/Critical Components (G) Recommendation Themes N Percentag Emphasize diagnosis 5 20% IT Don’t emphasize policy 4 16% Emphasize farms and farmers 4 16% Keep policy emphasis indirect 3 12% Emphasize community 2 8% Forces beyond SARE 2 8% Emphasize policy 1 4% Definitions 1 4% Emphasize empowerment 1 4% Work with Extension, Land Grants 1 4% Reach mainstream l 4% Media 1 4% Emphasize reviewer selection 1 4% Total 27 Total Responses = 25 Five (20%) of the respondents recommended more attention to diagnosing problems within SARE. I think there is room to do more detailed ethnographic-type research on decision strategies and farmer views of their research priorities. (W112) Identify the barriers carefully. Train the technical and experts advising and 109 providing technical assistance to address the barriers directly with farmers. Include communications expertise to help improve communication that impact. (M11 0) I don’t think we’ve done enough to identify barriers and why people change or don’t change. We should put more emphasis on this in our calls for proposals. (MNO6) All three paragraphs of potential problems associated with this question reveal biases. I read them to say 1) if we can’t educate people to do what we think is the right thing, maybe we can force them to do it; 2) we know the right thing to do but we can’t convince most of the farmers, and 3) seed and chemical companies and major farm organizations don’t want agriculture to be sustainable. Philip Kottler, Northwestern University’s guru of strategic marketing, lists clews that can be used to identify an organization-centered organization (prOgram) as opposed to a customer-centered organization (program). An organization-centered organization 1) regards the offering as inherently good, regardless of how well it is accepted by customers, and 2) attributes its failures to ignorance on the part of customers and clients. When a lot of the customers don’t adopt something, it may be because it’s not practical for them or relevant to their Situation. Trying harder to sell it won’t change that. (IL08) Four (16%) of the respondents recommended against emphasizing policy issues within NCR SARE. Policy advocacy is not worth jeopardizing SARE funding. SARE’s role is not to go head-to-toe with evil agribusiness. Even Congress aside, SARE doesn’t have the resources to win a direct confrontation. Think Jujitsu. Go around. Give birth to something better. (K801) Policy seems like a black hole for research. Keep funding alternative research. (N E1 1) Three (12%) of the respondents recommended including policy, but with an indirect emphasis. The educational programs must be a process that leads to policy change. The components of the educational program must be organized in a way that the end product of the process is policy change prompted by individual or group action. (M112) Four (16%) of the respondents recommended focusing on barriers and critical 110 components related to farms and farmers. Fund projects that are well-connected with sustainable agriculture farmers and those farmers you want to influence. (W106) Producers need to own the need (?) to change or see an opportunity to improve. It is as much an ethic as a result of critical thinking of various inputs, is it not? (ND03) Two (8%) of the respondents recommended focusing on barriers and critical components related to communities. The greatest barriers are at the community-social level. SARE can promote itself as a viable alternative to those wishing to enhance their quality of life or standard of living. (NE10) Two (8%) of the respondents made suggestions related to NCR SARE’S position within outside forces. Recognize that things are changing. Identify a role within the context of change. (IN07) Macro changes in the structure of agriculture are drawing most farmers in an unsustainable direction — even if they have the information from SARE on how to farm more sustainably. That’s not to say SARE could reverse those macro changes. (IA01) Recommendation Themes for Grounding in Research (L) Twenty—three respondents made 24 codable responses that dealt with problems related to grounding in research. These codable responses were organized into 10 Recommendation Themes. Table 4-13 displays Recommendation Themes for grounding in research and the number of respondents whose responses are included in each theme. Eight (35%) of the respondents recommended that NCR SARE include a range of research approaches and synthesize research results. 111 There is an undercurrent of suspicion concerning “replicated, reductionist research methods by a land grant university” among some who support sustainable agriculture practices. There are good reasons for replication in experiments. The reductionist approach, which is a completely different issue, is OK as long as there is some synthesis of the findings of reductionist research. In fact, the combination of reduction and synthesis is a powerful approach, and probably the only approach, to improving systems. Sound research techniques, including replication, randomization, blocking, etc., are not unique to land grant universities. They are sound approaches to experiment design in any situation and essential in most field experiments. They are used to assure the validity and relevance of experimental measurements and results. (IL08) Maybe a better guiding principle besides “being grounded in research” is to be grounded in “meaningful evaluation”. (K801) What are the social science methods of research that are very insightful? Can some of those processes of research be helpful? (M110) Table 4-13: Recommendation Themes for Grounding in Research (L) [Recommendation Themes N Percentag; Combine and synthesize research approaches 8 35% Emphasize farm/farmer focus 3 13% Facilitate networking 3 13% Basic research 3 13% Not basic research 2 9% Partner 1 4% Not reductionist 1 4% Research to influence colleagues 1 4% Focus more on education than research 1 4% Innovations 1 4% Total 24 Total Responses = 23 Observational findings have value. At some point there needs to be replication in order to develop principles. Current SARE on farm research is a “a stand alone 112 event.” There are methods that don’t lead to reductionist science — partner to gain this expertise. (IN07) Three (13%) of the respondents commented on the importance of maintaining a farm and farmer focus for SARE work. The original mandate from Congress was to find viable farming practices that would sustain family farms. As we saw that responsibility, we looked into whole farm systems. Secondly we were mandated to involve ag practitioners and that created generalization of research. (SD07) Three (13%) of the respondents recommended that NCR SARE emphasize basic research. Get some good researchers to visit with AC and TC — folks like Ben Stinner and Dick Harwood, about the need for basic research. (OH06) Major problem in getting basic & systems work funded is the needs of producers no_w which drive the Technical and Administrative committees to support less basic/systems work. It is more costly as well. 80 more scientists on the committees, carefully selected for disciplinary representation. (ND03) Two (9%) of the respondents recommended that NCR SARE not emphasize basic research. Observational findings have value. At some point there needs to be replication in order to develop principles. Current SARE on farm research is a ‘a stand alone event.’ There are methods that don’t lead to reductionist science — partner to gain this expertise. (IN07) Recommendation Themes for Measuring (J) Twenty-four respondents made 21 codable responses that dealt with problems related to measuring. These codable responses were organized into six Recommendation Themes. Table 4-14 displays Recommendation Themes for measuring and the number of respondents whose responses are included in each theme. 113 Table 4-14: Recommendation Themes: Measuring (J) Recommendation Themes N Percentag Agree on goals 6 25% Don’t over-quantify 6 25% Sustainability a process 4 17% Participant evaluation 3 13% Community focus 1 4% Need symbolism 1 4% Total 21 Total Responses = 24 Six (25%) of the respondents suggested goal clarification and importance of measurement. SARE needs its definition, but needs goals more than a definition. The committee each year can revisit this but must set goals for the program. (NEl 1) We need to agree on a set of indicators of sustainability and then start monitoring changes relative to these indicators. (MNO6) Measuring a 1-5 year project just reinforces the linear cause-effect logic so destructive of holistic change. (M105) Six (25%) of the respondents recommended that NCR SARE not over-quantify program results. Bean counters need numbers, but the more precisely you quantify these effects, the more meaningless are the numbers. (IAOI) We are usually too quick to want measurements, which leads us to usually be too quick to judge. (M112) Four (17%) of the respondents suggested that sustainability is a process. It’s a moving target. There needs to be baseline information and an agreed-upon set of goals. (IN07) 114 Three (13%) of the respondents suggested that NCR SARE involve participants in measuring. We can only be sure we have had an impact by the endorsement of those producers who have been directly impacted by a project. (W114) My approach is to help farmers, the ultimate decision makers on local land use, to measure their own sustainability. Maybe the evaluation should be the number of farmers who are intentionally measuring their own sustainability in an intentional, disciplined way. (K801) Recommendation Themes for Sustainable Agriculture Practices(1) Twenty-two respondents made 19 codable responses that dealt with problems related to sustainable agriculture practices. These codable responses were organized into seven Recommendation Themes. Table 4-14 displays Recommendation Themes for sustainable agriculture practices and the number of respondents whose responses are included in each theme. 115 Table 4-14: Recommendation Themes: Sustainable Agriculture Practices (1) Recommendation Themes N Percentag Sustainability is a process, not a result or practice 9 47% Sustainable includes conventional, organic practices 3 16% Clarify expectations 3 16% Sustainable does not equal organic 1 5% Facilitate networking 1 5% Don’t depend on Extension/Land Grant 1 5% Partner 1 5% Total 19 Total Responses = 22 Nine (47%) Two respondents suggested that sustainability be regarded as a process rather than a product or practice. It’s impossible to define sustainability in terms of a “practice” — as it is impossible to define health or family as a “practice”. Give it up! We need to define “ecological practices and distinguish them from “industrial’ ones.” (N D05) Do not confuse practices, systems and concepts!! Sustaining deals with the temporal dimensions of something! (M103) “Sustainable” is by definition a process rather than a result. (NEl 1) Perhaps the focus needs to be on defining principals of sustainability rather than on organic, etc. (M110) Sustainable per ce? Or more sustainable than before? Suggest need to not stereotype sustainable ag. (N D03) Three (16%) of the respondents suggested that sustainable agriculture includes practices regarded as “conventional” and as “organic.” Both organic and conventional systems are dynamic. Sustainable is also a dynamic concept, so it is likely a system would include components of each 116 approach. (SD01) Three (16%) of the respondents suggested that NCR SARE work to clarify expectations for projects. SARE, both on NCR and national levels needs better/clearer picture of what sustainable agriculture will/does look like. A better, clearer, more readily communicated goals and indicators. (M119) 117 CHAPTER 5 CONCLUSIONS AND IMPLICATIONS This chapter begins with a summary of the study. The next section summarizes conclusions about the emergence of stakeholder consensus about evaluation issues. problems and recommendations. Additional sections address recommendations from this study, implications and suggestions for future research. Summary of the Study Although stakeholder involvement in program development is widely believed to be important, a search of the literature produced relatively few tested models for meaningful stakeholder involvement in program evaluation. This is the case even within the area of sustainable agriculture, where the basic tenets support and possibly demand that stakeholders be involved. This research was undertaken to determine whether it was possible to construct a meaningful participatory framework that would allow stakeholders to be actively involved in defining evaluation issues and the problems associated with these issues. Respondents were stakeholders of the NCR SARE program. The approach was qualitative and holistic, designed to maintain participants’ individuality and allow their voices to inform ongoing discourse and discussions about evaluation within NCR SARE specifically and the sustainable agriculture community in general. Three relevant areas of literature were reviewed -- program evaluation, Delphi 118 technique and sustainable agriculture. An examination of program evaluation literature focused on efforts to evaluate educational programs, evaluating broad-aim programs, evaluation utilization, and participation in evaluation. Literature reviewed about the Delphi technique dealt with its definition, process and format, panel selection, limitations, advantages, and use of expert judgment. A review of sustainable agriculture literature focused on the concept’s background, attempts to define sustainable agriculture, levels and processes within sustainable systems, dimensions of sustainability, and the Sustainable Agriculture Research and Education program. NCR SARE stakeholders were defined as current and past administrative council members, technical committee members, state coordinators, and grant recipients and current staff members — 431 people. Invitations to participate in the research were mailed to a sample of 144 people drawn from these groups based on nominations from NCR SARE leaders and balancing for both group and state representation. Eighty-five people agreed to participate and 67 followed through to respond to one or more of the survey rounds. . A series of three Delphi surveys was used, with the second and third surveys based on results of the earlier surveys. The first survey asked respondents to identify two pressing issues or questions that they felt must be considered when evaluating sustainable agriculture research and education. The second survey summarized those issues/questions. It asked respondents to select the two evaluation issues that they believed would be most important for NCR SARE over the next five years and to describe any potential problems that they anticipated NCR SARE would face in dealing 119 with the selected issues/questions. The third survey summarized the issues and problems and asked respondents to suggest ways in which NCR SARE could overcome any of the problems. An inductive approach and a form of content analysis were used to analyze the content of each survey round. The data were sifted, examined, combined, reduced and interpreted as they were collected. Data were examined for regularities, patterns, and major ideas which were used for clustering and coding. Conclusions: Emergence of Stakeholder Consensus Results of this study showed that it is possible to construct a meaningful participatory framework to allow stakeholders to be actively involved in defining evaluation issues and the problems associated with those issues. People from each state in the region and every identified NCR SARE stakeholder group responded, using their own words and providing their ideas. Because surveys in rounds two and three were based on responses to the earlier surveys, they were responding to one another, reacting to and building upon other stakeholders’ ideas. After three survey rounds, respondents’ ideas began to show consensus in several areas; several other areas showed little consensus among respondents. More specific conclusions, organized around findings related to each of the subsidiary research questions, also support this general notion. Consensus is defined as “group solidarity in sentiment and belief; general agreement; the judgment arrived at by most of those concerned” (Webster’s new collegiate dictionary, 1974). For this research the concept of consensus was 120 operationalized as Issue/Question Clusters and Recommendation Themes selected by 15% or more of the respondents. This relatively low threshold was chosen because of the extremely open-ended nature of the survey questions and the absence of a specific survey round to confirm or reject agreement on issues. In addition, the content analysis process included looking for areas of possible agreement or linkage across the Issue/Question Clusters. The issues, problems and recommendations that emerged from this research were highly consistent with the basic tenets of sustainable agriculture found in its literature and reviewed in Chapter 2. Evaluation Issues This primary question incorporated several subsidiary questions. One set of subsidiary questions dealt with evaluation issues. Are there clearly recognizable issues regarding evaluation in sustainable agriculture? If so, what are the issues? Do evaluation issues in sustainable agriculture clearly group in major themes? Analysis of the 106 pressing issues/questions originally identified by respondents in the first Delphi survey resulted in 14 Issue/Question Clusters. Each Cluster represented a major issue theme, with the essence of each theme expressed by a short identifying phrase that was displayed in bold type. In the second Delphi survey, respondents were provided an opportunity to add any additional pressing issues that they thought were missing from the list. Only one respondent added an issue, indicating that the 14 Issue/Question Clusters did a reasonable 121 job of describing the range of issues for survey respondents. The second Delphi survey also asked respondents to select the two Issue/Question Clusters that they thought would be most important for NCR SARE over the next five years. Moderate stakeholder consensus did emerge around several issues. The number of respondents selecting each Cluster provided one indication of consensus about its importance. Twelve of the 14 Issue/Question Clusters were selected by two or more respondents. Because each respondent was constrained to select only two of fourteen options, Issue/Question Clusters selected by at least 15% of the respondents were deemed to represent reasonable consensus. Following are the Issue/Question Clusters in order of importance to the respOndents, with the percentage of respondents selecting each cluster as one of their top two: F. To what extent are NCR SARE projects analyzed for findings and lessons learned? How well is data from SARE projects translated into useful information that is specifically targeted for and disseminated to farmers, lenders, consumers and policy makers? To what extent has NCR SARE developed peer-to-peer structures for information distribution? (30%) E. To what extent does NCR SARE address a series of whole systems questions, including basic and applied research and education in the context of the system in which it will be used? To what extent does each project include (with varying emphasis) components of inquiry and evaluation of all three dimensions of sustainability —— economic, environmental and social — and include both social and natural sciences. To 122 what extent does NCR SARE address root causes, not just symptoms? (28%) M. To what extent do NCR SARE projects contribute to a new agriculture and a new way of producing, consuming and being versus making a growth-oriented industry more socially and environmentally sensitive? How deep is the sustainable agriculture research agenda? (21%) C. Sustainability has to do with the long term. It may take more than a year or two (the length of NCR SARE projects) to determine project effects on long-term sustainability. (19%) H. To what extent has NCR SARE really made agriculture more sustainable — farmers more profitable, the environment cleaner, quality of life/rural community viability improved? What are SARE’S specific, measurable impacts in these areas? (16%) D. Although sustainable agriculture extension may include technology transfer as a component, its overall objective is to empower people to develop and access the information they need to understand and solve their own problems, to consider a larger set of alternatives. Evaluation of such a program cannot be done in terms of actions and outcomes, but must be accomplished in terms of changes in understandings and abilities. Only the people SARE programs are designed to serve can tell us how well the programs are working. To what extent have farmers begun to routinely question the rationale behind consultant recommendations? Are farmers starting to ask, “How can I get long- term control of this situation?” Are they developing farm plans that address their quality of life questions? Are they asking for more than one opinion when seeking advice on an 123 emerging production question? (16%) A. Who is currently involved in NCR SARE programs and are they the right people? How can current SARE participants be characterized? Does/should SARE focus on small to moderate-size family-Operated farms? To what extent is SARE relevant to mainstream agriculture? To what extent is SARE helping to develop people who can lead change? To what extent is SARE helping to link farmers, researchers and extensionists in meaningful ways? (16%) B. To what extent does NCR SARE include a diversity of approaches, practices and systems in recognition that there is no single right way to practice sustainable agriculture and to take into account the diversity of agriculture, climate and social structure across the region? (14%) K. NCR SARE projects and information must be practical. To what extent are NCR SARE projects designed to work on real farms, taking account of economic, time, scale issues? How quickly does NCR SARE move research to practical application on farms? To what extent do projects operate at reasonable and competitive levels of productivity, profitability and net income? (12%) G. To what extent has NCR SARE identified the greatest barriers preventing farmers from adopting sustainable practices? The most critical components that cause farmers to change to more sustainable practices? (12%) L. To what extent are NCR SARE projects grounded in research? (7%) J. Challenges exist in measuring sustainability. To what extent does NCR SARE measure and evaluate incremental steps that show progress in the direction of 124 greater sustainability? How much can we attribute measurable/measured changes to SARE? I. What is considered to be a sustainable agriculture practice? To what extent are appropriate organic and conventional methods incorporated into sustainable systems? (2%) N. To what extent should NCR SARE assure that an outside evaluator visit or otherwise review each funded project? (0) O. Summarize and share information. (1%; written in). Evaluation Problems A second set of subsidiary questions dealt with problems related to these evaluation issues. Are there clearly definable problems associated with sustainable agriculture evaluation issues? If so, what are the problems? Is there consensus regarding the problems associated with sustainable agriculture evaluation issues? Are there differences in the types of problems associated with each sustainable agriculture evaluation issue? Respondents listed problems associated with most of the evaluation issues identified. Because respondents were asked to identify problems associated only with the issues they selected as most important to NCR SARE over the next five years, the issues that were seen as most important also tended to have more problems identified. Most comments received on the Round Three survey were in direct response to the problems it listed. In only a few cases did respondents indicate that there was “no problem” and 125 those instances were scattered. Evaluation Suggestions and Recommendations A third set of research questions dealt with evaluation suggestions and recommendations. What are the suggestions and recommendations? Is there consensus regarding the suggestions and recommendations associated with sustainable agriculture evaluation issues? Are there differences in the types of suggestions and recommendations associated with sustainable agriculture issues? In this research, respondents were asked to make suggestions and recommendations only after they had first been asked to identify pressing evaluation issues, to select the most important of those issues, and to describe problems in dealing with those important issues. Respondents were asked to use those issues and problems which they had already identified as the context for their recommendations and suggestions. A number of the Recommendation Themes in the research findings showed consensus. They included the following: 1. Clarify project expectations to get more useful information. Not only was the issue of useful information rated as most important by respondents, but nearly half (45%) of those who made recommendations about that problem suggested that NCR SARE work to clarify project expectations to help address this issue. This Recommendation Theme was also repeated at lower response levels in suggestions about whole systems (14%) and sustainable agriculture practices (7%). 126 2. Target portions of SARE funding to accomplish important objectives. Over a third (41%) of respondents who made recommendations related to useful information wanted NCR SARE to target some of its funding to collect and disseminate such information. Targeted funding was also suggested by a third (33%) of those who made recommendations about determining specific, measurable impacts. 3. Increase follow-up efforts after projects end to better capture outcomes. Over a quarter (28%) of respondents making recommendations about useful information suggested ways for NCR SARE to improve its project follow-up efforts. 4. Better utilize media to disseminate information from projects. About one fifth (21%) of respondents who made recommendations related to useful information suggested ways in which NCR SARE should take action to follow up after completion of funded projects to extract additional useful information from the projects. 5. NCR SARE should become more active in disseminating information. Respondents who made recommendations related to useful information made suggestions about more extensive media use (19%) and ways to facilitate networking (11%). These Recommendation Themes were echoed in recommendations related to other issues and they support the notion that NCR SARE may need to increase its involvement in disseminating results of the projects which it funds. 6. Include some deeper, redesign efforts in NCR SARE funded projects. Among respondents who made recommendations related to a new way of thinking, producing, consuming, over half (54%) suggested ways for NCR SARE to move into deeper efforts that go past substituting components in conventional systems to designing 127 new systems. This idea was also reflected to some extent by 18% of the respondents who made recommendations related whole systems questions and who recommended NCR SARE emphasize whole systems questions. It appears again in recommendations related to greatest barriers/critical components, where 18% of those responding made suggestions that NCR SARE put more emphasis on problem diagnosis. And 19% of those making recommendations about specific, measurable impacts spoke to quality of life issues, which can also be seen as a way of deepening SARE work. 7. Provide some funding for projects longer than two years. Half (50%) of the respondents who made recommendations related to long-term sustainability favored NCR SARE shifting a portion of its funding to support projects longer than the current one or two years. An additional 15% recommended using the existing option of renewing effective projects accomplish the same result. 8. Pay increased attention to reviewer selection. This Recommendation Theme appeared in recommendations around several issues. Among respondents who made recommendations related to a diversity of approaches, practices and systems, over a third (37%) made suggestions that dealt with improving processes for reviewer selection. Reviewers might include members of the administrative council, members of the technical committee, outside reviewers, and evaluators. This notion was also reinforced by the quarter (26%) of respondents on this item whose suggestions highlighted the importance of diversity within NCR SARE. 9. Clarify the overlapping but distinct identities of organic systems and sustainable systems. Among respondents who made recommendations related to a 128 diversity of approaches, practices and systems, 30% suggested that NCR SARE maintain a distinction between sustainable and organic agriculture. Among respondents who made recommendations related to sustainable agriculture practice, several (15%) suggested that NCR SARE acknowledge that sustainable agriculture can and does include practices considered “conventional” as well as practices considered “organic.” 10. Maintain the historical SARE focus on small to mid-sized farming systems, community-based food systems and people who are already committed to a more sustainable future. Among respondents who made recommendations related to who is currently involved, nearly half (48%) made recommendations to stay focused, to work with the people who are interested, committed to change, and who are not well- served by other programs. 11. Keep the NCR SARE focused first on people. Among the respondents who made recommendations related to empowering people, 29% made suggestions that NCR SARE support a facilitative approach to learning, and 29% made suggestions that NCR SARE support a qualitative focus. These two Recommendation Themes keep people at the center of the systems and remind NCR SARE that people must be its primary focus. This notion is further supported by respondents who made recommendations related to practicality. In that group, 43% made suggestions that emphasized the importance of a farmer/farm focus for NCR SARE. Of the respondents who made recommendations about specific, measurable impacts, 19% spoke to quality of life issues. 12. Agree on general goals, but don’t try to over-quantify NCR SARE results. Among respondents who made recommendations related to measuring, a third (3 3%) 129 made suggestions about clarifying or agreeing on goals, while over a quarter (28%) warned against over-quantifying. The Recommendation Theme that sustainability is a process (not a result or practice) appears in recommendations related to measuring (17%) and to sustainable ag practice (35%). Also, 15% of those who made recommendations about There were also several Recommendation Themes in the findings that Reflected little or no consensus. They included: 1. Two distinct and opposing Recommendation Themes regarding NCR SARE’s relationship with land grants and specifically with Extension emerged. Recommendations to work more closely with land grants/Extension were offered to deal with problems related to useful information (31%), greatest barriers/critical components (4%), and empowering people (17%). But at the same time, recommendations not to depend on land grants/Extension were offered to deal with problems related to useful information (10%); new ways of thinking, producing, consuming, being (4%); whole systems questions (4%); sustainable agriculture practices (5%). NCR SARE stakeholders seem quite divided about the desirability of working closely with land grant institutions and Extension. One recommendation seemed to bridge these two opposing perspectives by suggesting that SARE provides an opportunity to reinvent Extension, based on an empowerment model rather than a technology transfer model. 2. Recommendation Themes about the appropriate research focus for SARE were very scattered and reflected little consensus. Respondents who commented about whole systems research included 11% supporting basic research, 14% opposing 130 basic research, and 11% offering options to basic research. Eight percent of respondents commenting on long-term sustainability and four percent of those commenting on practicality indicated support for basic research. Respondents commenting on grounding in research also split their comments, 13% in support and 9% in opposition. The lack of consensus about research approaches is not surprising given the multi- and interdisciplinary nature of much sustainable agriculture work and the diverse backgrounds of the stakeholders who responded to the surveys. 3. There is some tension between the Recommendation Theme of staying focused on those currently involved in NCR SARE and Recommendation Themes that deal with diversity as well as those that support active outreach to mainstream agriculture. The notion of staying focused was expressed by 46% of those commenting on who is currently involved. But other comments endorsing diversity (30% of those commenting on diversity of approaches, practices and systems) and reaching out to mainstream agriculture (14% of those commenting on who is currently involved) may pull NCR in a different direction. 4. Respondents in this study appear to conceptualize evaluation in a very broad and inclusive manner. The saw evaluation as selecting projects, keeping them on track, harvesting their useful information, estimating their costs and benefits, determining their long-term impacts, and more. While some degree of evaluation certainly does enter in each of these facets of program operation, it may be valuable for NCR SARE to identify a narrower focus for its formal project and program evaluation efforts and to come up with terminology that helps people distinguish one particular piece of evaluation from another. 131 Recommendations Based on conclusions drawn from the results of this research, the NCR SARE Administrative Council and staff may wish to consider the following recommendations: 1. Clarify NCR SARE expectations about projects developing and sharing useful information, making those expectations clear in invitations for proposals and project agreement, and increase follow-up efforts to better capture project information and outcomes. 2. Target portions of SARE funding to focus work towards critical objectives. This is certainly not a new idea. NCR SARE has previously targeted funding for quality of life work, marketing work, and other priorities and objectives. 3. Continue emphasis on actively disseminating NCR SARE project information. Having a full-time communications staff position has helped this process. To move it along even more, the current NCR SARE Sustainable Agriculture Network representatives might consider adding some people from outside the Administrative Council and staff to form a regional communications committee. Involving the state NCR SARE Professional Development Program liaisons may also be a useful option. 4. Include in the NCR SARE project portfolio some projects that are designed to extend beyond two years and some projects that strive for system redesign. It is likely that the redesign efforts may require longer than two years. This could be combined with recommendation number two and a specified amount of funding be earmarked for longer 132 term system redesign project proposals. 5. Develop a forum to continue discussing issues of program focus. This may be an especially important area for continued NCR SARE discussion and debate. With its small funding base relative to other research programs, NCR SARE must develop some areas of focus if it is to make an impact. Developing that focus while honoring diversity and influencing the mainstream will be difficult but must be addressed by the Administrative Council, preferably with broad outside involvement. The results of this discussion have major implications for several subsidiary issues — how closely NCR SARE should work with land grants and Extension, what types of people should be recruited to the Administrative Council and Technical Committee and what should the nomination and decision process be, what criteria should be used to established desired outcomes and review proposals, etc. 6. Address the relationship between organic systems and sustainable systems. 7. Help NCR SARE stakeholders and reviewers to better recognize excellence in many types of research and effectively match research approaches with desired outcomes. Suggestions for Further Research and Related Work Several changes should be considered if this research approach is used in the future. This series of surveys could have been timed better, with the request to participate sent in early November, and the three Delphi surveys sent in early January, February and March. Such a shift might tend to increase farmer participation and decrease 133 research/extension participation, but hopefully increase the total response. Adding a fourth round to allow respondents to confirm or deny the conclusions drawn from the first three rounds would be helpful. Using a modified Delphi approach over an extended time to help NCR SARE (or other groups) understand stakeholder issues and interests and handling the entire process with computers — e-mail and/or web-based — would reduce costs and speed the process. Although it would be important to be sure that a move to computer technology did not eliminate important voices or perspectives, recent data concerning computer and Internet use indicates that such an approach should be feasible in the near future. Not only would computer technology eliminate the delay caused by mailing out surveys and mailing back responses, but it would also eliminate the time-consuming and potentially error producing transcription process to speed analysis and feedback. Another area for future investigation is for NCR SARE to build upon its existing database of past and current participant lists to develop and maintain e-mail lists for each of those groups. Each individual would be contacted and informed that NCR SARE would periodically be using the list to send information (calls for proposals, announcement of new information available, etc.) and to occasionally request information. Then, perhaps two or three times a year. the stakeholder groups could be polled about issues the Administrative Council is facing or asked for information that is needed by SARE. This could provide part of the follow-up that is needed from a fairly broad group in a fairly cost-effective way. This function could be handled from either the regional office or contracted out. 134 A potential follow-up project would be to use the Results Mapping approach to evaluate several NCR SARE-funded projects. This effort might entail: (a) selecting projects that have involved more than just a few people; (b) selecting 12-15 participants from each program whom the project director believes represents the best work of the program; (c) getting these participant narratives or “stories” that explain how the program first got involved with the participant, what actions it subsequently initiated to promote participant grth and change, and how the participant responded to these actions; (d) mapping and scoring the stories; and (e) reviewing with project directors and NCR SARE Administrative Council the results. It would also be valuable to examine computer-based tools such as concept mapping and virtual polling to further support and encourage stakeholder dialogue. Implications This research has implications at two levels. One deals specifically with NCR SARE and its program evaluation efforts. The second level of implications deals more broadly with use of the Delphi technique for stakeholder involvement. Specifically within NCR SARE, findings and recommendations from this research, which have been provided in some detail above, have potential to (a) help shape NCR SARE evaluation framework; (b) provide background information for NCR SARE Administrative Council and regional staff to inform budget and funding decisions, prioritize meeting discussion topics; (c) provide background information for SARE state 135 PDP liaisons; and ((1) provide background for national SARE program staff. With regard to its broader implications, the Delphi technique offers considerable potential to increase stakeholder involvement in many types of programs. It provides a qualitatively different type of input than organizations frequently receive via results of polls or votes, because the input is more in the voice of the stakeholders and less in the voice of the researcher. It does not displace more controlled, more quantitative methodologies, it supplements and balances them. Two of the three surveys used in this research invited very open-ended responses from stakeholders. Many of the respondents provided significant, thoughtful, thorough observations in response to this invitation. Some commented that the survey questions and others’ comments had really caused them to think deeply about the issues. Not all stakeholders of any organization are likely to wish to participate in the Delphi process. But offering the option is one way to let people share ideas at a relatively low cost of their time and money. It is a strategy consistent with empowerment, participation, and inviting people to become involved at whatever level they wish. It is an approach that has considerable potential for continuing and expanded use. 136 APPENDIX A LETTER OF INVITATION AND RETURN CARD 137 E Department ol Mutual It Extension Education Cantu for Evaluative Shite: Agriculture Hal Michigan State Umversrty East Lansmg. Michigai 48824-1039 5317—3536580 FAX: 517-353-4981 fhe Mariam State Uan IOU rs Institutional Diversity Excellence in Aaron MSU is an Merriam-action. aerial-W0 militia) MICHIGAN STATE UNIVERSITY June 30, 1999 To: You may be aware that the North Central Region Sustainable Agriculture Research and Education program (NCR SARE) has asked Michigan State University's Agriculture and Extension Education Center for Evaluative Studies to develop an evaluation framework to improve SARE’s ability to document and share the impacts of its work. As part of this project,a panel of NCR SARE leaders nominated you to participate in a research study to better understand evaluation perspectives and preferences of people involved with NCR SARE. You are among a group of NCR SARE current and former administrative council members, technical committee members and grant recipients who are being invited to participate in this NCR SARE evaluation project. - What will you be asked to do? You will be asked to respond to a series of three surveys about evaluating NCR SARE projects and programs. After you complete and return each one, you’ll receive a summary and analysis of group responses along with the next survey. - How much of your time will it take? Although that will vary, we estimate that you will be able to respond to each survey in less than one hour for a total time commitment of four hours or less. The series of surveys will be sent over a period of about 10 weeks. - Is there any expense to you? No, there will be no money expense; all surveys will include stamped return envelopes. - Will your name be used? No, your name will not be used in any part of the study or reports of research findings. Your privacy will be protected to the maximum extent of the law. - What will you receive from this study? You will receive the ongoing analyses of each survey instrument and a copy of the final report. Your participation will also provide an opportunity for you to influence decisions about future NCR SARE evaluation strategies. . How will this information be used? The final report will be submitted to the NCR SARE Administrative Council along with recommendations for their action. Information from the surveys will be used as the basis for the report, a Ph.D. dissertation and various articles and presentations. . Whatifldioosenottoparticipateorldecidetowithdraw? Althoughwehopeyou will choose to participate in the entire study, your participation is entirely voluntary. If you choose not to participate or decide to withdraw at some point, there is no penalty or loss of benefits. If you have any questions about participants' rights as human subjects of research, you may contact Dr. David E. Wright, 517-355-2180. We hope you will agree to participate and make contributions to this evaluation effort. Please return the enclosed postcard with your answer either by US. mail or fax (517-353-4981). If you have questions, please feel freetooontactoneofus. WehopetohearfromyoubyJulyQ, andwelook forward to working with you in this study. Sincerely, ‘ ( 2 Z ‘ Murari Suvedi, Associate Professor Susan B. Smalley,Ext ensf’jSpecialist Agricultural 81 Extension Education MSU Extension 517-355-6580 or suvedi©msue.msu.edu 517-432—0049 or smalley@msue.msu.edu 138 Please circle your response and return it by July 9, 1999. You may mail or fax it (517-353-4981). YES I agree to participate in three surveys about NCR SARE evaluation. (You indicate your voluntary agreement to participate by completing and returning this card.) NO I will not participate in the project. Signature: Date: (Affix mailing label here) Please correct mailing address as needed 139 APPENDIX B ROUND ONE SURVEY 140 E W at Aerial-at I Ens-slo- Education Center lot from Mes Agatha Hail Midiioan Slate [hivesity East Lansing, Michigan 48824-1039 517-W FAX: Si 74153-4981 '\ 26 July 1999 To: Thank you for agreeing to help with this project. This is the first of three survey rounds. Each survey will include two parallel sections or tracks. Section A questions deal with sustainable agriculture evaluation issues; section B questions deal with sustainable agriculture evaluation criteria. When we receive your responses, we will summarize them and include the summary when we mail you the next survey. You’ll be able to see the responses from other participants and to comment on their ideas. This type of research is named “Delphi' in honor of the ancient Greek oracle and to indicate the strength of the shared wisdom that we hope to generate together. Please mail your response in the envelope provided or fax it to 517-353- 4981. Deadline for response is August 6, 1999. If you have questions or concerns. contact Susan Smalley at 517 432 0049 or smalley@msue.msu.edu. Murari Suvedi Susan Smalley Associate Professor Extension Specialist Agricultural & Extension Education MSU Extension 517-355-6580 517432-0049 suvedi@msue.msu.edu smalley@msue.msu.edu 141 Sustainable Agriculture Evaluation Framework Delphi Evaluation Survey #1 A. EVALUATION ISSUES. There are many approaches to evaluation and no single ‘right” way to handle it. What do you think is a pressing issue or question when you consider evaluating sustainable agriculture research and education efforts? Please describe the issue or question and why you believe it is important. If you wish to write about several issues or questions, that’s fine, but we’d rather that you tell us more about one or two issuesthan providealong list. You may usethespacebelowtorespond. Handwritingisfineifwecanreadit! A nesting issue/question that I feel must be considered when evaluating sustainable agn’cultum Imam-It and eduab’on is... - 8. EVALUATION CRITERIA. What two or three criteria (standards on which judgements or decisions may be based)d0youuiinkshouldbeusedtoevaluatetheNCRSAREprogramanditsmissiontogenerateand disseminate sound and practical information about alternative farming systems with potential to increase the sustainability of agriculture? Morflrreecn‘tefiaWtwsmubembadsformmluafionolmmmmvgramand its’mwonare... 1. Pleasemailywrresponseintheemelope providedorfaxitto517-353-4981. DeadlineforresponseisAugustG, 1999.1fyou have_questions or concerns, contact Susan Smalley at 517 432 0049 or smalley@mwemsuedu. 142 APPENDIX C ROUND ONE RESPONSES 143 NCR SARE EVALUATION ISSUES - RD I SIID ISSI IA01 R Meaningful collaboration of farmers with scientist or extensionists. Nether party should be "window dressing." Each can contribute uniquely to a research or education program. IA02E No response IA03R Where are we in sustainable agriculture? I think evaluation of SARE and related programs must be honest and ask the penetrating questions: Are the programs making a real difference, or are we fooling ourselve with glowing inner evaluations? Is the research missing the main stream? Can our audiences. the small farmers and ranchers, part-time operators of the land. and those in the niche markets. really make a difference? Is keeping these people going the object of SARE at a time when large integrated firms are the norm, when bulk commodities aim at global markets the object of much of the policies and programs of the government? How do we get messages across to the consumer. the politician? And with the current decline in funding for N60 and state programs. is there a way to pep up SARE programs. The federal SARE program can not go it alone. It needs the cooperation of everyone. lA04R No response IA05E How much of the measurable change can be attributed to sustainable ag research and education efforts versus other research and education efforts that may not have a "sustainable ag" label attached to them? Often times there are many reasons why a farmer may make some kind of change. Sustainable ag research and demonstration may have only played a partial role in the change occurring. Other programs. personal reasons, etc. may play a role and can't be separated out. Cast in point: Iowa farmers reduced their use of nitrogen on corn. but cannot point to a single effort as the primary reason for the change. IA06R No response IA07F No response IA08R No response IA09F No response IA1 0F No response IA11F No response lLOiN No response ILOZN The most important issue confronting sustainable agriculture is economics. It is becoming increasingly difficult for small and medium-sized farmers to earn a living and sustainable agriculture has to recognize that challenge and provide programs that improve the bottom line for growers. All SA efforts should incorporate economic realities. 144 SrID ILOBG IL04 ILOSR ILO6 IL07R IL08R IL10 IL11 IL12F IL13N IL14E IL15F IL16F IN01R INOZF IN03F ISSI No response In the definition of sustainable agriculture - an agriculture that is environmentally sound. economically viable and socially responsible - the question for me is how can we MEASURE if the research and education that has been funded will lead us to an agriculture that is sustainable? We have spent a lot of human and financial resources on these efforts, but have difficulties determining our progress in meeting these goals. No response No response No response No response No response No response Will the project result in providing the operator with a ”profit", or a net income greater than or equal to reaching the same ends by a non-sustainable manner? How has the research or education contributed to: 1) the farmer's bottom line/is profitable for farmers; 2) stewardship of the environmentlnatural resources; 3) rural community viability; 4) buidling partnerships between university researchers. ag agencies and educators, not-for-profits, and other stakeholders. In effect. what's being asked is what are the impacts in these various areas. If we cannot articulate the answers to the above areas, then we really can't say how effective that particular project was. Sustainable agriculture intertwines the economic. environmental and social aspects. While a sustainable research or education project may focus more on one aspect (it's unrealistic to expect to see all aspects equally addressed in a given project). there should not be a total disconnect from the other areas. Further. a main principle of sustainable agriculture research and education efforts is to involve the various stakeholders in a particular project. Again, while there's varying degrees of active stakeholder participation there should not be a total disconnect. In the definition of sustainable agriculture - an agriculture that is environmentally sound. economically viable and socially responsible - the question for me is how can we measure if the research and education that has been funded will lead us to an agriculture that is sustainable? We have spent a lot of human and financial resources on these efforts. but have difficulties determining our progress in meeting these goals. No response No response No response No response No response 145 SrID IN04 INOSF INOGF INO7R IN08 IN08R INTOE IN11R IN12R IN13E KSOlN KSOZG KSO3E [SS] No response No response No response To what extent have principles and discovery been incorporated or mainstreamed in production agriculture? The incorporation of social and ethical components. What are the long term effects on agriculture. and society in general, of getting larger and concentrating essentially all aspects of supply, production and marketing into the hands of a few? Are all of the buy-outs and mergers really in the best interests of agriculture, or in the best interests of a few stock holders? While economics are important, is short-term economic gain the only yardstick to use to measure the success of agriculture? Do we really want only a few supply companies providing inputs to a imited number of producers, who market their product by contract through 2 or 3 mega corporations? is this in the best interest of agriculture and the global supply? No response Changes in decision-maker behavior. Many programs are designed to raise awareness of alterntaives. While having people consider a larger set of alternatives is useful, changing behavior so that new alternatives are implemented is even more important. Whether a diversity of approaches, practices and systems are being included. Is there a good diversity or mix in the "portfolio" of research and education efforts? My concern is that we not get caught in a too-narrow definition of "sustainable" agriculture, such that we end up with only one way to do the "alternative" to conventional agriculture. Just as there is no single "right" way to approach evaluation (from your sentence above), there is no single ”right" way to practice sustainable agriculture. I think it's important that we have research on agricultural practices that may just be small changes from the current practices (but over many acres could amount to significant change) as well as research on practices or systems that may be very different from current ones. This inclusion of a diversity of approaches would also take into account the diversity of agriculture. climate. and social structure across the region (for example. easter vs. western Corn Belt). No response No response 1) Diffusion of innovations (reference Everett Rogers) are management innovations passed on to the broader target audience. A subject question is whether the peer-to—peer structure for information distribution set up. 2) Development of leaders for change. Does the program empower individuals who will model the desired change? 3) What works? What didn't work? Why? No response No response 146 StID K8048 KSO7R KSOBF KSOQF KSTOF MOIZE MO|9N MIOTF MIO3R MIO4R MIOSR MIOGR M|07R ISSI Are SARE research projects and activities affecting the behavior of farmers/ranchers who would not otherwise change their behavior? Are real environmental, economic. and/or social benefits attributable to SARE? No response I feel that the most pressing issue in the agricultural environment today is the economics of sustainable agriculture. The crisis situation of traditional agriculture is causing a lot of farmers to start self-evaluating themselves and their operations. It sustainable ag can be a proven solid economic lifestyle, many farmers that are trapped in the high input farming of today would gladly operate their farms sustainably if they can just convince their lenders that this is a realistic alternative. One factor to prove this to the current lending establishments is a more improved marketing system for the produce from sustainable farming. l have been operating my farm traditionally and will be able to certify organic on most of my farm next year. Kansas Organic producers does a good job of marketing my grain. however I still have part of 97's soybeans and all of 98's in the bin. Bankers today are not comfortable with sporadic marketing. No response Does this research accomplish anything or just rediscovering the wheel? Does it make sense or a lot of hot air? "What is considered to be a sustainable ag practice.” if a practice is done in association with organic farming it’s okay in some circles. However if a traditional farmer does it. it's not sustainable. I think we need to get ALL farmers working together rather than trying to pigeon-hole them. No response 00 the research projects being considered work on real working farms? Are economic factors thoroughly considered? Time issues need to be given primary considerations. No response No response Are we contributing to a new agriculture and a new way of producing, consuming and being or are we making a growth- oriented industry more socially and environmentally acceptable? How DEEP is the sustainable ag research agenda? Continuity. l.e. does the research result in education and technology transfer that is adopted? Driving forces - should SARE be "farmer driven" or "research driven“ or what are the plusses and minuses of each approach. The approach suggested must maintain a "reasonable and competitive" level of productivity (with non-sustainable practices) and must be economically as well as resource ”sustainable.” Is the long-term viability of production agriculture more assured (in all aspects) as a result of the use of sustainable techniques. strategies and technologies (as compared to non-sustainable)? 147 SrID MlOBG MITON MI11F M|12F Ml13R MI14R MIISE MI16F MI17N MITBF Ml19E MN01 ISSI No response What is the economic viability of sustainable agriculture practices compared to conventional agriculture production systems? What are the greatest barriers preventing farmers from adopting sustainable production practices? What are the most critical components that cause farmers to change production practices to more sustainable approaches? (Source of information. Extension, field days, other farmers. etc.) No response Can organic methods and traditional methods of production both play a role in sustainable agriculture systems? I believe sustainable agriculture must make room for both methods. Unless society decides to make dramatic changes and commit vast resources in the form of fiscal dollars and physical strength to the production of food and fiber. the use of synthetic crop production products can and will contribute to agricullture's sustainability. Sustainable agriculture research must be broad enough to assess the positive and negative implications of any innovative or new idea then measure the results against a set of criteria that has been developed from the most inclusive sources of knowledge. Successful research must then become part of our educational toolbox. and be shared by a varity of ways so that all have an opportunity to benefit. There are not enough funding sources for sustainable agriculture and thus the competition is tough. SARE is fantastic, however, the Land Grant Universities lag far behind in the funding of sustainable agriculture. No response We need to evaluate small, incremental steps that show progress in the direction of sustainability. One criticism I have toward sustainable ag folks is that they want an all or nothing - l.e. you are sustainable or you're not. This isn't how S.A. happens. Evaluations that show where a farmer was 3 years ago and where he is now and what he is thinking about doing is needed. You also need to evaluate over many years to see significant changes. No response Are farmers (agricultural producers) demanding alternatives? That is to say. are farmers asking for more than one opinion when seeking advice on an emerging production question? Are farmers requesting a "low input" alternative for pest control, fertility requirements. disease control? Have farmers begun to routinely question the rationale behind consultant recommendations? Are farmers starting to ask. "How can I get long term control of this situation”? Are farmers developing farm plans that address their quality-of-Iife quesfions? No response Separating activities from outcomes. ls research getting to producers and having impact? No response 148 SrlD MNOZG MN03 MNO4 MNOSN MN06E MNOQF MNTOF M001 F M003 MOO4E MOOSR ISS I No response No response No response Are we addressing a series of whole-systems questions in projects we fund? Sustainable agriculture research and education projects to date, to the best of my knowledge, have done a very good job of increasing multidisciplinary involvement of a number of groups and individuals. both at universities, and with those not often previously involved in research (Le. farmers. agency personnel. non-profit organizations, etc.) However SA research and extension.— particularly within the SARE program, appears to be "science phobic" and very much focused on applied. ”farmer-driven" and "farmer friendly” research (both useful approaches. but not in isolation - these approaches seem to be no better to me than "researcher-driven" projects. I think that if We are truly going to look at changing agricultural systems to make them more sustainable, then we need to be viewing ”systems research" in a much larger context. Basic research is needed. but it is important that it be done in the context of the system in which the technology will be used. Basic and applied research and education on everything from cell-level research and genomics, to field level questions need to be addressed. While all of these aspects need not be addressed in single projects. we have erred on the side of ignoring what basic researchers have to offer to sustainable agriculture research and education. We need to find more approaches to SA research and education than SARE has in the past. Are the things that we're trying actually making agriculture more sustainable? Are farmers more profitable. is the environment cleaner with more wildlife, and has quality of life improved on farms and in rural communities as the result of adoption practices that we believe will enhance sustainability? No response Does it improve the life of the family/individual farmer? Are farmers apt to benefit in terms of increased income or increased free time? No response No response No response No response 149 SrlD M007E MOO8F M009R M010F M01 1F NDOI E NDOZE ISSI The impact of sustainable agriculture education and research cannot be evaluated in terms of conventional quantifiable outcomes. SA extension is "not technology transfer” - although technology transfer may be a component. The objective is not to transfer research based information to people that will solve their problems. thereby creating measurable outcomes. Instead the objective of SA programming is to empower people to develop and access the information they need to understand and solve their own problems. No single program or project can be expected to accomplish empowerment. it only comes after a sufficient number of learning experiences to slowly change one's attitude from dependence. to independence. to interdependence. A program can successfully move a person toward empowerment. yet have no quantifiable outcome or action - other than some incremental change in attitude or perception. On the other hand. a single program can seemingly produce dramatic results. but only because previous learning had already brought the person to the verge of action. Even more bothersome. programs may create outcomes or change actions of individuals and do absolutely nothing to empower the person to make decisions and take actions on their own. Programs designed to promote and persuade can be evaluated in terms of actions and outcomes. Programs that educate and empower must be evaluated in terms of changes in understanding and ability. which may or may not be accompanied by outcomes or actions. The only people who can possibly know whether their understanding has been enhanced and their abilities improved are the people who now possess the understanding and ability. If we want to know whether or not our programs are working. we need to ask the people they are designed to serve. The research truly meets the criteria of being sustainable if it's profitable. environmentally sound. socially acceptable and research that can be used by large numbers of farmers (more than just a few). The research must be moved to the practical application on the farm phase as quickly as possible. We lose about 100 farmers every day. Agripreneurs can't wait 3 years for university data. They are risk takers who understand that by the time 40% of the farmers adopt any new practice. they've lost 50% of the price advantage. All research must be explained to the four major societal groups - farmers. lenders. consumers. institutions - to educate them. If they are not convinced it's in their own best interest to adopt the sustainable agriculture practice. they won't use it. No response No response No response No response How will this research or program help producers reduce the amount of pesticides used? Will this research help make sustainable producers more profitable? Is this research producer-driven or researcher driven? 150 SIID NDO3R NDO4R NDOSF NDOG NDO7E NDOB NE01 N NEOZE NE03S NEO4$ NEOSS NEOGS NE07R NEOBR NEOQR NE10R ISSI What impact has the research had or what changes have producers made to become more sustainable? Nested in this is a motivational challenge of how to encourage and support producers as they consider "changing.” The age old extension philosophy of “here is a publication or two; read it and decide for yourself is NOT effective. We need new techniques with the personal investment of extension. researchers and producers. Some kitchen table mentoring exists; we need more of that. Producers tell me “we're drowning in data. but we want useful information." A second issue is the saturation/dissemination of information from NCR SARE. No response No response No response What is it which we are really trying to evaluate? Actually focusing in on what information you want from an evaluation and what it will be used for is the most critical piece of setting up an evaluation. No response No response No response No response Outreach. Particularly in the SARE program. It seems as if we've made great strides in researching. demonstrating. and identifying "sustainable” practices and systems. However we have not transferred our knowledge sufficiently to our "end users" - farmers. ranchers. educators and other researchers. We need to put a high priority on information/knowledge transfer and then evaluate our efforts. How well have the results and data of NCR SARE projects been diseminated? No response The beef industry has moved to larger feedlots and greater reliance on grain feeding. This creates waste management concerns because the waste must be hauled greater distances and practically moves from a mentality of manure as a resource to manure as a disposal problem. The manure does not get onto the land the corn was produced on. The second concern is that forage production and utilization is much more sustainable than grain production. Beef production is more sustainable and is done in rural communities if it is forage based. No response Does this production system create household income? If not. production innovations have very little value. No response 151 SrlD NE11F NE12N NE13N NEMF 0H01F OHOZR 0H03F OHOSF OHOGE 0H07R OHOBR OHOQE 0H10 SDOlR [SS] Does it address social. economic and environmental considerations? To what extent the issue takes a holistic approach. The research or education project should address more than economics or environment or human factors. but should include components of inquiry and evaluation of each. This integrative approach is a difficult one for our tradtiional methods of conducting research. but many of our standards for success are based on incomplete consideration of the Whole that is affected by our activities. A corollary to this is that research and education must address root causes of problems. not symptoms. For example research into water use efficiency could address plant adaptation to the environment and soil moisture holding capacity rather than irrigation structures. No response No response Sometimes university personnel might try to duplicate an organic farming system on university property using very small plots. While this will provide much needed technical information. the plots may be so small that they have no relationship to a real farm or the type of machinery used on the farm. Many times on-farm research is much more practical. No response No response ls there farmer involvement present? Are we changing attitudes of the "trainees" (Extension Agents and NRCS staff)? Are our efforts grounded in research-based knowledge? No response No response Sustainable agriculture means an agriculture that endures over a long period of time. SARE research grants are for a period of 1-2 years. There is a discrepancy here and short- term research results can logically be questioned as to their validity over the long term. I'm not sure how to deal with this. but we have to recognize that when we are considering farming systems it probably takes more than 1-2 years for many parts of the system to reach new equilibrium. No response The decline in number of small-moderate sized FAMILY- OPERATED farms. Techniques. concepts. systems that enable these types of farms to reduce production costs and provide a reasonable standard of living should be a major portion of proposed projects. Projects should emphasize holistic approach and include BOTH social and natural sciences at initiation of study. 152 SrID soozn SDO4E 8005 $006 SDO7F SDOB W|01 N W|02 WI03F WIO4N WIOSR WIOGR WIO7E ISSI In evaluating research efforts and education. I think we need to ask two questions. Were the project goals appropriate for the goals of the SARE program? Did the project accomplish its goals? The first question is asked before funding and should be periodically reviewed. The second is reviewed with progress reports and evaluated from the fimal report. Has the research been done? Some of the producer grants are just a demonstration of research published in the 50's. Some of the sustainable ag research is linked so close to organic and doesn't represent a broad enough market. If everyone were to adopt some of the practices. the niche market would be destroyed. No response The actual PURPOSE of the research and that that purpose is compatible with sustainability. I felt that members of the Council needed to visit each of the projects funded during the course of the research. When I say council members it would mean 2 or 3 people. and then they would report to the whole group their findings. This would be the evaluation of the effectiveness of the P.l.'s in accomplishing their goals as set out in their application. For me this was not to be a critical review but a learning experience for all involved. We needed to look at the council‘s own effectiveness at selection of projects and whether we wasted time and money to accomplish our own personal goals. No response No response Will the research make it more or less likely that innovative. dynamic young people will become motivated to enter agriculture? No response No response No response A pressing issue in my mind is whether our public investment in sustainable agriculture has made a difference in any number of ways: 1) Awareness/knowledge of how alternative production and marketing systems can influence agricultural sustainability; 2) Output in the form of publications. educational materials, formation of learning groups. etc.; 3) Impacts on the acceptance of the outputs produced in these projects by university faculty and administrators as well as journal editors: 4) Impacts on the research/extensicn/instruction programs of university personnel who have been involved in research/education efforts; 5) Impacts in the form of changed practices by farmers involved in research/education effrorts or by farmers influenced by these efforts. Evidence of long-term sustainability. Research and education projects are usually one or two year research trials. There is a need for longer term research projects that will lead greater adoption and proven results. 153 StID WI08N WIOQR WI12R W|13 WI14F ISSI Sustainable agriculture research and education efforts should be evaluated in part by the following questions: How have the efforts contributed to the practice of sustainable farming methods that emphasize agroecological management. l.e. management of inter-relations of crops (through time). animals. insect pests and beneficials. soil organisms. and other plant species to reap the greatest benefit both to the surrounding environment and to profitability for the farmers? 2) Have the efforts translated into actual adoption of alternative practices in production. processing or marketing of agricultural prodeucts? How and why? 3) Who have been the primary audiences for the research and education efforts? Who benefits most? Who loses? How have these audiences been involved in each stage of the research and education eflon? No response No response No response No response 154 APPENDIX D ROUND TWO SURVEY 155 North Central Region Sustainable Agriculture Research and Education Delphi Evaluation Survey #2 C:\Myfiles\ncrsare-survey2.wpd 1. EVALUATION ISSUES. Here are the pressing issues and questions that were identified in the first survey. We reviewed each of the 53 responses we received and then combined and edited similar responses. 1. Please read the pressing issues/questions listed below. After you have read all of them, go back and put an X in the far right column next to the two evaluation issues that you believe will be most important for NCR SARE over the next five years. Notice that there are some places to add issues if you think of some that were missed in the first round. Feel free to write in additions if you believe something important is missing 2. Write the identifier letters of the two pressing issues/questions that you X'd, one in each box below. Then describe any potential problems you anticipate NCR SARE will face as they attempt to deal with these issues/questions. Identifier Pressing Issues/Questions Select two by placing an x in this column A Who is currently involved in NCR SARE programs and are they the right people? How can current SARE participants be characterized? Does/should SARE focus on small to moderate-size family-operated farms? To what extent is SARE relevant to mainstream agriculture? To what extent is SARE helping to develop pe0ple who can lead change? To what extent is SARE helping to link farmers. researchers and extensionists in meaningful ways? B To what extent does NCR SARE include a diversity of approaches, practices and systems in recognition that there is no single right way to practice sustainable agriculture and to take into account the diversity of agriculture, climate and social structure across the region? C Sustainability has to do with the long term. It may take more than a year or two (the length of NCR SARE projects) to determine project effects on long-term sustainability. 0 Although sustainable agriculture extension may include technology transfer as a component. its overall objective is to empower people to develop and access the information they need to understand and solve their own problems, to consider a larger set of alternatives. Evaluation of such a program cannot be done in terms of actions and outcomes. but must be accomplished in terms of changes in understandings and abilities. Only the people SARE programs are designed to serve can tell us how well the programs are working. To what extent have farmers begun to routinely question the rationale behind consultant recommendations? Are farmers starting to ask. “How can I get long-term control of this situation?" Are they developing farm plans that address their quality of life questions? Are they asking for more than one opinion when seeking advice on an emerging production question? E To what extent does NCR SARE address a series of whole systems questions. including basic and applied research and education in the context of the system in which it will be used? To what extent does each project include (with varying emphasis) components of inquiry and evaluation of all three dimensions of sustainability — economic, environmental and social — and include both social and natural sciences. To what extent does NCR SARE address root causes, not just symptom? F To what extent are NCR SARE projects analyzed for findings and lessons learned? How well is 'data from SARE projects translated into useful information that is specifically targeted for and disseminated to farrhers, lenders, consumers and policy makers? To what extent has NCR SARE developed peer-to-peer structures for information distribution? 156 G To what extent has NCR SARE identified the greatest barriers preventing farmers from adopting sustainable practices? The most critical components that cause farmers to change to more sustainable practices? H To what extent has NCR SARE really made agriculture more sustainable — farmers more profitable, the environment cleaner, quality of life/rural community viability improved? What are SARE's specific, measurable impacts in these areas? 1 What is considered to be a sustainable agriculture practice? To what extent are appropriate organic and conventional methods incorporated into sustainable systems? 1 Challenges exist in measuring sustainability. To what extent does NCR SARE measure and evaluate incremental steps that show progress in the direction of greater sustainability? How much can we attribute measurable/measured changes to SARE? K NCR SARE projects and information must be practical. To what extent are NCR SARE projects designed to work on real farms, taking account of economic. time, scale issues? How quickly does NCR SARE move research to practical application on farms? To what extent do projects operate at reasonable and competitive levels of productivity, profitability and net income? L To what extent are NCR SARE projects grounded in research? M To what extent do NCR SARE projects contribute to a new agriculture and a new way of producing, consuming and being versus making a growth-oriented industry more socially and environmentally acceptable? How DEEP is the sustainable agriculture research agenda? N To what extent should NCR SARE assure that an outside evaluator visit or otherwise review each funded project? 0 P Q R 2. Write the identifier letters of the two pressing issues/questions that you X'd, one in each box below. Than describe what potential problems you anticipate NCR SARE will face as they attempt to deal with these issues/questions. 157 Identifier Potential Problems in Dealing with this Pressing Issue/Question Identifier Potential Problems in Dealing with this Pressing Issue/Question II. EVALUATION CRITERIA. Here is a list of evaluation criteria from the first round and from a literature review. Please review the list and rate each evaluation criterion for both its desirability and its feasibility, using the following scales: Highly Neither desirable desirable Desirable nor undesirable Undesirable 5 4 3 2 Definitely Feasible May or may not Probably feasible be feasible unfeasible Highly undesirable 1 Definitely unfeasible (More detailed descriptions of each term are provided on the enclosed sheet.) Please rate each item by circling the appropriate numbers for both desirability and feasibility. If there are any criteria you feel you cannot rate, please circle NJ for No Judgment. Notice that there are some places at the end to add criteria if you think of some that are missing. Criteria How desirable How feasible is this criterion? is this criterion? A ExtenttowhichchangesmotivatedbySAREprojectshavemadefanns. Hum 54321») communities more sustainable (economic, environmental, social dimensions) 3 Extenttowhichpeooleselectalternativesusingcriteriathataccountfor 54321"J 54321NJ economics, environmental impact and social consequences (all three) C Long-tennsustainabilityofpractice/program;extenttowhichitaddresses 54321") 54321NJ future impacts of today’s actions 0 Extentoffannermovementinasustainabledirection 54321NJ 54321NJ E Extenttowhichprojectcontributestoordeh’actsfromANYofsustainable 54321NJ 54321NJ agriwlture’s dimensions F' ExtenttowhicheadiSAREdecisionmovesinthedirectionofanideal(or s 4 3 21m 5 4 3 21": at least more sustainable) agricultural system 158 Criteria How desirable How feasible isthis oitenon? isthrs cntenon? 5 Potential to contribute to new thinking; attitude changes 5 4 3 2 1 NJ 5 4 3 2 1 M H Collectpeople'sstories 54321~3 54321NJ I Extent to which SARE has built capacity of all types of researchers and 5 4 3 2 1 NJ 5 4 3 2 1 NJ educators - 1 Overall changes in the lives of rural people rather than specific outcomes 5 4 3 2 1 NJ 5 4 3 2 1 NJ associated with individual projects or programs K Supportoflocal families as a percent of gross income 5 4 3 2 1 NJ 5 4 3 2 1 NJ l Well-being of farm families 5 4 3 2 1 NJ 5 4 3 2 1 M Changes in existing profile of farmers, number of entries into farming, 5 4 3 2 1 NJ 5 4 3 2 1 number of farmers who would recommend farming as a career for their children N Changes in rural communities; is there an increased sense of belonging or 5 4 3 2 1 NJ 5 4 3 2 1 NJ interdependence 0 Self-efficacy among program participants 5 4 3 2 1 5 4 3 2 1 P Extent to which local assets are retumedto public use, increasing those 5 4 3 2 1 5 4 3 2 1 spaces. energies. values that build a sense of the commons Q Economic feasibility 5 4 3 2 1 5 4 3 2 1 R Profit or loss from farming; profit or loss changes from SARE project practices 5 4 3 2 1 5 4 3 2 l and technologies 5 Extent to which SARE technologies improve resource productivity 5 4 3 2 1 5 4 3 2 1 T Achievement of reasonable financial standard of living for farm families; 5 4 3 2 1 NJ 5 4 3 2 1 percentage of household income generated by sustainable agriculture U Dollars from farm Operations that stay in community 5 4 3 2 1 NJ 5 4 3 2 1 NJ V Slowing or reversal oflossoffamily-operated farms 5 4 3 2 1 NJ 5 4 3 2 1 NJ W Shareoffooddollarretainedbyfanner 54321NJ 54321111 X Energyandmachinerycostsaspercentageofgrossincome 5432“” 54321") Y Policy changes that support more sustainable agriculture 5 4 3 2 1 NJ 5 4 3 2 1 NJ 2 Potential environmental benefits (including economic benefitsof 5 4 3 2 1 N) 5 4 3 2 1 NJ environmental improvements) M Physicalchangesinrurallandscapes,inthefamisandsurroundingsof 5432”” 5432“” people_whohave been involved in SARE; aretheymorediverse?isthereless bare ground? more wildlife habitat? Etc. as 5°“unme 5432111) 54321NJ CC Nutrient levels. balance, availability, management 5 4 3 2 1 NJ 5 4 3 2 1 NJ Pestmanagementstrategies 54321NJ 54321ru EE Biodiversityenhancement 5432110 54321NJ 159 Criteria How desirable is this criterion2 How feasible Is this criterion? rr Feed production and use balance 5 4 3 2 1 NJ 5 4 3 2 1 NJ 65 Agronomic soundness s 4 3 21 NJ 5 4 3 2 1 NJ H" Safety for people and environment 5 4 3 2 1 NJ 5 4 3 2 1 NJ 11 Number of people personally/directly involved in SARE projects; number 5 4 3 2 1 NJ 5 4 3 2 1 NJ of farmers who get SARE grants; percentage of applicants 13 Number of people who receive information/results from SARE projects; 5 4 3 2 1 NJ 5 4 3 2 1 NJ level of farmer interest (number of inquiries); number of others reached through SARE programs; extent to which SARE-developed information is disseminated and diffused; number of producers who know of SARE and how they perceive it '04 Number of farmers who adopt /successfully implement sustainable 5 4 3 2 1 NJ 5 4 3 2 1 NJ practices; number of people using a particular management approach Ll Acreage under a desired management practice 5 4 3 2 1 s 4 3 2 1 NM Rate of adoption 5 4 3 2 1 s 4 3 2 1 "N Extent to which SARE projects build farmer support networks and farmer- 5 4 3 2 1 NJ 5 4 3 2 l non-farmer collaborative relationships; equitable involvement of farmer groups and non-profits; farmers as partners with meaningful involvement 00 Extent to which SARE projects develop long-lasting multi-organizational 5 4 3 2 1 NJ 5 4 3 2 1 NJ partnerships; meaningful involvement of diverse audiences PP Extent to which stakeholders are represented and actively involved in 5 4 3 2 1 NJ 5 4 3 2 1 NJ project; inclusion of stakeholders; extent to which stakeholders perceive SARE meets their needs QQ Extent of farmer (or other SARE participant) perceptions and understanding 5 4 3 2 1 NJ 5 4 3 2 1 NJ of how projects have affected them RR Maintain diversity (of language, culture, bio-region) and increase the 5 4 3 2 1 NJ 5 4 3 2 1 NJ diversity of solutions rather than standardize them to fit bureaucratic process; build local, rural culture and economy 55 Extent to which project portfolio is balanced— reflects region and its 5 4 3 2 1 NJ 5 4 3 2 1 NJ stakeholders Ti Balancebetweenreachingnewaudiencesandsupportingpeople 54321") 54321") already involved in SARE W Extent to which sustainable agriculture principals have penetrated 5 4 3 2 1 NJ 5 4 3 2 1 NJ mainstream agriculture and those who rely on farming as a major source of their family income; extent to whidi SARE projects are used as models for others and are replicable; potential for wide applicability W AmountofmatchingfundsgeneratedforSAREprojects 54321 54321NJ WW Whoisgettingandusinginformation 54321 54321NJ XX Extent to which SARE projects provide information and resources to people 5 4 3 2 1 NJ 5 4 3 2 1 NJ who would otherwise not have access to hem; usefulness to limited resource farmers, ranchers, communities 160 Criteria How desirable How feasible lS this criterion? lS this criterion2 W Extent to which a systems approach is used to develop problem solutions; 5 4 3 2 1 NJ 5 4 3 2 1 NJ problem versus symptom approach 22 Look at SARE holistically as a systematic attempt to empower people to 5 4 3 2 NJ 5 4 3 2 NJ develop sustainable agriculture systems AM Equity in distributing money 5 4 3 2 NJ 5 4 3 2 NJ 333 Integrity of technical review 5 4 3 2 NJ 5 4 3 2 NJ CCC Extentofimpacton policy makers 5 4 3 2 ' NJ 5 4 3 2 NJ 000 Educational methods, practices, learning that most effectively impact rate 5 4 3 2 NJ 5 4 3 2 NJ of practice adoption EEE Education/experience qualifications of SARE project personnel 5 4 3 2 NJ 5 4 3 2 NJ FFF Extent to which projects accomplish goals 5 4 3 2 NJ 5 4 3 2 NJ 665 User feedback and adaption 5 4 3 2 N] s 4 3 2 NJ HHH Appmpfiatem of projects for SARE 5 4 3 '2 NJ 5 4 3 2 NJ 111 Percentage of SARE funds spent on administration 5 4 3 2 N3 5 4 3 2 NJ 333 Cost effectiveness of each project 5 4 3 2 NJ 5 4 3 2 NJ KKK Extent to which SARE projects advance excellence; quality of science; 5 4 3 2 NJ 5 4 3 2 "3 verification in 5 4 3 2 NJ 5 4 3 2 NJ MMM s 4 3 2 NJ 5 4 3 2 NJ NNN s 4 3 2 NJ 5 4 3 2 NJ 000 5 4 3 2 NJ 5 4 3 2 NJ PF? 5 4 3 2 NJ 5 4 3 2 NJ QQQ 5 4 3 2 NJ 5 4 3 2 1 NJ RR 5 4 3 2 NJ 5 4 3 2 1 NJ Please mail your response in the envelope provided or fax it to 517-3534981. We need to receive your response by 10/22/99 in order to include it in developing Survey 3. If you have questions or concerns, you may contact Susan Smalley at 517 432 0049 or smalley@malemsuedu. 161 APPENDIX E ROUND TWO RESPONSES 162 NCR SARE EVALUATION ISSUES - SURVEY 2 RESULTS POTENTIAL PROBLEMS IN DEALING WITH PRESSING ISSUES C:\MyFiles\disscrtation.wpd A. (7 respondents) Who is currently involved in NCR SARE programs and are they the right people? How can current SARE participants be characterized? Does/should SARE focus on small to moderate-size family—operated farms? To what extent is SARE relevant to mainstream agriculture? To what extent is SARE helping to develop people who can lead change? To what extent is SARE helping to link farmers, researchers and extensionists in meaningul ways? How does one decide who the “right people” are? I think SARE should be focusing on the needs of family farms, but not by some litmus test of size or structure. Rather SARE should focus on the possibilities for real systemic change, and this will mostly benefit those whose systems are more complex and diversified. Even component studies can be useful if they take this approach. IA01 This appears to be several different questions — audience, size and leadership development — may be too many issues to deal with. IA02 We need broad evaluation go_t just by the insiders. To what extent have SARE developments been mainstreamed versus addressed to a few? What are some general or overarching principles/discovery from SARE activities? INO7 Mainline farming operations will adopt sustainable approaches if they are efficient, effective and profitable as well as socially acceptable and environmentally sound. 1 don’t see NCR SARE having mainline ag impacts with present players. M107 People involved —- committed to research in ways and methods to bring change to agriculture. Willing and able to think outside normal thought patterns. SD07 Item selected 2 times with no comments written. INO3, OHOl B. (6 respondents) To what extent does NC R SARE include a diversity of approaches, practices and systems in recognition that there is no single right way to practice sustainable agriculture and to take into account the diversity of agriculture, climate and social structure across the region? How diverse is “diverse”? A reasonable diversity in the NCR SARE portfolio does 39’; necessarily mean that evegy project should have a different approach. 163 Some evaluators may (knowingly or unknowingly) favor their “pet approach” in the ranking of projects, and thus contribute to ggt doing work in other areas (thereby lowering diversity of approach). IN 11 It appears that the ag community is being fragmented into organic/small-mid farms and large family farms. There is a view that large (LOGO-2,000 acre) family farms can’t be sustainable. Not everyone, nor is it practical to have everyone in small farms. Not enough niches. M102 We must recognize that there are MANY more than one way to achieve sustainability. This demands that MANY alternative methods MUST BE SHARED. F armers’ most efficient method of learning is from other farmers. M112 Some have a rather narrow view of sustainability in agriculture and dismiss anything “conventional.” A broad definition risks alienating some strong SARE supporters. This issue relates to E, whole systems questions, in the sense that sustainability cannot be judged on the basis of a single practice, but a specific practice must be judged on how it fits a system. There are many options that can be pieced together in different ways and sustainability is determined by the functioning of the whole system. OHO9 The biggest problem or challenge in dealing with this issue is a change in mind set by “organic” promoters that this is the 911): type of production that is sustainable. Currently that mind set limits the scope of what is sustainable and drives people away from the concept of sustainability. SD08 Item selected once with no comments written. W112 C. (8 respondents) Sustainability has to do with the long term. It may take more than a year or two (the length of NCR SARE projects) to determine project effects on long-term sustainability. Encourage by funding requirements long term projects. IAO9 Fund some long-term projects. M113 Evaluating sustainable agricultural practices certainly requires looking at their impacts over more than two years. However long term research ties up funds and limits the number and diversity of funded projects. We also want innovative research, which means funding new ideas that may not work, so these projects should be short duration. OHO9 164 Funding over the long-term . . . , but also the difficulty in keeping a multi- disciplinary team together long enough to measure the various facets of sustainable ag — particularly when faced with hostile peers. SD01 I see no “problem” in defining in broad terms the end goal of “sustainability.” Will this action/practice/ method lead to the goal of sustainability, defined . . . . SD06 This issue will require funding some projects beyond two years and making a commitment to projects financially for a longer term. This will require another change in mind set that funding more projects is better instead of longer-term projects that may have more impact region-wide. Consider combining this with J -- defining long term sustainability and measuring/evaluating the steps. SD08 Item selected 2 times with no comments written. IL12, ND D. (7 respondents) Although sustainable agriculture extension may include technology transfer as a component, its overall objective is to empower people to develop and access the information they need to understand and solve their own problems, to consider a larger set of alternatives. Evaluation of such a program cannot be done in terms of actions and outcomes, but must be accomplished in terms of changes in understandings and abilities. Only the people SARE programs are designed to serve can tell us how well the programs are working. T 0 what extent have farmers begun to routinely question the rationale behind consultant recommendations? Are farmers starting to ask, “How can I get long-term control of this situation? " Are they developing farm plans that address their quality of life questions? Are they asking for more than one opinion when seeking advice on an emerging production question? Difficult to measure. Doesn’t fit into two-year funding cycles. Requires new ways of doing evaluation. Likely won’t appeal to government funders; may threaten them. Appears as “soft science.” KSOl Getting people to believe more sustainable practices are the right way to go needs to be followed with programs to empower the front line people — the farmers — with enough knowledge, confidence and incentive to make the necessary changes. Getting from the academic research and demonstration agenda to routine adoption by commercial farmers will be a major problem. M107 Changes in understanding cannot be measured easily or with complete accuracy. Evaluation time line will be long. The vast majority of farmers/people are not at this level. Constituency is very small. MNIO How can we overcome a conventional evaluation process mind set and reluctance 165 to change convention to address real issues? How can we move away from technology transfer. M007 This will take external evaluators — not self-evaluation — to accomplish. This will also take face-to-face interviewing to get good data. It should include #A in terms cultural and economic diversity. NEll We need to move away from “technology transfer” to “facilitating learning.” For many “university types” this will be difficult. WI 07 Item selected but no comments written. 1N03 E. (12 respondents) To what extent does NCR SARE address a series of whole systems questions, including basic and applied research and education in the context of the system in which it will be used? To what extent does each project include (with varying emphasis) components of inquiry and evaluation of all three dimensions of sustainability — economic, environmental and social — and include both social and natural sciences. To what extent does NCR SARE address root causes, not just symptoms? How to educate people on a holistic approach. IAO9 Small farm systems do not receive the same attention throughout agriculture as do large operations. Many economic incentives aren’t available to small farmers. Maybe commodity prices should be based on resource use efficiency? Should farmers and merchants who help maintain the rural community and resist urban pressures and special interests be rewarded? SARE needs to more efficiently educate the public on the importance of social and environmental sustainability of agriculture and the desirability of maintaining our rural communities. INO9 Difficult to measure. Doesn’t fit into two-year funding cycles. Requires new ways of doing evaluation. There is never a “neat” or “complete” answer or end point; doesn’t fit well with media or people who want dramatic numbers. Gets into value systems which can polarize people rendering a lack of unity about desired outcomes. KSOl This is very critical if there is to be “systems change” within the food and fiber production systems in the US. However, this needs to be addressed and measured looking at multiple contributing factors within an evaluation process. It will take evaluators from multiple disciplines to measure this. M110 NC SARE seems phobic about funding or supporting any basic research. Methods are poorly understood, resulting in an almost paranoid reaction to any type of systems research that even suggests using basic research approaches. 166 MNOS Answering whole systems questions will probably take large interdisciplinary teams that need lots of dollars and several years to do their work. Also, people might need help in figuring out how to do this kind of research. SARE might need to change size and time lines for some grants and offer help in developing proposals. MN 06 Research tends to be reductionist, but even specific projects need to have a whole farm, family, community perspective. MN19 Conventional approach to technology development and transfer and reluctance to confront the challenge of dealing with “systems” or “wholes”. M007 Majority of people in research and education continue to focus on single or simple components and take a short-term perspective. NEll Difficulty in quantifying environmental and social impacts in the one or two year life of a grant. Symptoms much easier to identify than root causes. NE05 We have skeletal protocols for looking at whole farms. This will entail ??? and economic investigation, which takes special skills. Long-term efforts (#C) are important. NEll ' Need for team approach. Much of this work is conducted in land grant institutions, most of which do not adequately reward or even recognize the importance of teamwork. We are still promoted and tenured primarily on the basis of individual accomplishments. SD01 F. (13 respondents) To what extent are NCR SARE projects analyzed for findings and lessons learned? How well is data fiom SARE projects translated into useful information that is specifically targeted for and disseminated to farmers, lenders, consumers and policy makers? To what extent has NCR SARE developed peer-to-peer structures for information distribution? Greater emphasis on dissemination could mean PR competes with research in every proposal. SARE should create structures and opportunities outside of project funding to foster exchange and dissemination. Maybe regional workshops, a funded speakers’ bureau for recent projects, or distance learning products. IA01 It may be difficult to quantify at this point if there hasn’t been an attempt to track this information or if efforts haven’t been consistent or set up well. IL13 167 The final report will need to be modified to specifically address the question of useful information. NCR SARE should hire an R&E grant manager to work directly with grant recipients to develop a comprehensive portfolio of SARE information; expand annual report to include this information. Staff is already over-extended; make links with Professional Development Program to help disseminate information. IL14 I’ve seen some of the projects and data, and question the validity of the conclusions. M102 Getting information providers to accurately summarize how the information was disseminated. OHO6 Information is not bundled in mainstream ag ext but “labeled” as sustainable and often given less attention. ND03 ‘ Problems faced will include tracking down products resulting from SARE grants. Is it a SARE product if it was developed years later? NEO4 Obtaining qualified and unbiased analysis of grants and dissemination of information. NE05 We need an information distribution system in conjunction with Extension. I believe the magazine style information such as “Furrow” (John Deere) is the most effective. We need repeated mailings of printed materials to farmers throughout the year. NE10 Producer grant results are not well-disseminated to educators, other farmers, etc. W107 Item selected 3 times, with no comments written. IL12, ND, OHIO G. (5 respondents) To what extent has NCR SA RE identified the greatest barriers preventing farmers from adopting sustainable practices? The most critical components that cause farmers to change to more sustainable practices? The greatest barriers may not be able to be addressed through educational programs as we know them. It may be necessary to redirect much of SARE’s resources to policy change, for example, and this may cause problems! It may make more sense to put SARE’s resources on changing the paradigm so the demand for the practices increases across rural and urban groups. IAOS This is an excellent question for the overall program, but I suspect that many 168 individual projects never thought in quite these terms in the design phase. They’ve identified some area or practices where they think they can improve sustainability, or the thing that limits sustainability, but an overall evaluation of such may be missing, i.e. individual projects and reports may not help very much in this issue. IN 11 The traditional farming patterns and the power of the seed and chemical industries, which includes the support of major farm organizations. K808 This issue has to do with diagnosing the problem and it’s hard to show action results when you put emphasis on doing the right thing instead of doing things right. M119 Getting producers (conventional) to consider new options, think out of the box, versus cookbook farming practices handed down by corporate agriculture. ND03 H. (7 respondents) To what extent has NCR SARE really made agriculture more sustainable — farmers more profitable, the environment cleaner, quality of life/rural community viability improved? What are SARE 's specific, measurable impacts in these areas? Who will do the measuring? What will be measured? IA02 The problem is what are the specific, measurable impacts —— this Delphi process will help us answer this question (and others). Cost (human and financial) of integrating measure into research, education/demonstration projects. IL14 With the trend toward mega conglomerates in agriculture, agricultural producers are losing more control. Large units do not contribute to improved quality of life for most rural residents. It seems SARE could have a greater impact in this area. INO9 I would assume this will be very important for the long term support of the SARE program. Identifying measurable impacts in all three areas will be a challenge but very important to do. M110 First we need to figure out what to measure and how to measure it. Then we need to figure out SARE’s contribution relative to that of other organizations. MN 06 Getting hard data on changes made, practices adopted, etc. OHO6 Qualig of life is equated with social/ethical issues; to an extent so is community. There is little room for discussion of these issues in a growth economy. 169 Sustainable agriculture deals in people and how well they can provide for themselves and add to the well-being of society. Current agriculture fits into the extractive economy and values only profit, so really you are comparing apples to oranges. W114 I. (1 respondent) What is considered to be a sustainable agriculture practice? T 0 what extent are appropriate organic and conventional methods incorporated into sustainable systems? Must recognize that if organic is defined and conventional is defined, that a sustainable system could include both. M112 J. ( 2 respondents) Challenges exist in measuring sustainability. To what extent does NCR SARE measure and evaluate incremental steps that show progress in the direction of greater sustainability? How much can we attribute measurable/measured changes to SARE? I have no idea how this can be measured, but it seems like if we’re going to show the program’s effectiveness and/or figure out ways to improve the program, we need this information. 1L13 No problem once sustainability is defined in broad terms. SARE’s two year “projects” can only be incremental steps and should not be difficult to evaluate — whether empowering people, are holistic/whole system, innovative or Consider combining this with C -- defining long term sustainability and measuring/evaluating the steps. SD06 K. (5 respondents) NCR SARE projects and information must be practical. To what extent are NCR SA RE projects designed to work on real farms, taking account of economic, time, scale issues? How quickly does NCR SARE move research to practical application on farms? To what extent do projects operate at reasonable and competitive levels of productivity, profitability and net income? How do we take the project results and get them to be practiced on real farms? ND02 The common criticism of organic/sustainable farmers is that the cannot “feed the world” farming as they do. High tech ag is the only answer. SARE projects must be viewed as fitting into conventional ag and not just providing food for the elite few that can afford low yield, high cost “clean” food. SARE must fund projects that are practical, profitable and productive and can be adopted to larger scale use. 170 W114 Item selected 3 times, with no comments written. NEO7, OHOl, OHIO L. (4 respondents) T 0 what extent are NCR SARE projects grounded in research? A science-based peer evaluation is needed for the program. There will be a diversity of opinion on what constitutes “credible research.” INO7 NC SARE seems phobic about funding or supporting any basic research. Methods are poorly understood, resulting in an almost paranoid reaction to any type of systems research that even suggests using basic research approaches. If we are to move sustainable agriculture research forward, the $100,000 line in the sand for two-year projects must be rethought. MNOS Item selected 1 time, with no comments written. NE07 Not selected as a top priority, but this comment written: I usually find this a loaded question. In my experience, individuals use this to quickly dismiss the different —— that which hasn’t been researched by replicated, reductionist methods by a land grant university. This immediately shrinks options into the small world of what fits the prevalent paradigm and which is acceptable to land grant funders. KSOl SARE (especially TC) ought to place more stress on increasing the quality of research it funds. SARE projects are typically assessed more so in terms of criteria other than quality of the research. There is a great deal of resistance on TC’s to fundamental research. SARE research needs to stress the development of new systems that will be useful one or more decades hence. W112. M. (9 respondents) To what extent do NC R SARE projects contribute to a new agriculture and a new way of producing, consuming and being versus making a growth- oriented industry more socially and environmentally acceptable? How DEEP is the sustainable agriculture research agenda? It seems that many of SARE’s programs deal with changes for efficiency of producers or changes in substituting one management practice for another. Redesign of the ag system may be needed and this type of work is hard to evaluate and high risk. SARE’s need for quick impact and accountability may stymie much redesign work. IAOS We need to highlight more on the economics of the situation. To make 171 sustainability a major industry in itself, it needs to be more profitable for the producers. K808 Allow for some grants to be outside the box. M113 Deciding with any accuracy which problems/studies lead into a deeper understanding. Who are the SARE employees capable of/daring to make these choices? MN 10 SARE must be a leader in the sustainable movement. Are they ready to be one? ND02 We need DEVELOPMENT (qualitative) without GROWTH (quantitative) and especially need to conserve, recycle, reduce throughput of materials and stuff. NEll Problems faced will include finding evaluation criteria to measure. This is a broad question and may be difficult to focus. NEO4. What is it that causes people to shift from linear thinking patterns to a more holistic approach? NE10 It is difficult to design a new way or approach to national practices or to change the overall way people do things. But on the same token, SARE was developed because of the farm crisis of 1982-85 and we are once again in a farm crisis of 1999. Without huge government intervention, thousands more farmers will fail in the years 2000 and 2001. Cycles of agriculture are related to cycles of nature. SD07 N. (0 responses) To what extent should NCR SARE assure that an outside evaluator visit or otherwise review each funded project? 0. (Written in) We how have 11 years of research under the funding of SARE. We need someone or group to summarize this wealth of information and condense or act on the outcome. SD07 172 APPENDIX F ROUND THREE SURVEY 173 .2829? 03:9. 808 a 8 mEoza 9.355 Soc: Eek c2... 9 0302* 333 :53 Sons 2:: 305. 03 etc? 57.32 :02: 35b... .38 3:33:38“ 23 83:: x23 8.. woo: Pmmma tongs... 35 £8.83 San—«>0 Ea omflaooco 2 :35? as: «smegma 533.2 3536th 33583.0 a: & RMMQ so: «~393qu Azukcoszoééu hen A2892. 988 E335 huknutoéeaew u 935: 32»; M53 3.3 ”553.38 .memuanfik .3 A9: a»: a has 2339??» so: a 2 2:95:09 283i MES”. «02 on 22% .33 o k .voncouxoago €35“ m. tSm .owcozmco a m_ cocancoks ..o cotaEEommB Ea 852w .6 warns“ commie: can ugh—«:6 mEEaBO $28st 53252.3, canoe—L85 82:me mxo E 5388 .23 2383 2 ME 8:3 2:8 cowaEEommB :o $92380 3320 mecca—585 .88: kc 3:337 05 3063 3.353% 3.89: 80.35 3:: 2:97. Bo: 232 who» come—gov 83 z .z .2605 mx6: 2:039..— 3283.— 283350 62.8— «£30...— .fits .6» a... 2:0an 05 ..o .32: me ..o 30.. me o. 98%?— ?E 20> 2:959:— 82: .3 a...“ 0:89.96 :3 gm £02 9?? 3.. 33.83:... .59» 5 8:3 ammo—k .333 Box .8 mac—ma 2.03 262 .Na 28 5 355. E 35202 no» 65 $5an 05 23 33.2 nouns—«>0 05 0.8 3o: .mmbmm— ZOFH J. 32 .2 895262 . on?nfitam._a>?u§:o§3=u_kgo ...—=3— .afik . m akotsm 552:3”.— 2...»: 5:325 33 5.333. o..===u_._u< 03333.5 .833. .9330 5.82 174 000.000 550080.000 ..0 .00» 2.. 2 08. 8...»... 02.02.2200 0.... :0... ...0052803. ..0 .00» 000 0... 08.0. 000... 0. 2.8000 0. ..80...0.0.. 00 000 . .8008. 00.3.0030. 0.08 0>0.. >08 .0... 0.00.3.0 8.0.9.030. 830.. 3.00... 00... 8.8.. 0. 0.00.8.0 0.08 3.00.... .0... 8000.8 3.4% 80.08 0... 00w00..0..0 0...... 0.00.8.0 0000.... ..0 b.0830 000 80800 0... 008.. 000 000.... 0.. 00.. 00.0000. 8.8. w00.. 0.0.3.330 £80.38. 00 0.00.? 80.800 058.080 0. «0.00.0.0... a...“ £028 538. 0.... 03. ..c 000A 0 =0... 0.0... 0.8. .98 .k .800. 3.0. 0... 5...: 00 c. 03 b.0308830. 000000.000 00.0000. 0.00.. w0.0.. 0.00300 8.8 .0... 00.0000. 080.000 ..0 00... .000 0. 00:000.. 0.00080 .008.0 00 0. m0...000. 000.0800: .....000 0.0 000502 ...20000. 0.00.. A00 2.050000 .0 M068... 0.0..... 052.0 08000 52m 02 00.08.8800 .00... 9.8.0.808 ..0 00.32.30 0... 000 3.3083000 0000802300 000 .0800 0.0.3.0050 ..0 0008.008. 0.... 00 0.300 0... 0.00000 0. 00000 mx000 .000 .0» 03 000 30... 020.0 .0.0000 00.0. 00.03 .00..0w..00>0. 0.800000 000 0... ..0 .98.... 20.00 ...3 0...... .0800. 0.0..3 .0 m0...00. .0. 0.0090... 8.0.3.0 ...00 03... 03 00800.00 00.0000 .0000 .080 ..0 0.00. 0 m0.8000. 0.0000 00.8.00 000 00.03 080.000 00.0.. 0.0. 0.00 0...... 000000 .00.. 00... @0000. 0. 8.000 ..008 0.0 080.08% 080.800 00080.0 .003 9.3 0.0000 ..0 0.008 ....3 ..03 .... ..00000 .. 00.00 000 .0 83000 :0.0.0800: .0 ...000: 0 8.80 0. 0.0..... ...0.0000. ..0 00... 0.... 00 o. 30.. ...0 3.3»... 0. 0.0.. 0000 :38 0.000.. 0.00.. 03. 00... 0.08 8.6 0.0..00 ..0 0.0. ....3 008.080... 0.0.....8 80.... 0.080.. 3.05.0000 0.0.....8 3.0.80.8 00.3.8. .. 000 0.00008 0. 0005.0 0. .. 0.080.080... .330: 0.0300 .00.. 00000.00 52a. «02 0000 .0080 3...: ck 00000.80 .9300: 0.6 .8000 $0.. 000.02.. 000 I 3.000 000 3.000.000.320 0.800000 .l b...00=..0.§0\c 30.000000 000... 208 20.005030 0:0 .088... \0 0.5000800 305.080 M009... 0...}. 0030.... 80.3.0 .0000 0000 .0008 3...: 0 k 0.000: 0.. 2...: .. 00...... .... 0.0.00.0 00.8 .3800 0... .... 00.000000 0.6 0000000.. 00.0%? 0.... 0.3.0.. 30.030... 5.0.5.030 0.50.00... 0.0..... \c 00.20.. 0 0800.00 .323 «DZ 800 08.00 .90: ck 008030.... 0... 0800.830 gm 000 30: 08030.... 3.00.00 000.8000 .808— «0.30.... 175 .Qmm 254 :2: anew—Fun 2.: .323 So? Such 3.53 3?: ck fl 92:93::— a:. 2:39.35 why—(n use .36: nEu_ao..n~ —a==o.ck \ 2.2.9.50 .33.: uc‘nnohk\ ...:on 0:22: 2.50 E29? ozassgm a fiasco: 2 35:52.8 :5 “.250: m. scams : «.3259. 933.63.» on: $33585 €232: 352533 28 2:39: 253035» 98 325» ES: ck «ousai 253.32%: 2358.3... 3 a 3:: .33 wt: 5 62:5 .252» n 26: 2:8 m¢525 toga—E 9 235:8 8: 0: 2:5 omhfi ..o::oo 22: mime. 2a 203:9... 35:223.: .2332th 5 822050350 «was. “.338 :5: 2: .23 .505 3:0 mos—S, :5 38:80 3:255 2: BE 2: 23.32:? 2.250 528.... :0 mEon :03 2: 9 :3 28 328805 .8 0:38: :3 >2: :03 26: :5 0.33 E 23: 22.3er 033863 $52.80 532» a E 2: :0 A526 5 gaze—.88 .8 c2332: 8.: Eco: 2:: 2 22; ~38.qu mx533: mar—030. 3223 32a 2:0 5 :83 9.8: a: o. 2.53:8 2:: :5 £62.03 ..o mac—=8 2: E .2822? 8a.. :2: .85.. 3353225 8 waBog :2: 293.96 223 9.0203? 2 0203: 30m ...—233% 5:262: 032.523. “also 2: mm 2.590 .2: 220an 2590 2:8 :0 .0358 a 82:3 2 fl owcozasu 2.0 22.23% m2Z00£:\m0_0_0cta 3200203 :0 _::0:0w .3: EEEwE 2 29.50:: m_ z 030.“ 0 0. :030:::0 333 3800:3508 :00: 35:50.05: 33$ 02...: E038 .023 0 H 30:35 .3 an: :0: 5:00.03 :00:: :00: 03 00.3 :03000 23.... 05 0.3 0200: 0:0 000: Box «.30.: 3902008 5 20:20:283 :50 23000000.: .00030\«:.: 0: M530: $20.: 0.: E0300 :33 0 .N 0.03030 :03 :00 0*: 030K RS008: 0: 30:00: Eva. 0.: 30:8 :04... 0 .N 0.005302? 80003.85 0: 3330» N53 0.: 3038 09:: 0 h muExu\:0.0L0&0.A=Ec\ 00.2-030035 0: :35. :0 300\ .32.»: 3305:0009 0:03:0000003 00 03030.:th 3x5. E00030 :00 30m «.0103 Em: 05 A03 000 :00 0.50033 95.6 «02 2.. 3393.. $300030 0... 03: mm:0:0~_::m.0 Etc 00.32: _0 0.00:3 05 «0:305 :02? £030.35 30.805 :5 :08. 0.: 00 .0300 0.: :5 mEozag mEEfl 3:05:05 ..0 2:05 05 0509.03 03 :00 30: .52: $55 wEo: .8 30.0.5 msfi Em: 05 mfio: :0 2.9.295 S: :0» :23, 3.30.. :0300 305 9 :00; 0.: .0002 £5 E :03: .00> 20; .0: bx: 3:000: ::a 200.35 2523::— 380305 0300 >.:E 25 ::0 6.0588 :0.“ 69.20 >020: 9 80500.0: mx0§ 0.002000 Ragga 0S :0SE0E 52%. «U2 8: E0000 ~33 0h 005030.... 0.: 0500026 ”53% :00 .50: 05030.5 30:03A— 055850 .0003— 9.38.5 177 .8: 23m 892 2 28203 on :8 28 252622 new 032.203 48:86 0.3 35 $8.85 223 use 57% 2.25.. :3. no 23283 on 9 E2: “om can 338.. 80.35 8.8 956 50 Box «...—282.2 32 222 bcsgxogm 5.3232922. 39$ 23.53328 222 $2220.82 6 22228 2.62.8.3 ch 2828 B23 ck “SEQ 20 223.21% 3.283% 2 2333.2 $822 33%. ~52 .322 3.352 38E 2.32.3... .228. 622.: 6.2222823 222392 M233 .3222\ 33 2o «29: 8 22.23.82 38.2.93 MSG ~52 ~22 2282» 2223 as 23.53% 3 2.322 29:22:02.2 Be 82.222 .326 52 .225 be, .2 8288.5 3.05 .8 mac—oo— 5:02:38 2:. .30. on EB 2.: 22: scans—gm $08282 30.923 5.3 8 $30 2838.: on 8:23 mEvcsmuoucz E woman—6 .Bonce EoEEo>om 9 Boga Ho: 29:: 28 22:2: 2 92:28:: :26 ES .3056 2. >22 = .3223 ..coM. on 2 E25“ ‘38 mi... .9258. wcznzzofl o. 3.222. $20252 50¢ 2332 305 9 woo: 2:03 535 .mozmfl .22 3963 2 5:528 0935 2 322038 23 .8 255 3000:. 20.3296 3203:3200 2 220226 9 92w 8:62 2203 $5. #2 v 2.22.2322 22.5229R «2.23228 22 22 3.322 M233... 23.: 222.32 ~22 223 98.2 LQ\M2.&.8 29$ 3w «2.20.2322 2.21.0 5.2222 2.822 2.2222 2222 “.221 2:2\ 92.25232 A2: 2‘ .. «22.2222.» 252» 3.228 2:2.M2S 22% ~ 28 :81: id 2 $2.228» 222.8% 2! «2.202222222282 222222.28 22.33 3222.222 2: 20.2322 A3532 22 22mg Ensuux. 2.262 2qu ~33 ck $2.38.: 2.6 2.229%qu 2: :9: 33 3 =8 23 9:2. 2 222M232 ~22 Eagmem 53. 2282 22. 2.6 $223 222 ..M22222322222 2.2 833% \o 3228 2.2 2223232892 3 2.322 =5 6qu8:2 222 «.2228 \o 2.222 2.2 2222 .3 3228 265922. 2 292%? 22222~2>w “3522222222 22. .833 2 2322.28 8 £5393 23° 2.2.22 9:9. 222 22222222: 9 2222 A25 222228.22 22 2892. 222 2.8282 2 £232 292252» 2 2 23.28.82. 29.98 2.. 22220328 2 8 2.05.222 $822292 whats A92 29:22.8 92529.25» 2322.223“. 5223‘ 178 . .55....an Q .5523... 88...... 28...?!“ .883. :5. coca o. 883 8:62 2.83 5:2... :9. $.20.» $553.. Ea... 322.3... E 3.8.. 22: .o .39. m ”.32 :5... . 22.8 33.3. ..o 38...... 93.8 «88...? ._ .333 o. .528 .EuBoEBEEaE ..o Dzfiafisgu 3.8.23..— .0 misc... Ea... mac—baa... .0 $539.03 3023...: 3.: 5.830.... 30.. E... on: woo. 283.3 85.3 .v .3020 3:38 .o .92.» o_oa30=o._-co.. £30.82... €2.33? we own ... $.82on ...oE....o>om ..o 85:». ... :8 8.5m... m 0.32.8 mm. :3... 3.2323 9.236... ._o>o_ 5.3... .. $5.208... Eco» ._85E..o.._>..o 6.82.38 3:32:23. 9...... 3:23.580 .35.. 03.... 2:... 38.8... 57% 3 3.3.8... 8.5.... .023 2 .53. .< got... .2 .5 Esau. o. 32...: mac» n be... M02 320.. 32.5.. ..o 0.953. .33..» :33 _a=o_wo.\o.3m . 8.82:. 0.2.553... a ... .3526... 38.6 ..o 33.6 d state a... tan-2503.9... 8 .32.. ..o 2.02 30> m¢ 4...; 52m :02 ... 3.3.3 o... 25.9... .8 95:3... 2 ..o :5... =3 .8.» 9?: .85.. b... ... 3...: 33E do. 802 So» too: 95 .3 183 m: E «:85 2: 02.3.: EEE mmH .m 235 E 3.53 a. b__3...._a.m:m 3.5 .5395 a on .2. .23 m2... .mgou “.an o. 386:... «.938»... $395.62.. 33.2.... 9: :8 $8.... 36: «5.33.2.3... .2..me \o .8235» u... .... ....»EEQ 32... BS 32... 3.55.98... 33.9.» fins 93.3»... .33...» £02 8% .528 .33 as 5:33.83» «5.3...qu .... 5.3 nuanuzafiu $3030.... 0... 2.89.93 an :3 >3: 2.339..— 338.5 2.3.8.5 .83..— "£30...— 179 “555%: >526 Econ—5:3“. 88m 3 3.26:8 3:088:32: $36 333 vasohxooaam ”.26. .a:o_mo:o§m . .58388 ammg<.§m B 3.86.: 3 nouns—=88“. 833:5.2» ..8 “2.. ".96. :5“. . 358305.: 355.835 .3 Sauce: £82.89 9:335 8:055 3:08:23... 32:88 .N 9583.3 95.. .8 San mam ”33:3: 25.. ..8 83 338 mm :32 :oButBSm 25... vasomoizee ha “3. ..c .239. ..o 2:308 .> .3358 2:85 Bonanza: omega bcsoo 2 C 23238 9:53 So: mmoScoa mo 2:: :32 :5» 88.35.“ ozafisgm 3 38.28» 2:02: 20:83: .8 “ag.—8.2. “8.5.5 5...... ..o. 2:2. he 33...: 3055.: 2:20.82 .0 EoEo>o£u< .._. 325 :82 o. 80% 8.3.5.3.. we 0:2 ”.26. 5.3.: 38.5 A: 3582 ...—.3. .38 he 9.25 .3 m 0.3.2.8 my: Sea 3:25.. .82 8.. :233 2.23.3 “7.30. :5..— §§ano ... ban ...: 28:80.8 .Ee 89c 95...: .D Avian» .8qu .8 23c 35 co 80.8.5 mm5: .3 232 28> 52m muz E 832:3... on 8 92.3 2.3m «:35 .353: .32 180 5:28.88 35083.32. .8 v8.9. 83%. ..o .395: 2 333%.... ..o .382. ..o 0...... ”.96. bias—Eco .8 533.590 . ..aom 25 2d: 5 @6on o. 8:5,. “.8 ”.96. :5... . A82. :3 «853.388 3.8... v5 .895 3.5.5.235 60.82.80 3.. 2538 35 3.5.3 ag.»: 82:58... 80.3. 0.32. :023 o. 38%. .m .833... .o 3:338 5:2,... 832.80 35 3.2.3.598 ..o .382. .33 .2038 .8 o.8m\b.§EEoU _. 32.323... Suns—=2: 5.3 E0652. 3 808...... uncoaéo: can 332» .565... ..o EoEo>.o>:. 033.33 "2.330329. 03.83230 365.39.085 .33 5.352. tong... 888.. 2.3 38.8.... mdo .25. 35° 2 $2.52: 5.5.538 msfié u.na:.8n=m 23 35925.. md... 208 an... oh .mm m¢o «02 $88 3025.. .8 295... >255 H_o>o_ commflxo§m _. $33.3... 3.3 3.. 3.8.8 638.32 Ba v5 .55.. 3.. 232: 8 won: on $8.85 52m 523 o. 335 6885 his.“ :2: .«o 858 8.92: a ma weir—8 so 32 0:3 035 as 9.3.3:». Sacha-:2: 90.2.2.2. 9:2. 8.3053 88:85.“ £558?” 523 2 2.8.6 .3: 558.85 52m no»... 26: >05 6: 3 552.3 x8 Ba 33» m bo>o “52 3.28 flue—5.. go 0383 his... :32 .a:o_wo._o>c_ £23.05 Sosa mcocmusa 5:3 93.85 vocfic =a .23 on: 3.. >023 32283: 80.83 53$ ..25 3.300 . 800: :2: 38:. $36 0285 cow—0:83.... £023 o. .528 ”flow—onoxgm ..o 53305 38.85 E “8235 23:: use vow—8852 93 £022.33» 523 9 E25 .nE 5.58.3 at: 233.5322: 8 32. .8 :92 23> 52w mUZ ... 298932.. on 2 92.3 2.3m ...—8:0 «39:89 .32 182 .auo.=m8.o=mE©mu:mEm 8 025.3?“ _ m 8 >335 :85 33:8 .3388 8 28:33 26.. so» b .9302 95 :8: 2: 55.? 8:38.. .5?» ufizouog 8.33.39. 2:95 «3 .53-».3-» _ m 2 2 5c 5 323:. one—2:5 05 E 3:032 Box :2: 335 .30; .50» .5. so» x55. Samoa $323 mo. as? 388 83.23 23 87. £3 mdo_ .mcofifiESm ._ :3» .3383: $3qu 523 2 Eouxm at 8: 903 md».— _3__:.3:o has... .mmm no; no 232 9:83:05 .fi mains“ 38:2. .5333 ....oq 5.355 cocoa—om .3qu 8:55 .2333 $0.85 52m €35 . 52m .8 38.35 Ho 32.82.32.3< .13: 3:35.535 8:85 Scan x8 .23 who» n bo>o M02 328 who—Ea go 295.». avian u_o>o_ 3:29.536 ._ “cats 23 3053.. ..o “35:: 28833 ”.26. 80.8;— ._ 5853 E25358 8329.3 a min: oaooq go .8352. ”82895 0.3533... 525.95 b.3um8o=m\ Eoc- 23 Eng—...: .3 53:52 00. ao...8_.8 n3. tam-250922.. 8 32. no :32 23> 5.3:: a... 0.5.2: on 8 99:: 09:5 £25 258.... ... £58: .32 183 APPENDIX G ROUND THREE RESULTS 184 NCR SARE EVALUATION ISSUES - SURVEY 3 RESULTS - INITIAL CODING HOW CAN SARE OVERCOME THE PROBLEMS? C :\MyFiles\disscrtation.wpd USEFUL INFORMATION, N=26 CLARIF Y PROJECT EXPECTATIONS (l3) "‘ PR should be part of every proposal. Perhaps withhold some of grant until information dissemination is assured? (IN09) * Must have follow up or why have the project? What happened? What went right or wrong? Have the Pl write follow-up. (KS 10) "' It might be useful for the SARE researchers themselves to engage in information dissemination. (M105) * Provide investigators/grantees some small amounts of S to accomplish this (project products, dissemination, useful information). If you receive a grant, you get an additional $5,000 to promote the project. (M113) * Need clearer outcome of projects. (M119) "‘ Make sure that we ask report writers to tell us how they expect to use information after grant ends. Improve integration between Chapter 1 and Chapter 3 programs. Ask PDP proposals to use Chapter 1 work. (MNO6) * Ask for written reporting. (N51 1) * Researchers can be encouraged to use PDP in their outreach. (OHO9) * Outreach should have been one of the requirements for funding and should be part of the final report. (SD01) * Make sure final reports are completed. (W114) * Maybe we should provide a reporting form that is really easy to fill out for events. Also, a format for reporting to SARE would help — we could have a special section for publications. (MN05) * Current and future grant reports should ask more targeted questions about information products. (N E04) "' Each proposal should contain an explicit, feasible plan for dissemination of results. The RF P might address this as follows “Describe the deliverables of this project, in what form they will be delivered, and to whom they will be delivered. (IL08) TARGETED FUNDING (9) "‘ Perhaps divert funds to hire someone to analyze and summarize pertinent material. (IN09) * Every proposal/funded project could have monies built into it to a) allow the researcher to attend a conference where SARE projects will be profiled; and b) participate in a conference — perhaps conduct a workshop — in which SARE projects are profiled. This would further the networking and the educational process for everyone. (M105) * Provide investigators/grantees some small amounts of $ to accomplish this (project products, dissemination, useful information). If you receive a grant, you get an additional $5,000 to promote the project. (M113) * Maybe you could fund a project or two whose purpose is to summarize the summaries. (SD01) * We need a summarization of similar research data and then an article of that data and send it to mainstream press reporters. (SD07) " I think funds have to be made available to learn what products have been developed and to get them out. (W106) * Careful monitoring or an outside group could help monitor the outcomes of SARE funded projects.(WlO7) "' Issue specific calls for proposals to analyze and disseminate existing information. (MNO6) * SARE could hire a consultant to work in concert with the communications specialist to review results and information products from all grants previously funded. (N E04) EXTENSION/LAND GRANTS (7) "‘ Radio spots and press releases featuring SARE projects will be used by Extension as well as lots of small-town media. (IA01) "' Involve Extension people in projects from the start as collaborators. (NEl l) "' Continue mailings to all Extension offices in region like the excellent program note series which has been coming out. (OHO6) * Make sure extension offices have SARE web site addresses, Could distribute publications to all offices — $!?! (W106) * SARE can’t take it upon itself to create a whole new Extension service. Just do good research, make every effort to get it to users, and work to build a strong constituency for sustainable agriculture within extension. (W112) "' Try and involve Extension agents in SARE conferences; send newsletters to all agents. (W114) * Partner with Land Grant Universities - grad students have to review the literature for their research projects. University communications departments can lead in writing/telling about projects from their schools. May need feeds from SARE staff to initiate dissemination. Use interns, volunteers, retirees to do PR/media work. Web- based story/activity archives - use students to build the site for experience. (N E1 1) 185 FOLLOW UP (6) "' Ask for project follow-up or results on I, 3 and 5 year times. Must have follow up or why have the project? What happened? What went right or wrong? (KSIO) " It might be useful to do a 5- year follow-up on selected projects to see what else ‘new’ has happened. (M105) "‘ Make sure that we ask report writers to tell us how they expect to use information afier grant ends. Then, periodically survey past grant recipients to see if any new products have been developed. "' Periodically survey past grant recipients to see if any new products have been developed.(MN06) "‘ Phone interviews of grant holders 12 months after end of contract might get more results.(W106) " Keep all grant recipients involved in SARE and do follow-ups to grants, especially producer grants, out several years to ask if practices are still being used. (WIl4) " The SARE office should perform follow-up mailing to MI’s two and four years after project completion to ask for additional products. (N E04) MEDIA (5) "‘ Produce radio spots and press releases for local media featuring SARE projects. (IAOI) "' More use of popular press —— papers and magazines. (M119) "' Team up with the popular farm press at every opportunity. Farmers read ..., and the farm press is looking for stories of hope.(NE10) "' University communications departments can lead in writing/telling about projects from their schools. May need feeds from SARE staff to initiate dissemination. Use interns, volunteers, retirees to do PR/media work. Web- based story/activity archives - use students to build the site for experience. (NEI 1) "' We need a summarization of similar research date and then an article of that data and send it to mainstream press reporters. I think they would be interested and begin to follow up on it. (SD07) NETWORKING (3) “ Every proposal/funded project could have monies built into it to a) allow the researcher to attend a conference where SARE projects will be profiled; and b) participate in a conference — perhaps conduct a workshop — in which SARE projects are profiled. This would further the networking and the educational process for everyone. (M105) "‘ PDP’s in each state and researchers must become more aware of each other and communicate. Regional PDP coordinator could help facilitate this.(OI-IO9) Develop model plans and guidelines. Most of u_s don’t know how. Follow up calls (not written requests) for product possibilities. Then ask for written reporting. (NEI 1) NOT EXTENSION (3) ‘The role of Extension is confused at the moment. Don’t expect much. (ND05) "' Not Extension (for distributing SARE information). They’re on a different track (commercial, chemical, Monsanto) and SARE won’t change them.(SD06) * Funnel the information through ATTRA. They might set up a special SARE R&D division. They are doing the bestjob of outreach in the country. (ND05) PDP COORDINATORS (3) "‘ Emphasize use of Professional Development Program (PDP) to disseminate SARE results. To do this, the PDP’s in each state and researchers must become more aware of each other and communicate. Regional PDP coordinator could help facilitate this. Researchers can be encouraged to use PDP in their outreach. (OH09) "' Improve coordination between Chapter I and Chapter 3 programs. Ask PDP proposals to use Chapter I work. (MNO6) [SARE could distribute information through Extension] through PDP and state coordinators. Need additional staff person (half-time?) To follow R/E grants! (N E04) PARTNER (I) " Build critical partnerships. SARE is viewed by some/many as wanting to be unique. It is more effective to partner with the system to create change than to “appear separatist” and criticize. (IN 07) SHOW HOW (I) "' Develop model plans and guidelines. Most of g don’t know how. Follow up calls (not written requests) for product possibilities. Then ask for written reporting. (N El 1) NOT A PROBLEM (I) "' Not significant. Only the farmers can do this in a meaningful way and on their farms. “If the grower knows why, he will teach himself how.” Liberty Hyde Bailey, 1916. Therefore, this is a farmer education issue. (M103) 186 COMMUNITY FOCUS (I) "' SARE needs to get beyond the concern for specific outcomes and look more at the influence the TOTAL program is having on segments of the agriculture community, educators, input suppliers, marketing segments, as well as producers. The real question is, “has SARE reshaped audiences through its research, education and professional development programs?” (KS) NEW WAY OF THINKING, PRODUCING, CONSUMING, BEING, N=25 DEEPEN EFFORTS; REDESIGN (I3) * Only qualitative issues should be part of SARE. (Causing people to shift from linear thinking patterns to a more holistic approach) is not the issue; fundamental aspects of completely different cosmologies. (M103) "‘ Difficult, yes, but needed. (Ml05) * Set aside some of our funds for more comprehensive, longer term work (3-5 years). (MNO6) "' Develop a network of people who have made the paradigm shift and use them as a sounding board for developing a “deeper” sustainable ag research agenda. (ND05) " Select reviewers who understand and promote a holistic approach and limit funding to only those types of projects. (SDOI) "' Agriculture adjusts to the logistical demands of feeding large, relatively affluent urban populations. The adjustments are toward more highly organized value chains. NCR SARE contributes to sustainability when it generates information and technology that allows producers to cope with those changes, either by fitting into these highly organized value chains or creating new ones of their own. (IL08) * See above ("‘ SARE needs to get beyond the concern for specific outcomes and look more at the influence the TOTAL program is having on segments of the agriculture community, educators, input suppliers, marketing segments, as well as producers. The real question is, “has SARE reshaped audiences through its research, education and professional development programs?”) And to what extent is the agricultural scientific community beginning to legitimize the research and education conducted under SARE projects. (KS) " (Development without growth) requires a change in mind set and few people in land grants have made that change. Place strong emphasis on innovation and design, not just fine tuning current systems. Reward risk takers! (NEI I) "' Look to attitude surveys/evaluation. (MI 19) * Separate (development and growth). (Lack of redesign projects within SARE) can be “cured” by going back to initial purpose and evaluate impact/change separately from the research itself. People shift equals the 3" category — research, education and change. (SD06) "' Consult with Southern SARE about their experiences with systems research. Also, encourage more multi-disciplinary projects. (NEO4) LONGER TERM (3) * Set aside some of our funds for more comprehensive, longer term work (3-5 years). (MNO6) * I feel this is a time factor where people have to think this over and begin to act. (SD07) "' Unless SARE is willing to invest in longer term studies and provide insights into what makes these studies successful, neither (system redesign or shift to holistic approach) is likely to happen. This (development versus growth) is the difference between generating wealth and generating money. Focus on the ability to build equity. (NE10) NETWORK (3) * Develop a network of people who have made the paradigm shift and use them as a sounding board for developing a “deeper” sustainable ag research agenda. (ND05) "' Possibly a workshop/conference on holistic approach? (WI06) * Publish outcomes. (KSIO) NOT A PROBLEM (3) * Not a problem. SARE provides a space in which people MAY “think, consume, be” different. SARE can’t mandate that, it’s going to have to evolve. (IA01) * SARE has had a tremendous impact in Michigan with Extension, NRCS and farmers. However, probably not enough recognition has been given to SARE. (MI I 3) * I disagree with this statement (that we know little about what causes shifts to holistic approach). Educated people are able to expand their horizon and have a more holistic thought pattern. (W107) CLARIFY PROGRAM/PROJECT EXPECTATIONS (2) "' Encourage a holistic management approach in first defining a broad good, then having the research contribute to meeting it. (N E] I) ‘ Emphasize 187 research on management systems, agroecosystems, whole farm plans, etc. in calls for proposals. (OH09) REVIEWER SELECTION (2) " Select reviewers who understand and promote a holistic approach. (SD01) “ SARE needs to actively involve consumers and non-ag people. (W114) TARGETED FUNDING (2) "“" Bigger grants would help [system redesign efforts]. $l00.000 limits how much can be done. (MN04) FOLLOW UP (I) "' SARE should allocate funds for redesign work and track the impact of the work as it filters in the larger community; fund a human systems team that follows all research/education projects. (1A05) NOT EXTENSION/LAND GRANT (I) " (Development without growth) requires a change in mind set and few people in land grants have made that change. (NEI I) PARTNER ( I) " SARE cannot do it alone! Recognize and give credit to other entities. There are other organizations that specialize in” behavior modification”. Identify and consult. (IN07) WHOLE SYSTEMS QUESTIONS, N=25 WHOLE SYSTEMS (5) "' Consider requiring Chapter 1 projects to include all three factors (environmental, economic, social) and to include team members that are qualified to do so. (MN06) "' Search for the real systems research approaches and pass over component activities; they get funded ' elsewhere. Put out a call for whole farm research and economic and social and environmental! People will respond. Encourage proposals in this area (maintaining rural community). (NEI l) "‘ The problem is worse than envisioned by many with interests in SARE. The SARE program tends to view farms as complete systems and fails to recognize that farms are just part of a much larger food and agriculture sector. Some SARE advocates view processors, distributors, retailers, and input suppliers as the enemy, when, in fact, they are part of the system required to make agriculture work in modern times. Except in situations where farmers market their own products directly to end-users, farm enterprises are just stages in complex multi-stage value chains. Very few people in traditional agriculture are trained or inclined to study these very complex systems, where some of the most important messages are generated. Often, they don’t want to hear those messages. SARE should encourage industrial engineers, operations researchers, and other non-traditional scientists to analyze these value chains, using their sophisticated systems research approaches. (IL08) "' We are getting more of these [protocols for looking at whole farms] every day. The potential problem described does not address the pressing questions — only one small piece of it. To look at whole forms requires more time and money (see below). SARE funds need to be directed to a wider array of disciplines, to be sure, but every project can’t be expected to deal with “systems” or “wholes”. (MN05) " Through a portfolio orientation depicted on the preceding page. (SARE needs to have a portfolio approach in which there is a commitment to fund fundamental system redesign projects in order to develop the knowledge base for sustainable ag in the future.) (WII2) PARTNER (4) "' Perhaps do some SARE (farmers and merchants) “community” (local leaders, schools, institutions, churches) partnerships. (M112 "‘ Combine forces with ARS IFS. The must develop procedures for this too. (NEI I) "' Whole systems research is very costly. SARE doesn’t have the resources or expertise. Again, partner with appropriate entities. (IN07) * NRCS is a good partner as well as local Rural Development folks. (N E10) CLARIFY PROJECT EXPECTATIONS (4) * Better RFP — more outcome-based approach. (M119) "‘ Identify goals before tools/approaches. (N El 1) ‘ Consider requiring Chapter I projects to include all three factors (environmental, economic, social) and to include team members that are qualified to do so. (MNO6) "' Evaluate in separate categories — research, education and change. Economic, environmental and social can be included in each of these categories. As SARE began, with practitioners sharing their whole with neighbors, nigetting into the boxes of scientific research except where there is a very isolatable topic appropriate for scientific research. Change is a category. With change supported, rewards 188 will come. (SD06) NOT BASIC RESEARCH (3) * Basic research related to the mechanistic world view is inappropriate for SARE. (Ml03) "' SARE should not fund basic research period!!!! (M113) “Stay away from basic (research); there are other sources for support for that work! (N E] 1) BASIC RESEARCH (3) * We must do basic research; all projects need not be relevant to all farms. (W114) "‘ Relax! (About funding basic research). (SD06) "' Get over it [being phobic about funding basic research]! Basic research will happen -— it is part of NCR SARE’s job to support research that enhances SA and basic research is needed to do so. (MN05) OPTIONS TO SYSTEMS RESEARCH (3)* Begin by funding projects that look at more components or larger parts of the “whole.” (OH09) "' Whole systems research is very costly. SARE doesn’t have the resources or expertise. Study “sections of the system” within a context. (IN07) * (Use) well-documented case studies of whole farms. (W106) COMMUNITY FOCUS (2) *An effort should be conducted on a community basis. Developing indicators of sustainability and applying them to a community would be a better method of determining these dimensions. (W107) "' Reward them (people who maintain rural community) by helping create a community that supports them. (W114) SHOW HOW (2) * Provide training 7??? for agents, farmers, NRCS on these topics (whole farm approaches). You —- SARE — can’t (educate public on importance of sustainability). ???????? (OHO6) "‘ Sponsor projects or workshops that focus on how to do whole systems research. (MN06) SUPPORT TEAMS (2) * Encourage team approaches. (OH09) "‘ Multi disciplinary teams are never neat and clean. This is always a problem with a team approach, but it is vital to obtain the holistic information SARE requires. In short, don’t fret about this — press on. (SD01) NETWORK (2) "' Sponsor projects or workshops that focus on how to do whole systems research. (MNO6) "‘ Encourage multi-disciplinary, multi-professional projects. Co-fund projects with PDP and producer funds. (N E04) TARGET FUNDING (I) * Perhaps SARE should invite and fund institutions outside the university to do research of this kind as demonstration projects. (N D05) FARM FOCUS (I) * Including an on-farm component in ARE projects helps considerably in this area (educate public on importance of sustainability). (SD01) DEVELOP PEOPLE SKILLS (I) * SARE should look at providing support to encourage leadership and community building skills among land grant faculty, non-profits, grassroots workers and farmers. (IA05) NOT EXTENSION/LAND GRANT (I) * Perhaps SARE should invite and fund institutions outside the university to do research of this kind as demonstration projects. (ND05) LONGER TERM (l) "' Fund some projects for up to 5 years. Obviously the remedies are indirect and long-term. (1A0 I ) NOT LONGER TERM ( l) "' Basic research would normally take more than two years — our limited funding needs to be used towards answering immediate problems. With funding amounts in the $60 - $100,000 bracket, you can’t get into a very involved project and expect a lot of in-depth data. (SD07) NOT A PROBLEMS ( 1) "' Acknowledge that because ag is site specific, there are many good approaches but no one “best” approach. A little ambiguity is OK. (NE10) RECOGNITION (I) " SARE program can’t do much directly (about rewarding people who maintain rural 189 communities) but they could recognize these folks. (W106) " Farmers react to dollars faster than anything. Hart to turn down $10,000 an acre for the farm if your family is starving! (KSIO) LONG-TERM SUSTAINABILITY, N= 23 LONGER TERM (12) " I believe some of the grants need to be long-term. Great people are lost because of the short-term grants. In sustainability, we need these people on board. (M113) "' It might be ti me to consider shifting our emphasis and maybe setting aside some funding for longer-term projects. (MN06) " First, we need to recognize that sustainability is a process with a beginning but not end. What we may think is “sustainable” today will shift tomorrow, partly because nature is always changing. Second, why not invite collaboration on at least one, carefully thought out, long-term research project? (ND05) "' Encourage fl involved in SARE to become more of a futurist. If we broadly define sustainability, then a particular action or practice might only have to have a positive impact on one (or many) of the objectives that would move toward the goal of sustainability. (M112) "‘ So do some long-term work. (N El 1) “ Fund some projects for longer than 2 years. (1A0 I) "' “Sustainable ends” cannot be served with two year discontin??? funding. Partner for longer term activity. (IN07) "' Change mind set (that funding more projects id better than funding fewer long-term projects). (SD01) "' The long-term doesn’t take a lot of funding. Getting started, yes; further observations, some, but not a lot. Yes. (SD06) "‘ Fund at least 1-2 long term projects. (IA05) * The temporal dimension of “sustainability” is its fundamental basis!! One human generation is a minimum. Seven is better! (M103) "‘ There’s no rule that says you can’t do some of each. It is time for NCR SARE to look at increasing its impact through larger, longer-term projects. The organizational “mind set” may not be what you think it is given that the AC changes its membership. (MN05) RENEWAL (4) "' I think that two-year projects are ok provided that SARE is not biased against continuation proposals for promising, productive projects. (W112) * Make clear guidelines for the extension of time lines on longer projects. The corporate management world has bought into the Deming management model and found that they can function well. (NE 10) * We did some repeat funding. One project received two additional amounts for a total of six years of data. It was very useful. Current members should be free to recommend additional funding if a project is collecting valuable date and needs more time. (SD07) "' Look at continuing funding or renewals for high quality projects. (OH09) PARTNER (3) "‘ Work with other regions to not duplicate topics but agree to fund some basic research in each region and then share the hell out of the results! (OHO6) " Partner for longer term activity. (IN07) " Look to support projects where other funds are available. (Wl06) BROAD DEFINITION; SUSTAINABILITY AS PROCESS (3) "' A broad definition (of sustainability’s end goal) should be OK. Do not want to stifle innovative projects, though. (SD01) "' First, we need to recognize that sustainability is a process with a beginning but not end. What we may think is “sustainable” today will shift tomorrow, partly because nature is always changing. (N D05) * Trying to identify practices that will make agriculture sustainable in the long term is very difficult when technological and economic changes are so rapid and unpredictable. F arrners need ways to respond rapidly, effectively, and sustainably to change. In the long term, flexibility is the only sustainable approach. They cannot be restricted by long-term strategies, such as long-term rotations. (IL08) FOLLOW UP (2) "' 5 year surveys. (M119) * Require Pl to agree to a follow-up after proposal is over at 1,3, 5 years. (KSIO) BASIC RESEARCH (l) "' Work with other regions to not duplicate topics but agree to fund some basic research in each region. (OHO6) NO LONGER TERM (1) "' Stay with short-term, seed money grants; have one requirement be grant proposals for long-term funding. (NEI l) 190 MORE CENTRAL CONTROL (l)"' Rather than a competitive grant program, SARE should move to a managed grant program. (WI07) DIVERSITY OF APPROACHES, PRACTICES & SYSTEMS, N= 27 REVIEWER SELECTION (IO) " Be more strategic in selecting AC and TC members to make sure that we do represent a diversity of approaches. I think that most of us are biased in one way or another, so we need to include many different biases. (MN06) "' Recruit and/or train evaluators for diversity knowledge. (N E] I) "' Careful selection of proposal review panel is critical. Subject the process to extramural review versus the “internalized group.” (IN07) "' Include diverse groups and viewpoints on Administrative Council and Technical Committee. (OHO9) ‘. Rotate evaluators. (SD01) "' Carefully choose evaluators and/or be sure they understand SARE’s definition. (SD06). "‘ Through selection of TC and AC members who are open-minded. (W112) * An administrative problem to be overcome by removing narrowly focused reviewers. (IN09) "‘ Broaden pool of educators; go outside traditional pool into general consumers (for evaluators). (W114) "‘ Focus on the method, not the practice or technology. What are the major issues that condemn organic as not being sustainable? Identify and address. Probably not much improvement here (evaluator favoritism) as this probably relates to basic personality type. (N E10) ORGANIC/=lSUSTAINABLE (8) " Stick with “sustainable agriculture” — if your definition is right for you, then let my definition be right for me. Avoid pigeonholing into conventional, biodynamic, organic, etc. (M112) “' Broaden to ag and f_ogd systems, not just organic. (NEI l) * Organic and sustainable overlap but are not the same. (MII9) "' Organic food shipped 1500 miles is not sustainable when locally grown is available. Must look at more than how food was grown. (WI 14) "' There is no perfectly sustainable system, including organics. (IA01) * I can see how some could see SARE as being organic in nature. But in the area of ecology, any artificial input generally is harmful. The criteria set up to guide research lent itself to favoring organic production practices because of no chemical use and low inputs as well as diverse practices. (SD07) " SARE places a serious limitation on researchers through its bias against inputs. Many SARE-sponsored experiments are unrealistic in that they do not include treatments that require purchased inputs. In other words, they do not include many of the alternatives available or potentially available to producers. At least there is bias against experiments that explore all alternatives that don’t fit certain preconceived notions. I know this is an unrealistic recommendation, but if that bias could be removed, the program would be greatly improved and would attract more funds and more researchers. (IL) " SA is M than organic, as determined by the SARE program’s goals. We’ve erred on the organic side mor than not, which I think has been at the expense of good projects — the mention of a herbicide in proposals has been the death of great proposals in the past. I think we need more reviewers that can explain the real impacts of proposals so better thought can be given on the merit of proposals — not just a knee jerk “all pesticides are bad” rxn. (MN05) IMPORTANCE OF DIVERSITY (7) "' This gets back to the alternative world view issue. The current dominant world view (mechanistic world view) does not tolerate or allow for diversity!! (M103) * Develop community building and leadership programs that allow for people to tolerate ambiguity and embrace diversity. (IA05) "' Encourage innovative and unusual methods; too much research is the same stuff again and again. Cannot be too diverse in thinking — we need that! (N E] I) * Include diverse groups and viewpoints on Administrative Council and Technical Committee. (OHO9) "' I think that most of us are biased in one way or another, so we need to include many different biases. (MN06) * Programs must be made across the spectrum. Recruit and/or train evaluators for diversity knowledge. (N E] I) * Diversity of approaches could stem from diversity of project investigators from a diversity of organizations and institutions. SARE should broaden horizons beyond the land grant system in funding decisions. (NEO4) GOALS (3) * Can be resolved by a careful definition of sustainable. "‘ Our understanding and practice of sustainable agriculture must evolve to remain relevant. (IA01) "' Methods and approaches are secondary to the purpose of the project and how well the project advances the objective of the SARE program. It is important that the methods/approaches be appropriate for the issues being addressed. (Wl06) 191 ORGANIC=SUSTAINABLE (2) " Organic may be the only approach, but will not determine this if it is not compared to other systems. (SD01) " Let’s call it the organic way and the natural way. (KS 10) REVIEWER SELECTION (2) " Not “shared” but certainly respected. Selection can occur depending on the problem and orientation of the user. (Evaluator favoritism) is always a problem. What about a 3 or 5 year review addressing this very issue? (M105) * Carefully choose evaluators and/or be sure they understand SARE’s definition. (SD06) CLARIFY PROJECT EXPECTATIONS (I) "' Perhaps SARE could add a screen to RFP’s — “Will this project seek a new or alternative solution to a production problem?” (or is it simply a variation of a conventional approach?) It might also be helpful to develop broad principles of sustainability to use as a screen. Bill McDonough suggests three for sustainable subsistence: 1) all energy must be ????; 2) all waste must become food; 3) biodiversity is essential. (ND05) MORE CENTRAL CONTROL ( I) " Go to a managed grant program! (W107) NETWORKING (I) “ I agree; we do need better NC communications between projects. An annual NCR SARE conference is needed to present progress on projects funded. (M113) REACH MAINSTREAM (I) * (Strong SARE supporters) should not be our target audience in 2000! This is preaching to the choir. Use these folks as teachers and change ????? to get everyone else on board. (OHO6) SPECIFIC, MEASURABLE IMPACTS, N=24 TARGETED FUNDING (3) * We may need to fund projects that are designed to develop alternative ways of measuring impacts. (WIO6) "' Maybe we should set aside a certain percentage of our budget for evaluation and measurement of impacts. (MN06) "' Really tough — maybe ask for a couple of projects to develop evaluation tools? (N El 1) FORCES BEYOND SARE (4) " SARE participants need to be concerned about impact, but we can’t realistically expect huge impacts when farmers have to operate in an adverse policy environment. (W112) * The major objective should be to allow for alternatives to be considered in a serous manner. This is not possible with the current world view. (M103) "' We probably won’t solve this problem until we come to grips with the fundamental flaws in today’s money economy — we only ???? what “captures economic (i.e. financial) value” in the short term. Until we recognize that social and ecological values are as essential as financial ones for a “wealthy” society and agriculture, there isn’t much hope. But we can, and must, keep pointing out the problem. (N D05) "' Paragraphs two and three under potential problems reveals the SARE bias. The population served by modern agriculture is almost entirely urban. What about their community and quality of life? It is their desire for abundant, uniform, safe, high quality, affordable food, as expressed when they buy groceries, that drives agriculture toward the industrial model and toward highly coordinated markets. Certain restrictions in the subject matter of SARE research cause a significant part of the program to be irrelevant to a large segment of modern agriculture, namely the segment that serves those big grocery stores where most people buy their groceries and lots of other things. On the other hand, removing those restrictions would make it difficult to differentiate the SARE program from other federally sponsored agricultural research programs. The SARE program is different, however, in being more mission-linked (not just mission-related, like the NRI). SARE should remove the ideological restrictions and emphasize its mission-linked, goal-focused, systems-oriented nature. (IL08) QUALITY OF LIFE (4) * SARE should look at a new array of projects that delve into the psycho-social or mental health values of a more sustainable agriculture. Does a higher quality of life have a monetary value in terms of less resource degradation, less medical bills, etc? (IA05) * Quality of life if the most difficult concept to capture for obvious reasons, but in some ways it’s also the most important aspect of sustainable ag that we should measure. SARE should work with rural sociologists and others to try to gauge QOL impacts. (NEO4) "' Paragraphs two and three under potential problems reveals the SARE bias. 192 The population served by modern agriculture is almost entirely urban. What about their community and quality of life? It is their desire for abundant, uniform, safe, high quality, affordable food, as expressed when they buy groceries, that drives agriculture toward the industrial model and toward highly coordinated markets. Certain restrictions in the subject matter of SARE research cause a significant part of the program to be irrelevant to a large segment of modem agriculture, namely the segment that serves those big grocery stores where most people buy their groceries and lots of other things. On the other hand, removing those restrictions would make it difficult to differentiate the SARE program from other federally sponsored agricultural research programs. The SARE program is different, however, in being more mission-linked (not just mission-related, like the NRI). SARE should remove the ideological restrictions and emphasize its mission-linked, goal-focused, systems-oriented nature. (IL08) " Base line studies could have or could now be taken by academics experienced in this — probably sociologists. This is not difficult for those who know how. Time schedule can be set by SARE. Exactly, this is what SARE is for!! Yes they could!!! Change or quality of life has to be evaluated as a separate category. (SD06) CLARIFY PROJECT EXPECTATIONS (2) "‘ SARE, both on NCR and national levels needs better/clearer picture of what sustainable agriculture will/does look like. A better, clearer, more readily communicated goals and indicators. (MI 19) * All projects must have measurable outputs — quantitative or qualitative. We must be able to answer how many objectives we achieve and how well. Consumer interest is bringing appreciation of quality of life into the marketplace. We can be more explicit about what it means — in ways that are hard to co-opt. (N E] l) PARTNER (2) "' Beware of “taking credit”. There are likely many entities that have contributed to an outcome. Share credits. SARE in some instances has publicly claimed sole credit when they were “a participant”. (IN07) * This compartrnentalization of impacts and praise is disturbing, especially if we’re fostering a systems approach. It should be enough to document SARE’s input/support/insight at a particular moment in a project’s history. (M105) SHOW HOW (2) * We may need to fund projects that are designed to develop alternative ways of measuring impacts. These results then need to be shared with future project holders. (WIO6) "' This evaluation process should provide tools to improve impacts -- or to estimate them to the extent possible. (OHO9) SPECIFIC INDICATORS (2) "' Measurement should be in terms of cleaner water, more diverse wildlife, healthy animals, viable communities, cleaner air, viable farms on a whole. (SD07) * Shine the light of public opinion on the various systems and the truth will be evident. How many smiles/season does each system provide to the farmer? What is done with the profits extracted? What are the underlying values that determine happiness? Determine the difference between optimum production versus maximum production. (N E10) IMMEASURABLE (2) "' Seems it takes faith to play this game. Trouble is, we don’t have faith in our fellow man. If we truly do our best, we must believe our “neighbor” will do their best. Therefore we should expect the best of outcomes. At any point we must acknowledge that our best efforts can end in failure. (M112) " This is an item of faith. You can’t measure this any more than an individual can measure the goodness of a life. Do it because it needs doing. (MNIO) PARTICIPANT EVALUATION (2) *Those who are actively engaged in the project, or adopt aspects of it, or who reject it should have a voice in its evaluation. (M105) * First people talk about sustainability as a positive, where in the past it was a negative. The state of agriculture certainly has enhanced the need for sustainability. (M113) FOLLOW UP (I) "‘ Just keep asking. Must follow up on more projects to help keep Pl’s honest and accountable. This will take a long time to evaluate — 5 to 10 years. (KS l0) FARM FOCUS (I) " Grant more farmers’ grants and less university grants. (KSIO) 193 NO PROBLEM ( I) " These types of things have always been difficult to measure, and over only a 10 year period may not be determinate. Maybe small, family-sized, specialized operations and the large mega- farms g our future — at least until energy costs (5 and otherwise) rise. Given the funding levels of SARE, I think the program has done reasonably well. (SD01) REVIEWER SELECTION (l) "‘ We must bring these traditionally involved with social/religious issues into the picture. (WI 14) ‘ Shoot like hell and claim all the ducks that fall! (OHO6) SUSTAINABLE AGRICULTURE PRACTICE, N=20 SUSTAINABLE IS PROCESS, NOT A RESULT OR PRACTICE (7) ’ Both organic and conventional systems are dynamic. Sustainable is also a dynamic concept, so it is likely a system would include components of each approach. (SD01) " Certified organic '2??? prove that it is an approach toward an end, not just a recipe. Both organic and conventional have little eye to the future or ???? ‘Sustainable’ is by definition a process rather than a result. (NEI I) "' It’s impossible to define sustainability in terms of a “practice” — as it is impossible to define health or family as a “practice”. Give it up! We need to define “ecological practices and distinguish them from “industrial” ones. (N D05) " Do not confuse practices, systems and concepts!! Sustaining deals with the temporal dimensions of something! (M103) "' I think this is a slippery slope, given that SA is a goal —- not a set of practices. We must continually look for new tools, not silver-bullet practices. (MN05) "' Don’t waste more time on definitions; we’ve already spent too much energy there. (NEI I) * It’s a mind set, not a definition. (M113) SUSTAINABLE INCLUDES CONVENTIONAL, ORGANIC (3) "‘ Acknowledge that either system can and does use components of the other. Example, conventional using cover crops; organic using a tractor that uses non-renewable petroleum. (M112) * Both organic and conventional systems are dynamic. Sustainable is also a dynamic concept, so it is likely a system would include components of each approach. (SD01) "' Yes (a sustainable agriculture could include both organic and conventional aspects). (KS l0) CLARIF Y PROJECT EXPECTATIONS (3) "' SARE, both on NCR and national levels needs better/clearer picture of what sustainable agriculture will/does look like. Better, clearer, more readily communicated goals and indicators. (MI 19) "‘ Let’s try to agree upon a set of sustainability indicators and evaluate practices based on expected changes relative to these indicators. (MN06) "' Very few practices now identified as sustainable would work for a large number of people. For example, not all farmers can raise ruminant animals economically. Not all farmers can apply manure, because there would not be enough manure. Not all farmers could use low-input systems, because there is not enough land to produce the needed quantity of food at low yield levels. Perhaps the definition of sustainable should be made more restrictive. Perhaps it should apply only to practices that could be successfully adopted by a significant proportion of the nation’s farmers. Or, at least, the number of people potentially involved should be specified. Somebody needs to work on things that serve market niches, but a small program focused that way will not have much regional or national impact. (IL) NETWORKING (I) * Is it possible to partner in developing a conference to discuss sustainability within the context of today’s agriculture, biotech developments? (IN07) NOT EXTENSION/LAND GRANT (I) "' Of course; SARE can do its own definitions and go from there. SARE exists to fill a fault in Extension and the Land Grant colleges, yet seems hesitant to stay separate. (SD06) NOT LAND GRANT (l) "' Of course; SARE can do its own definitions and go from there. SARE exists to fill a fault in Extension and the Land Grant colleges, yet seems hesitant to stay separate. (SD06) ORGANIC/=/SUSTAINABLE (I) "' Organic is not necessarily more sustainable than conventional. You must look at all aspects of food production, not just chemical inputs. Sustainable must look at farm price 194 and sustainability as well as social issues and local marketing. (W114) PARTNER ( l) " Is it possible to partner in developing a conference to discuss sustainability within the context of today’s agriculture, biotech developments? (IN07) "' This issue is whether the system (organic or conventional) leads to results consistent with the SARE mission. (W106) "' Fund a variety of approaches and innovative methods and systems. (OHO9) ‘ Systems become more sustainable when net income is adequate and the above criteria is being met. (SD07) * Yes. (OH06) " Insist on repeatability for the for seeable future. (MNIO) GREATEST BARRIERS/CRITICAL COMPONENTS, N= 22 DIAGNOSTIC FOCUS (4) "' We need to refocus some of our grant efforts on diagnosis instead of 100% shotgun treatment, i.e. policy. (MI 19) " I think there is room to do more detailed ethnographic-type research on decision strategies and farmer views of their research priorities. (W112) "‘ I don’t think we’ve done enough to identify barriers and why people change or don’t change. We should put more emphasis on this in our calls for proposals. (MN06) "' All three paragraphs of potential problems associated with this question reveal biases. I read them to say I) if we can’t educate people to do what we think is the right thing, maybe we can force them to do it; 2) we know the right thing to do but we can’t convince most of the farmers, and 3) seed and chemical companies and major farm organizations don’t want agriculture to be sustainable. Philip Kottler, Northwestern University’s guru of strategic marketing, lists clews that can be used to identify an organization-centered organization (program) as opposed to a customer-centered organization (program). An organization-centered organization I) regards the offering as inherently good, regardless of how well it is accepted by customers, and 2) attributes its failures to ignorance on the part of customers and clients. When a lot of the customers don’t adopt something, it may be because it’s not practical for them or relevant to their situation. Trying harder to sell it won’t change that. (IL08) INDIRECT POLICY FOCUS (3) * We cannot ignore policy, but by involving more people in SARE through the grant programs and education, they become policy advocates for the program. People need to be financially rewarded for doing things right. We need the consumers to reject the pattern (of ag concentration) as they are in Europe. (W114) * We could fund projects that investigate the probably impact of various policy changes, but direct pressure to change policy is not a good idea. I’m not sure this (overcoming power of seed & chemical industries) is our charge. (SD01) "‘ The educational programs must be a process that leads to policy change. The components of the educational program must be organized in a way that the end product of the process is policy change prompted by individual or group action. (MIIZ) NOT POLICY FOCUS (3) " SARE funds should not be used for policy change. We can’t (overcome power of seed & chemical industries) with limited funding. (OHO6) "‘ I don’t think we’ve done very well in this regard. Fundamentally the ag agenda has not yet come to value a diversity of approach to agriculture. It’s probably essential to a resilient agriculture and therefore to food security. But we are still caught in the paradigm of producing as much “stuff” as possible. (ND05) "‘ Policy seems like a black hole for research. Keep funding alternative research. (N E1 1) FARM FOCUS (3) "' Still think that more farmers’ grants will solve this. $100,000 in farmers’ grants cover many more areas than just one grant to university. (KS 10) "‘ Fund projects that are well-connected with sustainable agriculture farmers and those farmers you want to influence. (WIO6) "' SARE identified some of the barriers but I believe others are: financial, markets, concern for weed control, change in 195 farming practices, community pressure. (SD07) FORCES BEYOND SARE (2) "' Recognize that things are changing. Identify a role within the context of change. (IN07) "' Macro changes in the structure of agriculture are drawing most farmers in an unsustainable direction — even if they have the information from SARE on how to farm more sustainably. That’s not to say SARE could reverse those macro changes. (IA01) COMMUNITY FOCUS (2) " Move beyond single farmer/farm projects into community bioregional projects. (MIOS) "‘ The greatest barriers are at the local community - social level. SARE can promote itself as a viable alternative to those wishing to enhance their quality of life or standard of living. (NE 1 0) DEFINITIONS (1) “Traditional” versus “conventional” is very important. The word “traditional” used in this way indicates we have a societal memory of only 50 years. Part of the problem! (MI03) EMPOWERMENT FOCUS (l) * SARE is primarily about empowering individuals. (IA01) EXTENSION (I) * Fund projects that are well-connected with sustainable agriculture farmers and those farmers you want to influence. These connections may be direct (farmer-to-farmer) or indirect (i.e. Extension). (WIO6) IMPACT MAINSTREAM (I) "' Recognize that things are changing. Identify a role within the context of change. “Larger operations” may have greater potential to contribute to environment and sustainability. (IN 07) MEDIA (I) "' PSA announcements are expensive and are met with counter-announcements. Probably still the best way to reach the public. (IN09) POLICY FOCUS (I) * Yes (about redirecting SARE resources to policy change and potentially cause problems). (Individual projects and reports) can (help) if they are distributed to Congress and the state legislatures on an individual basis (one-on-one). (M103) REVIEWER SELECTION (l) " Many of the farmers selected to evaluate the grants have certainly helped identify barriers. We need a more diverse group of farmers on these committees, especially the Administrative Council. (M113) "' (Problems) with whom? SARE either is or isn’t. Keep evaluations separate — research ag practices and their sustainability; education/outreach; change. (This is a separate category; that’s what this category is about.) Stand fast for sustainable; give those practitioners full support. (SD06) “ Keeping the focus on the stakeholders is key. Use the ‘right’ measures - look bigger. Focus on the goals — ‘sustainable’ works toward many ???? ???? Conventional does not. (NEI 1) WHO IS CURRENTLY INVOLVED, N= 25 STAY FOCUSED (I2) * Right people a_r§ involved. Mainstream has other supporters and does not need this support! (NEI l) * Nobody has the power to claim who the “right people” are. If they participate and believe they are right. Who cares about mainline agriculture? Mainline agriculture has resulted in middle- sized farmers leaving farming. (MI13) "' Focus should be on small and moderate-size farms. Emphasize public rather than corporate benefits. (OHO9) * Mainline ag is (or should be) the antithesis of a sustainable approach. It is diversity and solutions fitted to place that will be most effective for maintaining diversity and enduring systems. Forget centralizing at every opportunity — keep loose and active. (M105) * While the original legislation identified 5 groups, I currently believe only 2 groups should be at the table (farmers and consumers), the only primary stakeholders! (SARE impacting mainline ag is) very difficult since conventional ag is currently in a “survival mode.” (M103) * We have the right people except we need to invite our enemies into the fold. (Gain an understanding of Dennis Avery and his motivations.) I96 (NE10) "' People d_o_i_r_rg, ready to d_o — to practice sustainability (are the right people). Have to go back to SARE’s definition, whatever that is. Apply sustainability by SARE’s definition. These (current SARE) “players” have to be committed to sustainability by SARE’s definition, not am from sustainability or into commercial/chemical company practices. (SD06) "' If “mainline ag” means industrial ag, SARE should not bother with it. Industrial agriculture involves neither community nor family, except incidentally, so it can best achieve only environmental sustainability. Let them deal with EPA. (IA01) "‘ We need to spend more time talking about who “mainstream agriculture” is. To some, it means large farms, and I don’t think they need our help as much as smaller farms do. I think SARE works more with small and moderate sized farms and I’m comfortable with that. (MN06) " SARE is perhaps one of the most effective structures we have to mandate strong stakeholder involvement. Given the forces at work, we won’t likely influence “mainstream” agriculture very much. It will play itself out. SARE should concentrate on viable alternatives. The “new market economy” can be an alternative for a majority of farmers. “Mainstream” commodity production will probably be done by 50,000 farmers or less, in vertically integrated value chains where production methods are dictated by manufacturers. (N D05) "‘ I think that SARE would be best off if it is stated that its core audiences are smaller and moderate-size farms but that it welcomes its technologies being useful to larger farms. It’s contradictory but realistic. (W112) " SARE needs to develop viable alternatives to the current agricultural system. They may not be adopted now, but as the current system shows its weakness and fails, SARE alternatives will be available. (W114) IMPACT MAINSTREAM (4) "' Get state Extension directors and NRCS state conservationists involved. Not true in Ohio (that NCR SARE doesn’t have mainline ag impacts). (OHO6) " Identify the number of new people coming into sustainable agriculture. SARE should encourage new faces and new leadership instead of keeping the same people on the lecture circuit and at the conferences. These events are almost always the “choir”. (IA05) * SARE needs to aggressively sell sustainable agriculture to more conventional audiences n_ow because we have a very “teachable moment” where mainstream farmers are open to ideas. We need a media/information campaign directed toward mainstream ag. (N E04) * We can’t [decide who the “right people” are]. We need to reach as broad an audience as possible, not just those that pass some litmus test. (MN05) CLARIFY EXPECTATIONS (3) "' Need better goals and diagnosis of roadblocks. (MII9) * The big question is whether you want to support those interested in a local, value-added sustainable agriculture or move those in mainstream ag towards sustainable ag. The answer here tells you a lot about what you fund and how it gets communicated. (WIO6) * Depends on program objectives. If you want to impact ag on the larger scale, then move from being a “club” to include “mainliners”. “Mainline” involvement is critical to having impact across agriculture. Don’t exclude representatives from the “mainline”. Quit being “standoffish” and invite “mainliners” to the table as participants. (IN07) * The theme of this concern is embodied in one of the questions, namely, to what extent is SARE relevant to mainstream agriculture. I think each proposal should address the question, to what extent can the resulting technology or information be implemented successfully by a significant proportion of producers, or, what proportion of producers could, realistically, adopt the practices/technology tested in this project. Proposals should be evaluated in part on how realistically those questions are answered. This should not rule out projects on practices of limited application, but should help decision-makers balance the portfolio of SARE investments. (IL08) DIAGNOSTIC FOCUS (I) *Need better goals and diagnosis of roadblocks. (MII9) INVOLVE GOOD COMMUNICATORS (I) * Get the “right” people as spokespersons. People who are creditable (not movie actors) and who can explain sustainable agriculture in positive terms that are readily understood. (IN09) INVOLVE PEOPLE COMMITTED TO CHANGE (l) "' Anyone who identifies a need for change then commits to helping make change happen would be the “right” person. However, the individual must acknowledge that any potential change may require changes from within (ourselves or our institutions) as well as external changes. (M112) MORE CENTRAL CONTROL (1) * A managed grant program could solve this. (W107) 197 REVIEWER SELECTION (l) " Changes in personnel should be part of the program. (SD01) NO PROBLEM (I) "' I think SARE is having major impact at the university level because of the changes made on the (administrative) council and technical committee. It has opened avenues of contact with other expertise. I have found the council, as diverse as it is, to be a very close-knit group. (SD07) GROUNDED IN RESEARCH, N=20 COMBINE & SYNTHESIZE RESEARCH APPROACHES (6) "‘ There is an undercurrent of suspicion concerning “replicated, reductionist research methods by a land grant university” among some who support sustainable agriculture practices. There are good reasons for replication in experiments. The reductionist approach, which is a completely different issue, is OK as long as there is some synthesis of the findings of reductionist research. In fact, the combination of reduction and synthesis is a powerful approach, and probably the only approach, to improving systems. Sound research techniques, including replication, randomization, blocking, etc., are not unique to land grant universities. They are sound approaches to experiment design in any situation and essential in most field experiments. They are used to assure the validity and relevance of experimental measurements and results. (IL08) "' Chapter I projects seem to favor traditional research and Producer Grants seem to be biased against it. I think both programs could benefit from movement toward a middle ground. (MN06) "‘ Case-history and replicated trials are both valid research methodologies and both have strengths and weaknesses, but together can aid in developing more sustainable systems. (SD01) "' What we are really dealing with here is the validity of “Precautionary Science.” SARE should promote this as an accountable, inclusive approach to problem solving. (NE10) " Observational findings have value. At some point there needs to be replication in order to develop principles. Current SARE on farm research is a “a stand alone event.” There are methods that don’t lead to reductionist science —- partner to gain this expertise. (IN07) NETWORKING (3) * Perhaps a conference on diversity of research methodologies, including numerous approaches, could be convened. There are NGO’s (like the LOKA institute) that are doing some creative work in this regard. (N D05) * I think it would be good for SARE to support a workshop, white paper, or something similar in which there is a good discussion of reductionism, experiment designs, and institutional cultures as they impact SARE research. (IL08) * Enhance pool of reviewers and/or provide some more expertise and make it available to the AC. I’ve seen good proposal dismissed because the AC doesn’t understand some methods and concluded the research must not be worthwhile. (MN05) * FARM/FARMER FOCUS (3) "‘ The original mandate from Congress was to find viable farming practices that would sustain family farms. As we saw that responsibility, we looked into whole farm systems. Secondly we were mandated to involve ag practitioners and that created generalization of research. (SD07) * If a farmer has a project that works for him that may be more valid than a replicated study at a university. Basic research can be done by basic methods on small farms. Lets not get hung up on wanting all research to be done according to the industrial model. (WI 14) "' Use grant process to make/keep research, but make research farmer friendly. (MII9) BASIC RESEARCH (2) "' Basic research can be done by basic methods on small farms. Lets not get hung up on wanting all research to be done according to the industrial model. (WI 14) * Get some good researchers to visit with AC and TC — folks like Ben Stinner or Dick Harwood, about the need for basic research. (OHO6) * I agree that the SARE program would benefit by supporting some basic research, as long as the basic research has clearly defined practical goals. (IL08) NOT BASIC RESEARCH (2) "‘ NCR SARE should not fund basic research — No! No! No! (M113) * We need more creative approaches, case studies, whole-farm analysis, interviews and social science methods. (NEI l) FOCUS MORE ON ED THAN RESEARCH (I) "' Research is not the real issue. Most funds should go to development and education. (M103) 198 INNOVATIONS (I) "' Continue to fund projects on innovative practices that have not previously been researched. (OHO9) RESEARCH TO INFLUENCE COLLEAGUES (l) " We need to be able to present our research to our colleagues, though. (M113) NOT REDUCTIONISM ( l) “ There is comparatively lots of $ for reductionist research. SARE is a unique opportunity to exploit the innovation of non-traditional researchers and to explore new methodologies. (NEI I) PARTNER (1) "' Observational findings have value. At some point there needs to be replication in order to develop principles. Current SARE on farm research is a “a stand alone event.” There are methods that don’t lead to reductionist science — partner to gain this expertise. (IN07) " The sustainable agriculture movement has a much greater problem with promulgating coffee shop opinions as received truth than it does restricting the ways information is gained. (IA01) "' I believe it is important to fund projects when the methods/approach are appropriate for the issue being addressed. Can we learn something from their experiences and can these learnings be communicated? (W106) "' Is it a researchable project? Is it an education/apply project? It is either one or the other. Separate research, education/apply, observe change. (SD06) * I’ve commented on this above. (W112) EMPOWER PEOPLE, N=20 FACILITATION FOCUS (6) "' Australia has already made this switch (from tech transfer to facilitating learning). We have to do it or lose our public Extension either entirely or Argentine model. (NEI I) "‘ To empower people, we only need to give them an array of tools. Let them pick and choose the right ones for their needs. I like the idea (my idea) of facilitated learning (gives teaching a whole new concept). (M112) "‘ We need to help extension personnel expect to be questioned and we need to help everyone learn to ask questions in the form of respectful dialogue so they don’t immediately alienate others. (MN06) "' Absolutely. Facilitate the discovery and learning process. We need a complete mind set change here. But it goes against all our experience and also does not meet expectations of most audiences. (N El 1) * Again, 3 categories: research, education/apply, change (this is people). (SD06) * Ideas like these (need to move from tech transfer to facilitating learning) are true but will not be accepted in our current definition of ag research. SARE projects promote farming systems that do not put enough money back in the pockets of agribusiness. They do help farmers survive by spending less and keeping more money. They also promote a system that can deliver better food to the consumer. We are trying to justify a change in societal social values, i.e. farmers and social justice rather than vast profits. Right now, society values money more. We are not going to change the world; we need to keep working as we have been, converting a few at a time. (W114) QUALITATIVE FOCUS (5) "' The term technology transfer is totally inappropriate for most alternative ag systems. All of Rogers and Shoemaker and Zaltrnan and Duncan’s (work) still is good and appropriate. Farmers are outstanding at changel! Since it is a qualitative issue, the techniques of Jules Henri Poincare, Benuit Mandelbrot, etc. should be used. With this, the basic research argument is no longer an issue!! (MI03) " Check attitudes. Don’t mix project/tangible outcomes with long term shifls in attitudes and practices. (MII9) "' Some good case studies could answer some of these questions. (WIO7) "‘ Ask the participants about the changes in their understanding! (OHO6) " Peter Drucker recommends, under some circumstances, offering products to a test market and researching the people who don’t purchase them. Maybe some research directed at people who are aware of but do not use practices generated or supported by SARE research would be in order. (IL08) 199 RE-INVENT EXTENSION (4) "' Australia has already made this switch (from tech transfer to facilitating learning). We have to do it or lose our public Extension either entirely or Argentine model. SARE can help develop a new Extension-client relationship. Ties to public interest in quality of life, environment, rural communities, this can result in a broad public support, which we are losing. (NEI l) "‘ Extension is becoming increasingly irrelevant to most farmers. But this presents an opportunity for a small group of Extension people to re-invent Extension. Why not use some of the Extension training money to invite the development of an “empowerment” rather than a “technology transfer” model of extension within NCR SARE? Their objective would be to reinvent Extension on a small scale and demonstrate how it could be done on a larger scale. (N D05) "' The extension component of SARE has always been a foot-dragging deal, at least in my experience. But as least some funds directed toward “learning” may be worthwhile. (SD01) "' I think extension in my state has taken some steps toward the empowerment model. We need to help extension personnel expect to be questioned and we need to help everyone learn to ask questions in the form of respectful dialogue so they don’t immediately alienate others. (MN06) CLARIFY PROJECT EXPECTATIONS (1) "‘ There are new evaluation methods (the LOGIC model here at UW) that include a whole series of outcomes, including awareness, understanding and changed actions. The big questions are: 1) What do you want to achieve in the project? (outcome)? 2) How will you know the outcome has been achieved? (Evaluation) (W106) COMMUNITY FOCUS (l) "‘ The agricultural is cultural. It may be time for farmers to team up with non- farrning members of their communities to begin exploring how to re-introduce ag/food/natural resources back into local consciousness. (M105) FOLLOW UP (I) "‘ Because we are attempting to help make changes that will have long term impact, perhaps part of an evaluation process should come at a later date. (M112) LONGER TIME (I) "' Sustainable agriculture will require a time line of years for the mainline adoption. Millions of dollars are spent in advertising products used by farmers — mostly chemicals — to the point that many farmers — most farmers — believe that it is impossible to grow crops without chemicals. Most farmers are reluctant to experiment for fear of failure and lack of cash. When change does take place, it happens with amazing speedll It’s as if everyone decides at once to change. (SD07) NO PROBLEM (I) "' I think SARE is already doing a pretty good job on this front. (W112) PARTNER (1) * SARE must recognize that they are not the only group causing people to question approaches, recommendations, etc. (IN07) SHOW HOW (I) " Facilitate the discovery and learning process. We still need to attempt to do this evaluation and maybe have a project on how to do it. (N E1 1) "‘ What lessons can we learn from such programs at Deep Spring College in the Sierra Nevada mountains? What is it about this learning environment that is so unique? Why have so many prominent leaders come from this institution? Check it out on the web. (N E10) * My project [title withheld to maintain confidentiality] helped me to work out some practices that I can now use to farrow pigs qualified for summer Nieman Ranch market. I hadn’t heard of Nieman at the time. Evaluate that! (MN 10) PRACTICALITY: N=21 F ARM/F ARMER FOCUS (9) " Projects must be funded that move results on to farms as part of the project. (WIO6) " By having the work done on real farms and having farmers be the primary educators. (MI03) " On farm research/demonstration. (M113) * Involve “real farms” in the project from the onset! An interesting interpretation could occur here — is this a recognition that SARE isn’t working with “real farms?” Partner. (IN07) * More producer grants — tie to Chapter I work. (MII9) * Reviewers need to 200 fund only projects that can be applied to farms. To some extent research projects must mirror ideas from producer grants. (WI l4) "' Best way is to d_o the research on farms. “Appropriate” scale would be better than always larger scale. (NEI l) " Including an on-farm component helps greatly in obtaining practical information. (SD01) "' Demonstrate financial durability of participants’ farms. (MNIO) MEDIA (4) " Better public exposure in the farm periodicals. Attendance at trade shows, etc. (N E10) "' Reporters from our various ag news publications should be kept informed of SARE research data. They need to be educated along with everyone else (about) what is being found and demonstrated. I was interviewed by a reporter; he was overwhelmed by the difference to the point that he didn’t know where to begin. This reporter said to me on the phone later that he would like to spend a week or two just observing the whole operation as it was so different from anything he was familiar with. (SD07) * Publish results. Make available free or low cost. (KS 10) " We need a technical writer or committee to work in concert with the communications specialist to develop practical information products from grant results. (N E04) NO PROBLEM (2) " SARE already does well with practicality. (NEI I) ‘ I think the farmer members of the AC and TC do a pretty good job of keeping our projects practical. (MN06) BASIC RESEARCH (I) " I reject this premise (that SARE projects must be practical) as an absolute. A significant share, but not all, of SARE projects should be “practical,” but some share of the projects (ones looking at fundamental system redesign) should not be. (W112) BROADEN AUDIENCE (I) " Just make sure they are shared with a very broad audience. We may never know if an idea or innovation is practical, profitable or productive on a large scale. We must be pleased that we helped the idea work one place, then make results available for large scale adoption if others choose to do so. (MI12) EMPOWERMENT (I) "' In an empowerment model, you don’t try to “move research to farms.” You begin by ascertaining production problems from farmers, and then helping them connect with solutions adopted to their problems. It is at that point that a vision of a more sustainable approach can be entertained. (N D05) NETWORK (l) "' We do need to work on integrating our Chapter 1, Chapter 3 and Producer Grant Programs however. (MN06) MANAGED (I) "' Through a managed grant program rather than a competitive grant program. (WIO7) NO BASIC RESEARCH (l) * Reviewers need to fund only projects that can be applied to farms. To some extent research projects must mirror ideas from producer grants. (WI 14) REDESIGN (l) "‘ A significant share, but not all, of SARE projects should be “practical,” but some share of the projects (ones looking at fundamental system redesign) should not be. (W112) STAY FOCUSED (l) "‘ No, (SARE should not focus on projects that are geared for adoption on a larger scale); The program is designed to fund the needs of individuals that desire alternative agriculture” If benefits to the conventional system occur, so be it. (M103) " As they began -- Land Grant research. SARE supported practitioners apply, share with their neighbors. SARE observers see what’s happening. As composite — each project wouldn’t have to include everything. (SD06) "' Practicality in this case needs to be demonstrated by experiments in which all the practical alternatives are present and compared in unbiased ways. (IL08) MEASURING: N=21 201 GOALS (6) " First you must determine your goals, objectives, and desired system responses. This is part of alternative world views, but not a component of the dominant mechanistic world view!! (Ml03) “ It’s a moving target. There needs to be baseline information and an agreed-upon set of goals. (INO7) "' Back to NCR/national SARE goals. (MI 19) " We need to agree on a set of indicators of sustainability and then start monitoring changes relative to these indicators. (MN06) "' SARE needs its definition, but needs goals more than a definition. The committee each year can revisit this but must set goals for the program. (NEI l) " Define (sustainability). Change is one category. (SD06) DON’T OVER-QUANTIFY (5) "' Sustainability will always be a moving target and so cannot be measured precisely any more than truth or justice can. Give it up! We gap measure ecological, economic and social goals and it is our progress toward these goals we should be delineating and measuring. (N D05) "‘ Bean counters need numbers, but the more precisely you quantify these effects, the more meaningless are the numbers. (IA01) "' Don’t spend too much time; use current definitions and move ahead. (N El 1) "' Start by assuming that every idea has the potential to be a GREAT IDEA. We are usually too quick to want measurements, which leads us to usually be too quick to judge. (M112) "' Measuring a 1-5 year project just reinforces the linear cause-effect logic so destructive of holistic change. (MIOS) SUSTAINABILITY A PROCESS (3) * It’s a moving target. There needs to be baseline information and h an agreed-upon set of goals. (IN07) "' Sustainability will always be a moving target and so cannot be measured precisely any more than truth or justice can. Give it up! We gap measure ecological, economic and social goals and it is our progress toward these goals we should be delineating and measuring. (N D05) * Acknowledge that there is no pp; model of sustainability. We can measure the economic, environmental . and social aspects of various systems and determine which is mgr; sustainable though. (NE10) " PARTICIPANT EVALUATION (2) "' Talk with the farmers involved. (M113) * We can only be sure we have had an impact by the endorsement of those producers who have been directly impacted by a project. (W114) COMMUNITY FOCUS (I) * Need to stress the quality of life/community aspects of sustainable agriculture. (WIO6) NEED SYMBOLISM (I) "' Sustainability must have some sort of broad symbolism incorporated in present day myths and stories. Perhaps you need a famous poet like Wendell Berry to do this. (IA05) ‘ I think this is an important issue. SARE should make it a priority for project solicitation. (W112) "' Evaluate the impacts of projects, then evaluate SARE impact by the cumulative effects of projects. (OHO9) "' Patience. (SD01) "‘ SARE places a serious limitation on researchers through its bias against inputs. Many SARE-sponsored experiments are unrealistic in that they do not include treatments that require purchased inputs. In other words, they do not include many of the alternatives available or potentially available to producers. At least there is bias against experiments that explore all alternatives that don’t fit certain preconceived notions. I know this is an unrealistic recommendation, but if that bias could be removed, the program would be greatly improved and would attract more funds and more researchers. (IL08) * The standard is repeatability without damage. (MNIO) 202 BIBLIOGRAPHY Adler, M. & Ziglio, E. (1996). Gazing into the oracle: The Delphi method and its application to social policy and public health. London: Jessica Kingsley. Allen, P. (1993). (Ed), Food for the future: Conditions and contradictions of sustainability. New York: John Wiley & Sons. Andranovich, G. (1995). Developing community participation and consensus: The Delphi technique. (Community Ventures: Partnerships in Education and Research Circular Series, Western Regional Extension Publication WREP 0131). Pullman, WA: Washington State University. Andrews, M., Ashcraft, N., Thullen, M., & Lock S. (1992). Conceptualization, a "MUST Paper presented at American Evaluation Association annual meeting, Extension Evaluation Education Topical Interest-Group, November 5-7,1992, Seattle, WA. Appropriate Technology Transfer for Rural Areas. (1999). Sustainable agriculture: An introduction. F ayetteville, AR: ATTRA. Ashton, C. (1998). Strategic considerations in facilitative evaluation approaches. Paper prepared for Action Evaluation Research Initiatives online conference, September 1 998. Averch, H. A. (1994). The systematic use of expert judgment. In J. S. Wholey, H. P. Hattie & K. E. Newcomer (Eds.), Handbook of Practical Program Evaluation, pp. 293-309. San Francisco: Jossey-Bass. Barnett, V., Payne, R., & Steiner, R. (Eds). (1995). Agricultural sustainability: Economic, environmental and statistical considerations. New York: John Wiley and Sons. Batie, S. S. & Taylor, D. B. (1991). Assessing the character of agricultural production systems: Issues and implications. American Journal of Alterative Agriculture, 6 (4), 184-187. den Biggelaar, C. & Suvedi, M. (1998). An evaluation of the north-central region SARE producer grant program. East Lansing, MI: Michigan State University, Center for Evaluative Studies. 203 Brandon, P. R. (1999). Involving program stakeholders in reviews of evaluators’ recommendations for program revisions. Evaluation and Program Planning, 22, 363 -3 72. Brooks, A. & Watkins, K. E. (Eds). (1994). New Directions for Adult Education, Number 63: The emerging power of action inquiry technologies. San Francisco: J ossey-Bass. Burke, B. (1998). Evaluating for a change: Reflections on participatory methodology. In E. Whitmore (Ed.), New Directions for Evaluation, no. 80: Understanding and practicing participatory evaluation (pp. 43-56). San Francisco: Jossey Bass. H Campbell, R. M. & Hitchin, D. (1968). The Delphi technique: Implementation in the ' corporate environment. Management Services 5 (N ovember/December), 37-42. Council for Agricultural Science and Technology. (1990). Alternative Agriculture: .! Scientists' Review, Special Publication No. 16. Ames, IA: Council for Agricultural Science and Technology. ' Cousins, J .B. & Whitmore, E. (1998). Framing participatory evaluation. In E. Whitmore (Ed.), New Directions for Evaluation, no. 80: Understanding and practicing participatory evaluation (pp. 5-23). San Francisco: Jossey Bass. Cozzens, S. E. (1997). The knowledge pool: Measurement challenges in evaluating fundamental research programs. Evaluation and Program Planning, 20(1), 77- 89. Crews, T. E., Mohler, C. L. & Power, A. G. (1991). Energetics and ecosystem integrity: Defining principles of sustainable agriculture. American Journal of Alternative Agriculture, 6(3), 146-149. Dalkey, N. & Helmer, O. (1963). Management Science, 9(3), April 1963, 458-467. Delp, P., Thisen, A., Motiwalla, J ., Neelakantan, S. (1977). System tools for project planning. Bloomington, IN: Pasitam. Dick, Bob. Action Research and Evaluation. Paper prepared for Innovations in Education and Program Development On—line Conference, ARIA Associates, September 1998, http://206.21.41 .244/conference/ieconference/bdick.html. Duram, L. A. (1998). Taking a pragmatic behavioral approach to alternative agriculture research. American Journal of Alternative Agriculture, 13, (2) 90-95. 204 Douglass, G. (1994). When is agriculture sustainable? In T.C. Edens, C. Fridgen & S. Battenfield (Eds.), Sustainable agriculture & integrated farming systems (pp. 10-21). East Lansing, MI: Michigan State University Press. Edens, T.C., Fridgen, C. and Battenfield, S. (Eds). (1984). Sustainable agriculture & integrated farming systems. East Lansing, MI: Michigan State University Press. Edwards, C. (1987). The concept of integrated systems in lower input/sustainable agriculture. American Journal of Alternative Agriculture, 2 (2), 148-152. Fetterman, David. Empowerment evaluation: Collaboration, action research, and a case example. http://206.21 .41.244/conference/ieconference/fetterrnan.htrnl. Flora, C. B. (1998-99). Quality of life versus standard of living. Rural Development News, 22, (4), 113. F olkman, D. V. & Rai, K. (1997). Reflections on facilitating a participatory community self-evaluation. Evaluation and Program Planning, 20(4), 455-465. Food, Agriculture, Conservation and Trade Act, 7 U.S.C. 5811 et seq. (1990) Food Security Act, 7 U.S.C. 4701 et seq. (1985) Francis, C. A., Flora, C. B., & King, L. D. (1990). Sustainable agriculture in temperate zones. New York: John Wiley & Sons. GAO. (1992). Sustainable agriculture: Program management, accomplishments, and opportunities. Washington, DC: Government Accounting Office. GAO. (1998). Program measurement and evaluation: Definitions and relationships GAO/GGD-98-26 (966705). Washington, DC: Government Accounting Office. Goldschmidt, P. (1996). A comprehensive study of the ethical, legal and social implications of advances in biochemical and behavioural research and technology. In M. Adler & E. Ziglio, (Eds.), Gazing into the oracle: The Delphi method and its application to social policy and public health (pp. 89-132). London: Jessica Kingsley. Haapala, ~J. Sustainability Index, email. Hakim, S. & Weinblatt, J. (1993). The Delphi process as a tool for decision making. Evaluation and Program Planning, 16, 25-38. 205 Hart, Maureen. (1995). Guide to sustainable community indicators. Montreal, Quebec, Canada: QLFI/Atlantic Center for the Environment. Harwood, RR. (1990). A history of sustainable agriculture. In C.S. Edwards, R. Lal, P. Madden, R. Miller, and G. House, (Eds.), Sustainable Agricultural Systems (pp. 3-19). Ankeny, IA: The Soil and Water Conservation Society. Harwood, RR. (1993). A look back at USA's report and recommendations on organic farming. American Journal of Alternative Agriculture, 8 (4) 150-154. Heath, A. E., Neimeyer, G. J ., & Pedersen, P. B. (1988). The future of cross-cultural counseling: A Delphi poll. Journal of Counseling and Development, 67, 27-30. Helmer, O. (1966). Social Technology. New York: Basic Books. Helmet, 0. & Rescher, N. (1960). On the epistemology of the inexact sciences. Management Science, 6, 25-52. Henclay, Stephen P. and Yates, J. R. (1974). Futurism in education: Methodologies. Berkeley, CA: McCutchan. Henry, G. T., Julnes, G. Mark, M. M. (Eds). (1998). Realist evaluation: An emerging theory in support of practice: New Directions for Evaluation, Number 78. San Francisco: Jossey-Bass. Herdt, R.W. & Steiner, RA. (1995). Agricultural sustainability: Concepts and conundrums. In V.Bamett, R. Payne & R. Steiner (Eds.), Agricultural sustainability: Economic, environmental, and statistical considerations (pp. 3-14). Chichester, England: John Wiley and Sons. Holt, DA. (1992). Testimony submitted to the US. Senate Subcommittee on Agricultural Research and General Legislation. Ikerd, J. (1992). It’s aDilemma. Choices Ikerd, J. (I996a). Sustainable agriculture: Do we really need to define it? Presentation to the Michigan Agriculture Mega-Conference, January 12, 1996, Lansing, MI. Ikerd, J. (1996b). Sustainability: There is no end until we get through, 3/14/96 email posted to SANet-mg. 206 Jackson, W. (1984). A search for a unifying concept for sustainable agriculture. In W. Jackson, W. Berry, & B. Coleman (Eds.), Meeting the expectations of the land (pp. 208-229). San Francisco: North Point Press. J illson, I. A. (1975). The national drug-abuse policy Delphi: Progress report and findings to date. In H. A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications (pp. 124-158). Reading, MA: Addison-Wesley. Johnson, DE, Meiller, L.R., Miller, L.C., & Summers, G.F. (Eds). (1987). Needs assessment: Theory and methods. Ames, IA: Iowa State University Press. Kibel, BM. (1999). Success stories as hard data: An introduction to results mapping. New York: Kluwer Academic/Plenum. King, J .A. (1998). Making sense of participatory evaluation. In E. Whitmore (Ed.), New Directions for Evaluation, no. 80. Understanding and practicing participatory evaluation (pp. 57-67). San Francisco: Jossey Bass. Kriegl, Tom. (1999). Wisconsin dairy grazing profitability analysis: Preliminary third year summary. Paper presented at Life Cycle Approach to Sustainable Agriculture Indicators Workshop, Ann Arbor, MI. Lacy, W.B. (1993). Can agricultural colleges meet the needs of sustainable agriculture? American Journal of Alternative Agriculture, 8 (1), 40-45. Levins, D. (1996). Monitoring sustainable agriculture with conventional financial data. White Bear Lake, MN: Land Stewardship Project. Linstone, H. A. (1975). Eight basic pitfalls: A checklist. In H.A. Linstone & M. Turoff, (Eds.), The Delphi method: techniques and applications (pp. 571-586). Reading, MA: Addison-Wesley. Linstone, H. A. & Turoff, M. (Eds). (1975a). The Delphi method: Techniques and applications. Reading, MA: Addison-Wesley. Linstone, H. A. & Turoff, M. (Eds) (1975b). Introduction (to general applications). In H.A. Linstone & M. Turoff, (Eds.), The Delphi method: techniques and applications (pp. 75-84). Reading, MA: Addison-Wesley. Linstone, H. A. and Turoff, M. (19750). Introduction (to Evaluation). In H.A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications (pp. 229- 23 5). Reading, MA: Addison-Wesley. 207 Linstone, H. & Turoff, M. (1975d). Introduction (to specialized techniques). In H.A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications (pp. 385-3 86). Reading, MA: Addison-Wesley. Lipsey, M. W. & Pollard, J. A. (1989). Driving toward theory in program evaluation: More models to choose from. Evaluation and Program Planning, 12, 317-328. Lockeretz, W. (1989). Comparative local economic benefits of conventional and alternative cropping systems. American Journal of Alterative Agriculture, 4 (2), 75-83. Lowrance, R., Hendrix, RR, and Odum, Eugene P. (1986). A hierarchical approach to sustainable agriculture. American Journal of Alterative Agriculture, 1 (4), 169- 173. Ludlow, J. (1975). Delphi inquiries and knowledge utilization. In H.A. Linstone & M. Turoff (Eds). The Delphi method: Techniques and applications (pp. 102-123). Reading, MA: Addison-Wesley. Madden, J .P. (1991). Sustainable agriculture research and education: Enabling legislation and legislative history. Madden, J .P. (1998). The early years of the LISA, SARE and ACE programs: Reflections by the founding director. PO. Box 4186, Glendale, CA 91222. Madden, J .P. & Chaplowe, S.G. (1997). For all generations: Making world agriculture more sustainable. Glendale, CA: OM. Marshall, M. G. & Bennett, C. F. (1998). National extension targeted water quality program, 1992-1995, Volume I: Overview, reporting system, and summary of outputs and outcomes. College Station, TX: Texas A&M University. Martino, J. P. (1983). Technological forecasting for decision making, Second Edition. New York: North-Holland. Mathie, A. & Greene, J. C. (1997). Stakeholder participation in evaluation. Evaluation and Program Planning, 20 (3), 279-285. Mercier, C. (1997). Participation in stakeholder-based evaluation: a case study. Evaluation and Program Planning, 20(4), 467-475. Merriam, S. B. & Simpson, E. L. (1989). A guide to research for educators and trainers of adults (updated edition). Malabar, FL: Robert E. Krieger. 208 McNamee, P. B. (1985). Tools and techniques for strategic management. Oxford, UK: Pergamon Press. Michalski, G. V. (Date) Stakeholder variation in perceptions about training program results and evaluation: A concept mapping investigation (draft version). Http://www.conceptsystems,com/paperusr/michalsk/aeaS I .htm. Miller, L. C. & Hustedde, R. J. (1987). Group approaches. In DE. Johnson, L.R. Meiller, L.C. Miller, & G.F. Summers (Eds.), Needs assessment: Theory and methods (pp. 91-125). Ames, IA: Iowa State University Press. Mitrof’f, I. A. & Turoff, M. (1975). Philosophical and methodological foundations of Delphi. In H.A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications (pp. 17-35). Reading, MA: Addison-Wesley. Moore, C. M. (1987). Applied social research methods series, volume 9: Group techniques for idea building. Newbury Park: Sage Publications. Moran, J. (1997). Assessing adult learning: A guide for practitioners. Malabar, FL: Krieger. National Research Council. (1989). Alternative agriculture. Washington, DC: National Academy Press. National Research Council. (1991). Sustainable agriculture research and education in the field: A proceedings. Washington, DC: National Academy Press. NCR SARE. (n. d.) Administrative council and technical committee handbook. Lincoln, NE: North Central Region Sustainable Agriculture Research and Education Program. Neave, P., Kirkwood, V., & Dumanski, J. (1995). Review and assessment of available indicators for evaluating sustainable land management: Technical Bulletin 1995- 7E. Ottawa, Ontario, Canada: Centre for National Biological Resources Research, Agriculture and AgriFood Canada. (http://resagr.ca/ecorc/program3/pub/indicat/) Nevo, D. (1989). Expert opinion in program evaluation. In R.F. Conner and M. Hendricks (Eds.), International innovations in evaluation methodology (pp. 85- 93). San Francisco: Jossey-Bass. Nielsen, N. 0. (Ed.). (1994). Agroecosystem health. Guelph, Ontario, Canada: University of Guelph. 209 Niero, M. (1996). Delphi estimates on clients' perceptions of family planning services. In M. Adler & E. Ziglio, (Eds.), Gazing into the oracle: The Delphi method and its application to social policy and public health (pp. 213-239). London: Jessica Kingsley. Padgitt, S. & Petrzelka, P. (1994). Making sustainable agriculture the new conventional agriculture: Social change and sustainability. In J. L. Haffeld & D. L. Karlen (Eds.), Sustainable agriculture systems (pp. 261-286). Boca Raton, FL: Lewis. Papendick, R. (1993). Revisiting the USDA reports: Some additional observations. American Journal of Alterative Agriculture, 8 (4), 154-155. Parr, J. F., Papendick, R.I., Homick, S. B. & Meyer, RE. (1992). Soil quality: attributes and relationship to alternative and sustainable agriculture. American Journal of Alterative Agriculture, 7(1-2), 5-11. Patton, MO. (1997). Utilization-Focused Evaluation: The new century text, 3'“ edition. Thousand Oaks, CA: Sage Publications. Richardson, R.H. 10/10/94 SANet message. Rieper, O. & Toulemonde, J. (Eds) (1997). Politics and practices of intergovernmental evaluation. New Brunswick, NJ: Transaction. Roberts, W. S. and Swinton, S. M. (1996). Economic methods for comparing alternative crop production systems: A review of the literature. American Journal of Alterative Agriculture, 11 (1), 10-17. Robertson, GP. and Harwood, RR. (2000). Sustainable Agriculture. In press. Rocheleau, B. (1986). Public perception of program effectiveness and worth: a review. Evaluation and Program Planning, 9, 31-37. Roling, N.G. & Wagemakers, M.A.E. (Eds). (1998). Facilitating sustainable agriculture: Participatory learning and adaptive management. In times of environmental uncertainty, Cambridge, UK: Cambridge University Press. Rotondi, A. & Gustafson, D. (1996) Theoretical, methodological and practical issues arising out of the Delphi method. In M. Adler & E. Ziglio (Eds). Gazing into the oracle: The Delphi method and its application to social policy and public health (pp. 34-55). London: Jessica Kingsley. 210 Salamon, S., Farnsworth, R., Bullock, D. & Yusuf, R. (1997). Family factors affecting adoption of sustainable farming systems. Journal of Soil and Water Conservation 52 (2), 265-271. Sanders, J. R. (1992). Criteria for technically valid impact studies. Paper presented at Evaluation Network Annual Conference, Baltimore, MD, October 28, 1992. Scheele, D. S. (1975). Reality construction as a product of Delphi interaction. In H.A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications (pp. 37-72). Reading, MA: Addison-Wesley. Scheibe, M., Skutsch, M., and Schofer, H. (1975). Experiments in Delphi methodology. In H. A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications (pp. 262-287). Reading, MA: Addison-Wesley. Stake, R. E. (1972). The countenance of educational evaluation. In C. H. Weiss (Ed.), Evaluating action programs (pp. 31-51). Boston: Allyn and Bacon. Stockle, C.O, Papendick, R.I., Saxton, K.E., Campbell, G.S. & van Evert, F .K. (1994). A fi'amework for evaluating the sustainability of agricultural production systems. American Journal of Alterative Agriculture, 9 (1-2), 45-50. Strange, M. (1984). The economic structure of a sustainable agriculture. In W. Jackson, W. Berry, & B. Coleman (Eds.), Meeting the expectations of the land (pp. 115- 125). San Francisco: North Point Press. Sutherland, J. W. (1975). Architecting the future: A Delphi-based paradigm for normative system-building. In H.A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications (pp. 463-486). Reading, MA: Addison- Wesley. Taylor-Powell, E., Rossing, B., & Geran, J. (1998). Evaluating collaboratives: Reaching the potential. Madison, WI: University of Wisconsin. Thompson, PB. (1995). The Spirit of the Soil: Agriculture and Environmental Ethics. London: Routledge. Thompson, PB. (1999). Sustainability: What it is and what it is not. University of Illinois Agra-ecology: News and perspectives newsletter, 7(1), 1-8. Turnbull, B. (1999). The mediating effect of participation efficacy on evaluation use. Evaluation and program planning, 22, 131-140. 211 Turoff, M. & Hiltz, S. R. (1996). Computer-based Delphi processes. In M. Adler & E. Ziglio (Eds.), Gazing into the oracle: The Delphi method and its application to social policy and public health (pp. 56-85). London: Jessica Kingsley. Turoff, M. (1975). The policy Delphi. In H.A. Linstone & M. Turoff, (Eds.), The Delphi method: Techniques and applications (pp. 84-100). Reading, MA: Addison-Wesley. United States Congress. (1985). The Food Security Act of 1985. Public Law 99-198. Washington, DC. United States Congress. (1990). The Food, Agriculture, Conservation and Trade Act. Public Law 101-624. Washington, DC. United States Congress. (1991). The Food, Agriculture, Conservation and Trade Act Amendments. Public Law 102-237. Washington, DC. United States Department of Agriculture. (1980). Report and recommendations on organic farming. Washington, DC. Vickery, J. & Lohr, L. (1997). Sustainability assessment in agriculture: Annotated bibliography and resource list of methods. Faculty Series 97-05, Department of Agricultural and Applied Economics Athens, GA: University of Georgia. Walter, G. & Reisner, A. (1994). Midwestern land-grant university scientists' definitions of sustainable agriculture: A Delphi study. American Journal of Alterative Agriculture, 9 (3), 109-120. Waltner-Toews, (1994). Ecosystem health: A framework for implementing. In N.O. Nielsen(Ed.), Agroecosystem health: Proceedings of an international workshop (pp. 8-23). Guelph, Ontario, Canada: University of Guelph. Weatherman, R. & Swenson, K. (1974). Delphi technique. In S.P. Henclay & J .R. Yates (Eds.), Futurism in education: Methodologies (pp. 97-114). Berkeley, CA: McCutchan. Weaver, W. T. (1971). The Delphi forecasting method. Phi Delta Kappan, 52, 267-271. Weiss, C. (1972a). The politicization of evaluation research. In C. H. Weiss (Ed.), Evaluating action programs (pp. 327-338). Boston: Allyn and Bacon. Weiss, C. (1972b). Utilization of evaluation: toward comparative study. In C. H. Weiss (Ed.), Evaluating action programs, pp. 318-326. Boston: Allyn and Bacon. 212 Weiss, R. S. & Rein, M. (1972). The evaluation of broad-aim programs: Difficulties in experimental design and an alternative. In C. H. Weiss (Ed.), Evaluating action programs (pp.236-249). Boston: Allyn and Bacon. Welsh, R. & Ingram, K. (1988). Agricultural systems research handbook Southern Sustainable Agriculture Research and Extension project. White, D.C., Braden, J .B., & Higley, LG. (1994). Economics of sustainable agriculture. In J .L. Hatfield & D.L. Karlen (Eds.), Sustainable agriculture systems (pp. 229- 260). Boca Raton, FL: Lewis. Whitmore, E. (Ed.). (1998). New Directions for Evaluation, no. 80. Understanding and practicing participatory evaluation. San Francisco: J ossey Bass. Williams, H.S., Webb, A.Y., & Phillips, W.J. (1991). Outcome funding: A new approach to targeted grantmaking. Rensselaerville, NY: Rensselaerville Institute. Young, N., Gardner, S. , Coley, S., Schorr, L., & Brunet, C. (1994). Making a difierence: Moving to outcome-based accountability for comprehensive service reforms. Resource Brief 7. Falls Church, VA: National Center for Service Integration. Youngberg, G., Schaller, N. & Merrigan, K. (1993). The sustainable agriculture policy agenda in the United States: Politics and prospects. In P. Allen (Ed.), Food for the future: Conditions and contradictions of sustainability. New York: John Wiley & Sons. Ziglio, E. (1996). The Delphi Method and its contribution to decision-making. In M. Adler & E. Ziglio (Eds.), Gazing into the oracle: The Delphi method and its application to social policy and public health (pp. 3-33). London: Jessica Kingsley. 213 nrcuran STATE UNIV. LIBRARIES lllMilllllllllllWilliWilllllllllllllllllllllll 31293020485472