LL: h. . ._| r a -.— "W ~a‘fiafiwéw a .4.‘ q: E I l. 31 h h I“ i! a” 0'3?“ . .52" $333: .A ”m -m J.» an"! 4* “7 awn-M MICHIGADTSB S TATE UNIVERSITY EAST LANSING, MICH 48824-1048 ——__‘________ “—1— This is to certify that the dissertation entitled AN EVALUATION OF LEISURE AGRICULTURE POLICY IN TAIWAN UTILIZING THE ANALYTIC HIERARCHY PROCESS (AHP) presented by HUNG-HSU YEN has been accepted towards fulfillment of the requirements for the PhD. degree in PARK, RECREATION, AND TOURISM RESOURCES fozfl/g ' Major T’rofessor’s Signature 2,46. 532005 Date MSU is an Affirmative Action/Equal Opportunity Institution .-.—-.-.-o---—.--‘ -. - PLACE IN RETURN Box to remove this checkout from your record. TO AVOID FINES return on or before date due. MAY BE RECALLED with earlier due date if requested. DATE DUE DATE DUE DATE DUE Ap§2224420679 2/05 CLICIRC/DataDueJndd-pJS _fi_ ,__fi__ AN EVALUATION OF LEISURE AGRICULTURE POLICY IN TAIWAN UTILIZING THE ANALYTIC HIERARCHY PROCESS (AHP) By Hung-Hsu Yen A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Park, Recreation, and Tourism Resources 2005 ABSTRACT AN EVALUATION OF LEISURE AGRICULTURE POLICY IN TAIWAN UTILIZING THE ANALYTIC HIERARCHY PROCESS (AHP) By Hung-Hsu Yen The purpose of this study is to identify and measure the effectiveness of the Taiwanese Council of Agriculture’s overall success in promoting “leisure agriculture” development. An l8-member expert panel, consisting of farm owners, scholars, and policy enforcers, was interviewed to identify the potential indicators for the performance evaluation. A panel of three researchers then reviewed these indicators and developed the evaluation framework. Thirty-three performance indicators for the performance evaluation were embedded within three dimensions: economy, enjoyment, and ecology. Using a mailed survey, 509 stakeholders (including farm owners, scholars, and policy enforcers) were asked how satisfied they were with each of these 33 performance indicators. Using confirmatory factor analysis, data were analyzed to confirm the content validity of these three dimensions as well as an evaluation framework developed for this study. After developing the evaluation framework, the Analytic Hierarchy Process (AHP) was utilized to assign weights to selected evaluation indicators using the judgments of the lS-member expert panel. The AHP provided more useful quantitative information about group preferences, satisfaction levels, and an overall performance score then the importance-performance analysis did. The results of this research show that the stakeholders deem these three dimensions of the evaluation framework as equally important. This suggests that future development should focus evenly on the economy, enjoyment, and ecology. On a scale of 0 (low) to 10 (high) these stakeholders gave an overall policy evaluation score of 5.9. The scholars assigned a slightly higher average rating (6.1) to the policy than did the farmers (5.8) and the policy enforcers (5.9). Thus, the policy was judged to be only marginally successful by all groups of stakeholders. From the micro view, the ratings of most economic indicators were below the average, indicating the economic performance needs to be enhanced. TABLE OF CONTENTS LIST OF TABLES ............................................................................................................. vi LIST OF FIGURES .......................................................................................................... vii CHAPTER 1 INTRODUCTION ........................................................................................ 1 Problem Context .................................................................................................. 1 How Can Leisure Agriculture Help? .............................................................. 5 How Leisure Agriculture Can Help Agriculture Development in Taiwan ..... 6 Research Purpose ................................................................................................ 8 Problem Statement .............................................................................................. 9 CHAPTER 2 LITERATURE REVIEW ........................................................................... 14 Leisure Agriculture Policy in Taiwan ............................................................... 14 The Policy of Leisure Agriculture Development in Taiwan ......................... 15 The Goals of Leisure Agriculture Development in Taiwan .......................... 19 Performance Evaluation .................................................................................... 20 Defining Evaluation ...................................................................................... 22 The Performance Evaluation Process ........................................................... 24 Evaluation Methods ...................................................................................... 25 The Analytic Hierarchy Process (AHP) ............................................................ 33 What Is the Analytic Hierarchy Process? ..................................................... 33 How Does the AHP Work? ........................................................................... 34 Group Preference Aggregation with the AHP .............................................. 39 Applications of the AHP ............................................................................... 40 CHAPTER 3 METHODOLOGY AND PROCEDURES ................................................. 41 Research Design ................................................................................................ 41 Identify Criteria for Evaluation and Develop a Hierarchical Evaluation Model ......................................................................................................... 44 Sample ........................................................................................................... 45 Data Collection Procedures ........................................................................... 46 Data Analysis Procedures ............................................................................. 47 Limitations Associated with Identifying Criteria for Evaluation and Developing a Hierarchical Evaluation Model ........................................... 48 Assess Performance across Criteria .................................................................. 49 Sample ........................................................................................................... 50 Data Collection Procedures ........................................................................... 50 Limitations Associated with Assessing Performance across Criteria ........... 50 Determine Weights of Criteria Using AHP ....................................................... 53 Data Collection Procedures ........................................................................... 54 Limitations Associated with Determining Weights of Criteria Using AHP. 55 Validate Criteria (Refine Evaluation Model) .................................................... 56 iv Sample ........................................................................................................... 57 Limitations Associated with Validating Criteria (Refining Model) ............. 57 Calculate Performance Score ............................................................................ 58 CHAPTER 4 RESULTS AND DISCUSSION ................................................................. 59 Criteria for Evaluation and the Evaluation Model ............................................ 59 Results of Assessing Performance across Criteria ............................................ 67 Results of Determining Weights of Criteria using AHP ................................... 73 Criteria Validation (Model Refinement) ........................................................... 80 Performance Score Results ................................................................................ 96 CHAPTER 5 CONCLUSIONS ........................................................................................ 98 Summary of Study Results ................................................................................ 98 Limitations of the Study .................................................................................. 10] Conclusions and Implications ......................................................................... 102 Recommendations for Future Research .......................................................... 104 APPENDIX I: Consent Form I ....................................................................................... 108 APPENDIX II : Consent F orm II and Performance Assessment Questionnaire ............. 110 APPENDIX III: Consent Form 111 and AHP Questionnaire ........................................... 115 APPENDIX VI: List of Potential Indicators ................................................................... 139 BIBLIOGRAPHY ........................................................................................................... 149 LIST OF TABLES Table 1. Sources of national income in Taiwan by year. ..................................................... 2 Table 2. Taiwanese farmers’ incomes and source of income by year. ................................ 2 Table 3. Selected data from the Taiwan farming census for selected years. ....................... 4 Table 4. Trends in employment by sector in Taiwan ........................................................... 4 Table 5. Saaty’s scale of measurement for pair-wise comparisons. .................................. 37 Table 6. List of experts interviewed. ................................................................................. 60 Table 7. The response rate to the performance assessment survey by stakeholder groups. ............................................................................................................... 67 Table 8. Mean performance evaluation ratings by stakeholder group for 33 program performance indicators. ..................................................................................... 69 Table 9. Relative weights (local priorities) of evaluation elements by stakeholder group. ................................................................................................................. 75 Table 10. Relative weights (global priorities) of evaluation elements by stakeholder group. ................................................................................................................. 78 Table 11. Table of variance inflation factor (VIF*) values. .............................................. 82 Table 12. Examination of data for normality. .................................................................... 83 Table 13. Recalculated relative weights (local priorities) of evaluation elements by stakeholder group. ............................................................................................. 89 Table 14. Recalculated weights (global priorities) of the evaluation elements by stakeholder group. ............................................................................................. 92 Table 15. Results of recalculated weights’ mean comparisons within the three stakeholder groups. ............................................................................................ 94 Table 16. Accumulative weighted performance score by stakeholder group. ................... 97 vi LIST OF FIGURES Figure 1. Policy-analytic procedures associated with different phases of policy- making. .............................................................................................................. 21 Figure 2. The program delivery cycle. ............................................................................... 21 Figure 3. An example of importance-performance analysis graphic. ................................ 32 Figure 4. An example of the basic structure of a hierarchy framework design. ................ 35 Figure 5. The flow chart of this research. .......................................................................... 43 Figure 6. The hierarchical evaluation model. .................................................................... 66 Figure 7. The modified evaluation model and the standardized solution. ......................... 86 vii CHAPTER 1 INTRODUCTION Problem Context In developing and developed countries, manufacturing and service industries have gradually taken over the leading role from agriculture in most national economies. Taiwan is no exception; its economic development has also followed this global trend. Although agriculture no longer plays as significant a role as the manufacturing and service industries in economic development, no country can ignore its value. Today, the importance of agriculture is multifunctional. Agriculture provides services and outputs beyond food, fiber, and forestry. These outputs include goods desired by society such as open space, wildlife habitat, biodiversity, flood prevention, pleasing rural landscapes, cultural heritage, viable rural communities, and food security (Bohman et al., 1999; European Commission, 2001; Finland, 1997; Maier, Shobayashi, & Organisation for Economic Co-operation and Development, 2001; Prem & Michel, 1999; Sub, 2001) From the data shown in Table 1, it can be seen that Taiwan’s economy has evolved from agriculture to manufacturing and service industries. However, there are still about 3.5 million people (15% of population) who participate in agricultural related businesses, although they contribute less than 1% to the national income. This evidence reveals that agriculture is no longer a good way to make a living for most Taiwanese. Taiwan’s entry into the World Trade Organization (WTO) makes the farmers’ future even less promising than it otherwise would have been. Agriculture’s future is also threatened by a lack of interest among Taiwanese youth in pursuing it as a career. The older farmers that dominate the industry lack the energy and ability to adopt new technologies needed to remain competitive in modern agricultural enterprises. Table 1. Sources of national income in Taiwan by year. Year Amount (Unit: Million Taiwan Dollars) % of Total Income Total Agriculture Manufacturing Service Agiculture Manufacturing Service 1952 1,940 397 554 989 20.46 28.56 50.98 1960 10,361 1,559 3,813 4,989 15.05 36.8 48.15 1970 49,054 3,308 23,441 22,305 6.74 47.79 45.47 1980 456,446 13,732 212,846 229,868 3.01 46.63 5036 1990 965,580 24,072 356,986 584,522 2.49 36.97 60.54 2000 2,267,328 15,906 1,136,969 1L1 14,453 0.7 50.15 49.15 (Source: Directorate General of Budget Accounting and Statistics Executive Yuan, R.O.C.) Other evidence of the challenge facing Taiwan’s agricultural industry is evident in farmers’ incomes. The main problem is that the majority of farmers’ incomes no longer come from agricultural production. As can be seen in Table 2, 65.59% of farmers’ incomes came from agricultural production in 1966. However, the percentage decreased from 65.59% to 17.56% by 2000. This indicates that most farmers cannot make a comfortable living in agriculture enterprises alone. Table 2. Taiwanese farmers’ incomes and source of income by year. Average income per household Average income per person F anners’ income sources (Unit: Taiwan dollar) (Unit: Taiwan dollar) (Unit: Taiwan dollar) Year Farmer Non-Farmer Farmer Non-Famer From Ag. From other % from Ag. 1966 32,320 34,080 4,508 6,467 21,314 1 1, 65.95 1971 40,858 51,629 6,191 9,579 18,480 22,378 45.23 1976 106,25 134,662 17,448 27,204 41,377 64,880 38.94 1980 219,412 275,45 1 38,903 59,75 1 54,436 164,976 24.81 1985 3 10,58 390,641 59,613 87,982 76,88 233,696 24.76 1990 503,830 651,845 108,118 158,987 101,265 402,563 20.10 1995 871,082 1,052,834 198,424 272,050 172,083 698,999 19.76 2000 917,623 1,166,870 226,01 327,772 161,121 756,50 17.56 (Source: Directorate General of Budget Accounting and Statistics Executive Yuan, R.O.C.) Data from the Taiwan farming census survey that are presented in Table 3 and Table 4 show the decline of farming in Taiwan. For example: 0 The “percentage of total households” was 45.7% in 1955, but it dropped to 13.6% in 1995. 0 The farming population was 5.2 million people in 1955 and decreased to 3.9 million in 1995. As a percentage of the total population, farmers dropped from 57.6% to 18.4% over this period of time. 0 The average number of persons per farming household engaged in farming was 7.03 in 1955 but decreased to 4.96 in 1995. 0 The land in agriculture was 881,610 hectares in 1955 but decreased to 709,723 hectares in 1995. 0 The agricultural employment population was 1.67 million in 1955 but decreased to 740 thousand in 1995. In 2001, only 7.5% of the total population was engaged in agriculture. The above evidence shows agriculture is no longer a major industry in Taiwan. Many farmers have shifted their careers to other types of business or simply have retired. A.0.0..m .52; o>u=ooxm mozmtflm v5 weucsooo< 835m mo .3250 888025 ”350$ snow 3% seem RR :3 8s 33 58 £93 £2. SR 33 $3 o: ate 88 :58 $3 :53 33 see «a new 32 Res. 5% 3.9. San _..\&.~_ $3 SN.» 82 Fe: 3.3 was; $3 raw: 82 mg 32 seem 33” can? $3 {One RS $3 82 $5.4m :3 {3.3 we; :33 $3 and E: $4.3 :3 saga ”a; .53 53 23. OS. an: 3: and «.3 some $5 85. $2 30% as; $3 3 «a? 9.5 SA 82 seem an {bi 8m seam $2 $3 32 8E2 massage“: 23305.6. p a5 §> 3:885 é 5&5 finish. 5 .833 .3 25:32an E 35:. .v «Bah. A.O.O.m .:a=> oZSooxm mosmnflm v5 watczooo< 8935 Mo 35:00 888025 ”ooSomv med? REE $.80 .56? _8.§ 83% 333 :39 23: 988852332? 5 2.3 e? 8e Se 8e 3e ”3 55 8s 8s 2238; as 80830 omega. 3: S. a 2.x Rom 5?. e3. Sam Rem an? 8:238 39% .x. ”8.8% 25.33. 3.33. 33”? £983 95.2% canted 538% £38.“ @836 8:238 mafia $2 3.2 3.: 82 3.2 8% 8.2 3:. 8.3 £2.88; 32.6 .x. 839 Sam» and: 23% «8.3” 8%; Sofia 838 ”News £288; wage co 32 82 £2 82 £2 £2 82 82 $2 .930» 188—3. .8“ 28:3 wage—am 53:2. 2: 89c 52. cote—om .m mink. How Can Leisure Agriculture Help? Definition of leisure agriculture There are several terms for this new farming enterprise. The terms leisure agriculture, agricultural tourism, agritourism, and farm tourism are used interchangeably, although regional preferences are evident in the literature. In Taiwan, the term leisure agriculture is used; in England, farm tourism is used; in the United States, agricultural tourism or agritourism are used. For consistency, the term leisure agriculture is used throughout this report. Leisure agriculture is defined in the Leisure Agriculture Management Guidance Statute (1999) in Taiwan as: “a farming business which uses the resources of farming landscape, ecology, and natural environment combined with agricultural production, farming culture, farming activities, and farming living to provide leisure opportunities and farming experiences for the public.” Hilchey (1993) defines agritourism as a business model which “promotes the products of the farm, and thereby generates additional hospitality business, such as, farm tours, farm bed and breakfasts, wineries, petting zoos, fee hunting, fee fishing, farm vacations, horseback riding, hay rides, farm-based cross-country skiing, and camping.” Lobo (2002) defines agricultural tourism as: “the act of visiting a working farm or any agricultural, horticultural or agribusiness operation for the purpose of enjoyment, education, or active involvement in the activities of the farm or operation.” Gustafson (2002) defines agricultural tourism as: “a set of economic and social activities that occur and link travel with the products, services, and experiences of agriculture.” How Leisure Agriculture Can Help Agriculture Development in Taiwan Agriculture plays a less significant role in the economic development of Taiwan now than it did before. Because of the free market pressures associated with entry into the WTO, the government published the Agriculture Policy White Book in 1995 in which future agricultural policies are outlined. It states that future agricultural development should focus on: (1) improvements in agricultural production, (2) improvements in the quality of life, and (3) protection of the environment. Leisure agriculture is a way to reach these agricultural policy goals and help some farmers to maintain their farming enterprises. Leisure agriculture can benefit farmers by providing them with a large and relatively untapped market for their products. This is especially important in view of economic challenges facing small-scale farming operations. Improving the quality of life can benefit both farmers and travelers. Farmers can earn extra revenue from selling their products and services to travelers. Buying directly from the farmer enhances the traveler’s experience by providing local particularity. The interaction between travelers and farmers can foster better understanding between urban and rural populations. The rural area can offer the urban population a more relaxed and uncrowded environment. The rural population can also get some fashion and hi-tech information from the urban population. To protect the environment, the development of leisure agriculture can help to reduce the loss of farmland. Rising land values put pressure on farmers to sell their farms for development. Leisure agriculture provides another source of income for farmers and thus may help preserve farmland (Normile & Bohman, 2002). In 1995, the Agriculture Policy White Book clearly pointed out that promoting leisure agriculture development was an important goal of future agricultural development in Taiwan. Moreover, the policy document (Measures and Strategies in Response to the World Trade Organization Impacts on Taiwan’s Agriculture) and the key speech from the Minister of Council of Agriculture in 2001 both emphasized that promoting leisure agriculture development was an important government goal. Research Purpose The U. S. Performance and Results Act, enacted in 1993, focuses agency oversight attention on the performance and results of government activities by requiring that all federal agencies measure and report on the results of their activities annually. The need for systematic performance measurement in governmental organizations is well documented in the literature (Brown & Pyers, 1988; Wholey & Harty, 1992). This indicates that the United State government has attached considerable importance on the evaluation of performance. Moreover, performance measurement has even become something of an international movement. The focus of this research was to evaluate the Council of Agriculture’s policies to promote leisure agriculture develOpment in Taiwan. Specifically, this research addresses the following research questions: 1. What are the goals of leisure agriculture development in Taiwan? 2. What are the relative priorities of these goals? 3. How effective do stakeholders think the Council of Agriculture has been in developing leisure agriculture? Problem Statement “For public policy—making to satisfy the imperatives of a democratic political system, its decision-makers and their performance must be subject to external review. It is not sufficient that policies only are developed and implemented; those entrusted with these tasks must be held accountable, and in a democracy the threads of accountability lead to the general citizenry and their opinions of the merit or demerit of policy- makers’ performance. . . Democracy requires multiple points of external review to assure that those who hold office and make policy are held responsible for their actions” (Koenig, 1986, p. 183). Public policy makers and managers face four leadership challenges including: 1) setting organizational goals; 2) ensuring that priorities among goals are clearly understood and agreed on; 3) providing continuous feedback on organizational performance in terms of those goals; and 4) stimulating improved organizational performance (Wholey & Newcomer, 1989). Therefore, an efficient and effective government should always check its policies and programs against the above four main leadership effectiveness criteria. Promoting the development of leisure agriculture is one of the Taiwan govemment’s agricultural policies. Since 1980, the Council of Agriculture has undertaken several projects and subsidized local governments and farmers’ associations to promote leisure agriculture in order to stimulate development. During the past two decades, the government has encountered several problems and barriers to development. Modifications and improvements have been made to meet development needs. These will be discussed in detail in the literature review. However, no integrated evaluation of the performance of this policy has been conducted to date. Barriers to such a comprehensive analysis include: 1. Minimization of managerial accountability during periods of rapid change- Evaluation always carries the risk that findings might reflect negatively or reveal unwanted outcomes. Such fears may be warranted in some circumstances. Negative results may lower the public’s image of the policies and/or government officials and policy- makers. 2. Lack of confidence that the findings will yield practical benefits exceeding their cost- Govemment officials argue that even valid information is difficult to use in affecting desired improvements during periods of sharply limited resources and significant changes in philosophy. They may not believe the evaluations can solve current problems. 3. Length of time required to produce results- Information needs are great, but time limitations compound the difficulty in meeting these needs. Three problems may 10 arise while the assessment is in process: 1) the original questions may change, 2) other issues may assume a higher priority, and 3) policies themselves may change. 4. Unclear objectives for leisure agriculture development and a lack of objective criteria for evaluation- Leisure agriculture development policy has multiple objectives. The development of leisure agriculture can help to promote the sale of agricultural products, increase the income of farmers, improve development in rural communities, protect the environment, conserve the rural landscape, prevent the loss of agricultural land, and satisfy people’s need for recreation opportunities. The multiple objects of leisure agriculture have never been clearly identified; moreover, the priorities of the objectives are also unclear. Not knowing the objectives or their relative priorities makes it hard to evaluate the outcomes of leisure agriculture policy and to make effective recommendations for future development. Therefore, a performance evaluation of leisure agriculture policy to include establishing its objectives and their priorities is necessary to improve government leisure agriculture policies. Objective adjustments to leisure agriculture development policy can only be made through deriving relevant objectives and their priorities and then conducting a performance evaluation of them based upon inputs from stakeholders. 11 The need for systematic performance measurement in governmental organizations has even become something of a movement. Even though there are some barriers in conducting the evaluation, evaluating the govemment’s overall success in promoting leisure agriculture development and the effectiveness of specific elements of its leisure agriculture development policies are still necessary. Traditionally importance- performance analysis has been used to evaluate policy as an approach. However, this approach has one drawback: the weights assigned to various elements affecting performance are not clear. There is a need to evaluate the policy with a holistic approach that make the relative weights of various elements visible. The purpose of this chapter was to explain why the Council of Agriculture’s performance (effectiveness) of the leisure agriculture development needed to be evaluated. In chapter two, more background information is presented, including a detailed discussion regarding the policies to be evaluated and the policy evaluation literature deemed most relevant for this research problem. The following categories of literature are discussed in chapter two: 1) leisure agriculture policy in Taiwan, 2) policy evaluation, and 3) approaches to dealing with multi-criteria analysis-with a focus on the Analytic Hierarchy Process (AHP). Chapter three provides a detailed discussion of the 12 research design used in this study, which included identifying criteria for evaluation and developing a hierarchical evaluation model, assessing performance across criteria, determining weights of criteria using AHP, refining the model, and calculating the performance score for leisure agriculture policy in Taiwan. In chapter four, the application of the overall evaluation strategy is discussed and results are reported and analyzed. The last chapter contains a summary of research results, suggestions for modifications to leisure agriculture policies in Taiwan and recommendations for future research. 13 CHAPTER 2 LITERATURE REVIEW This review of literature is divided into three main parts. In the first section, an overview of leisure agriculture development in Taiwan is presented. It will include historic goals of leisure agriculture development in Taiwan. The second section contains an overview of the policy evaluation literature. In it, definitions of evaluation are discussed; the evaluation process is outlined, and alternative evaluation approaches are reviewed. In the last section, the analytic hierarchy process (AHP) for public policy evaluation is introduced and discussed in detail. Leisure Agriculture Policy in Taiwan Agriculture plays a less significant role in the economic development of Taiwan now than it did before. The development of leisure agriculture has the potential to benefit some farmers by providing them with a large and relatively untapped market for their products and helps them to shifi from agriculture to a service-type of career. Leisure agriculture in Taiwan is not new; rather this industry has evolved over the last twenty years. 14 The Policy of Leisure Agriculture Development in Taiwan There have been four major stages in the development of leisure agriculture in Taiwan. Each is discussed briefly below. Before 1989 Discussion of farm-based recreation began in 1980 in Taipei. The farmers’ association in Taipei first combined farm-based recreational activities with agriculture which allows tourists to experience a different type of recreational activity, while allowing farm owners to market their leisure agriculture products. Leisure agriculture then became highly valued by farm owners, tourists, and government. However, there was no clear policy from the Council of Agriculture to support its development in this stage. 1989 to 1994 Many new terms were created before 1989, such as “farm tourism,” “farm tour,” “farm leisure,” “agricultural tourism,” etc. This diversity of terms impeded the government’s ability to manage the development of leisure agriculture. The term “leisure agriculture” was clearly defined by the Council of Agriculture during a conference in 1989. Now more organized, the development of leisure agriculture blossomed. In 1992, 15 the initial Leisure Agriculture Management Guidance Statute was established. This statute clearly regulated the definitions of the terms used in leisure agriculture and the rules for the development. Leisure agriculture development then had its own standards to guide development and farm owners also had more clear regulations to follow. Additionally, a committee was formed to provide ongoing evaluation of the leisure agriculture industry. The National Bureau of Standards also approved the leisure agriculture logo. At this time, there were 31 leisure agriculture planning projects under way with government assistance. In this stage, the Council of Agriculture realized leisure agriculture was important and had produced a clear policy to support its development. That policy included: establishment of regulations, promotion funding, development funding, and grants to farm owners, local governments, farmers’ associations, and scholars. The Council of Agriculture assigned top priority to providing funding for individual farmers to improve their facilities. 1994 to 1999 Several problems with leisure agriculture were revealed after 15 years of experience. According to the Leisure Agriculture Management Guidance Statute, the 16 establishment of a leisure agriculture area must be larger than 12.4 hectares. Most individual farms in Taiwan are smaller than 12.4 hectares in size. Therefore, adjacent farms had to merge to reach the mandated size limit. However, many conflicts have occurred when different agricultural enterprises were merged. In addition, other issues have surfaced and have been debated. These have revolved around tax problems, water and electricity rates, number of rooms for guests, establishment of restaurant on leisure farms, size of the farm, and the definition of recreational facilities on farmland. In 1995, the Agriculture Policy White Book clearly indicated that promoting leisure agriculture development was one of government’s important agricultural development goals. Leisure agriculture became very popular; however, several farms took advantage of the name “leisure agriculture” and developed non-agricultural businesses in order to obtain tax benefits. The government did the following to adjust the direction of leisure agriculture development to reduce the chaos that had emerged: 1) modified the Leisure Agriculture Management Guidance Statute to meet the real need of the development; such as, modified the limitation of farm size from 12.4 hectares to 1.2 hectares, defined in more detail the types of facilities that could be provided on leisure 17 farms and 2) established leisure agriculture development associations to promote development. During this timeframe, the Council of Agriculture refined policy regulations to better fit its leisure agriculture development goals. In addition, it continued to provide funding for promotion of leisure agriculture enterprises. It is also noteworthy that the Council of Agriculture began providing funding for counties or townships to develop master plans for leisure agriculture development and to build some public facilities to support their leisure agriculture enterprises. After 1999 The modified Leisure Agriculture Management Guidance Statute and related legislation announced in 1999 put the industry under direct supervision of the government. Moreover, the government also formed a committee to examine and consider the further development of leisure agriculture. Since 1999, the Council of Agriculture finally has the regulation basis to manage leisure agriculture development. Promotion funding is still providing annually. A major remaining task is to integrate the potential resources of leisure agriculture in each region and to develop unique identities (brand images) for each region. 18 The Goals of Leisure Agriculture Development in Taiwan Several Taiwanese scholars have discussed goals for leisure agriculture (Cheng, 1996; Chiang, 1997; Jeng, Liu, & Chen, 1995; .l.-Y. Li, 1996; M.-H. Li, 1996; Liu, 1994, 1997; Yu, 1991); however, no one has clearly articulated specific goals for leisure agriculture development. Even though the goals have not been clearly defined, they can be divided into three general categories: economic ( 17% Sheng Chan), enjoyment ($72? Sheng Huo), and ecology (fié‘Sheng Tax). The objectives under each goal are outlined below. 1. Economic goal: The major objectives of the economic goal are to “improve farmers’ income,” “diversify farm business,” and “improve farm management.” 2. Enjoyment goal: The major objectives of the enjoyment goal are to “improve quality of $9 6‘ life,” “make more recreation opportunities available, provide opportunities to learn about agriculture,” and “increase interaction between rural and urban residents.” 3. Ecology goal: The major objectives of the ecology goal are to “maintain agricultural environments” and “protect farm land.” 19 Performance Evaluation Performance evaluation has been used by the US. governments for many years in order to improve public management and program outcomes. The Government Performance and Results Act of 1993 requires the federal government, most states, and many local governments to develop measurable outcomes for their programs (Kravchuk & Schack, 1996; Poister & Streib, 1999). Performance evaluation research has been conducted by US. government agencies (Centers for Disease Control and Prevention Office on Smoking and Health, 1999; Homik et al., 2003; MacDonald et al., 2001; United States Department of Agriculture & Service, 2002; United States Department of Education, 2004). There is no such act in Taiwan forcing government agencies to measure their performance; however, more and more researchers have begun to address this issue (Chen, 2003; Kao, 2000; Lin & Yang, 2002; Lu & Hsiao, 2003; Shen, Huang, & Chu, 2003; Wang, 2004). Evaluation is an important part of both the policy making process, as illustrated in Figure 1 (Dunn, 1994) and the program delivery cycle, as illustrated in Figure 2 (DeGraaf, Jordan, & DeGraaf, 1999). Evaluation yields policy-relevant knowledge about discrepancies between expected and actual policy/program performance, thus it will help 20 the decision maker in the assessment phase of policymaking process and the program delivery cycle. Problem Agenda SMfina Setting Policy @ Formulation I I. Policy 0 Adoption Policy l @ Implementation L O Policy Assessment Figure l. Policy-analytic procedures associated with different phases of policy- making. (Dunn, 1994, p.17) Figure 2. The program delivery cycle. (DeGraaf et al., 1999 ,p.53) 21 The basis of the policy-analytic procedure is that outcomes of policies or programs should be evaluated: that is, they should be examined to assess the extent to which they are achieving what they were intended to achieve (effectiveness) or whether they are doing so at an acceptable cost (efficiency). Evaluation is also seen as an important part of the program delivery cycle. Results from evaluation can feed into subsequent rounds of decision-making, thereby enhancing policy outcomes. Performance evaluation is discussed further in the following sections. First, the definition of performance evaluation is presented. Second, the steps in a typical performance evaluation process are outlined. Third, the types of evaluation models and approaches are discussed. Defining Evaluation Suchman stated, “an evaluation is basically a judgment of worth-an appraisal of value.” Similarly, Worthen and Sanders stated “ evaluation is the determination of the worth of a thing” (Rossman & Schlatter, 2003, p.355). They indicate that worth is the benchmark of evaluation, and evaluation is about judging the value or worth of the policy/program. 22 Koenig stated (1986, p.184), “An evaluation in its most formal sense is an examination of the effect of policies and programs on their targets in terms of the goals they are intended to achieve. (p.184)” DeGraaf, Jordan, and Degraaf (1999, p.246) stated “The other common definition of evaluation is that evaluation is a way to determine if program goals and objectives have been met. (p.246)” Henderson and Bialeschki (2002, p.5) stated that evaluation was “assessing where we are, where we want to be, and how we can reach our desired goals” (p.5). Thus, there appears to be a consensus that the purpose of an evaluation is to determine if the goals of policy have been met. Patton stated (1997, p.23), “Evaluation is a systematic process of collecting information about activities, characteristics, and outcomes of programs: to make judgments about programs, improve program effectiveness and/or inform decision- making about fiJture programming.(p.23)” The International Organization for Standardization has defined performance evaluation as “the process of informing a company’s managers and stakeholders on its performance by selecting indicators, collecting and analyzing data, assessing information against performance criteria, reporting and communicating, and periodically reviewing and improving the process” 23 (Bennett, James, & Klinkers, 1999). Thus, evaluation is a systematic process of collecting information/outcomes for a program or policy. After reviewing different authors’ definitions, the definition of evaluation to be used in this study is that evaluation is judging the worth of or improvement from a policy/program based on a set of criteria and an analysis of systematically—collected evidence/data. The Performance Evaluation Process There are three steps in the performance evaluation process (DeGraaf et al., 1999; Henderson & Bialeschki, 2002; Posavac & Carey, 2003; Rossman & Schlatter, 2003): l. Missions /Objectives Statement Establishing a performance evaluation process begins with the identification of the mission and its objectives. What does the policy/ program intend to accomplish? Performance information should flow from, and be based on, the answer to this fundamental question. 2. Outcomes and Outcome Indicators It is necessary to develop a specific list of important outcomes associated with the policy to be evaluated. Public agencies always have multiple objectives which reflect 24 categories of public concern. Thus, those making the selection of desired policy outcomes should attempt to include all relevant public perspectives and concerns. Selecting the appropriate indicators to be measured is a key part of developing a performance evaluation. Each outcome to be tracked needs to be translated into one or more outcome indicators. Efficiency indicators should meet the following criteria: 1) Relevance to the missions/ objectives, 2) Importance to the outcome, 3) Understandability to users, 4) Feasibility of collecting relevant data, 5) Uniqueness, 6) Manipulability, and 7) Comprehensiveness (Hatry & Wholey, 1999). 3. Data Collection and Analysis Finalizing a set of performance indicators requires that a data collection method be chosen. Cost, feasibility, accuracy, understandability, and credibility are the five criteria for developing data collection procedures. After the data have been collected, analysis is required in order to identify the appropriate actions that may be needed. Evaluation Methods A number of different approaches to evaluation have been developed to guide the evaluation process. In the following, the most prevalent models and approaches used for 25 evaluation will be described (Henderson & Bialeschki, 2002; Posavac & Carey, 2003; Russ-Eft & Preskill, 2001; Stufflebeam & American Evaluation Association, 2001). The Traditional Model: Evaluation was made by impressionistic evaluation or self- evaluation. Industrial Inspection Model: This evaluation involves inspecting the product at the end of the production line. Objectives-Based Evaluation: This approach emphasizes that the evaluator should work with clearly stated goals and objectives and then measure the degree to which such goals and objectives are achieved. Goal-Free Evaluation: This approach involves identifying all the positive and negative impacts of a program. Fiscal Evaluation: This approach involves projecting the financial investment needed to support a program and the return on that investment. Expert Opinion Model: This approach involves engaging outside experts to conduct the evaluation. It is most often used when the entity being evaluated is large, complex, and unique. 26 Collaborative Evaluation: Stakeholders are included in the decision-making and evaluation processes (Worthen, Sanders, & Fitzpatric, 1997). Empowerment Evaluation: In this approach, evaluators give voice to the people they work with and bring their concerns to policymakers. Theory-Driven Evaluation: This evaluation depends on key stakeholders’ needs, resources available for research, and the evaluators’ judgment (Chen, 1994). An Improvement-Focused Model: Evaluators help discover discrepancies between program objectives and the needs of the target population, between program implementation and program plans, between expectations of the target population and the services actually delivered, or between outcomes achieved and outcomes projected (Posavac & Carey, 2003). These ten evaluation models and approaches are most typical. They have been developed in the field of evaluation in response to several issues that concern evaluation researchers about the design, implementation, and concept of evaluation. In order to achieve a better evaluation, the basic concepts from three approaches Objectives-Based Evaluation, Collaborative Evaluation, and Objective-Based Evaluation will be adopted in this research. 27 In the following section, the discussion will focus on various performance evaluation research designs commonly used in recreation and tourism. They include indicator analysis, economic analysis, the benefit approach to leisure, satisfaction-based analysis, and importance-performance analysis (Henderson & Bialeschki, 2002; Rossman & Schlatter, 2003; Veal, 2002). Wilt Indicators analysis involves developing and identifying a set of performance indicators based on the goals and objectives of the policy/program and analyses of changes in these indicators. This approach requires the availability of a set of secondary data that can be used to represent performance. A typical example is the United Nations Commission on Sustainable Development and Organization for Economic Co-operation and Development which uses a DSR (Driving Force-State-Response) framework to identify a set of indicators to monitor sustainable development (Huan & O'Leary, 1999; Organisation for Economic Co-operation and Development, 1999; Organisation for Economic Co-operation and Development. Directorate for Food Agriculture and Fisheries. Policies and Environment Division, 2000). 28 W Economic evaluation is another approach for policy/program evaluation. Cost- benefit analysis and economic impact analysis are commonly used techniques as aids to decision-making at the planning stage or part of the evaluation of projects when they are implemented or completed. The two most commonly used techniques are discussed below: 1. Cost-Benefit Analysis: Cost-benefit analysis can be used to measure how effective a policy is in terms of how much it costs and how the benefits received relate to the evaluator’s investment. It measures a policy’s efficiency in monetary terms and is expressed as a ratio of the present net values of benefits to costs. It can be used in three different situations: 1) to study a single proposed project; 2) to compare alternative proposed projects, and 3) to study an existing project or projects. This is a well-known quantitative approach to measure a program’s inputs and outcomes and has been discussed by many authors in books and professional journal articles (DeGraaf et al., 1999; Fleischer & Felsenstein, 2000; Henderson & Bialeschki, 2002; 29 Lundegren & Farrell, 1985; Posavac & Carey, 2003; Purdon, Lessof, Woodfreld, & Bryson, 2001; Veal, 2002). 2. Economic Impact Analysis: Economic impact studies are not concerned with the costs of a project but only with its effects in terms of the direct and indirect financial benefits to a geographic region. This approach is used to assess the importance to an economy of a program or policy (Henderson & Bialeschki, 2002; Veal, 2002). W Driver and Bruns developed the BAL to provide a framework for the management of natural recreation areas (Jackson & Burton, 1999). Driver and Bruns outlined over one hundred types of benefits that have been identified by research as arising from leisure participation. Each benefit is potentially capable of being evaluated by means of one or more performance indicators. 5 I. ll I. ‘B l l l . Satisfaction-based analysis provides data about participant satisfaction with the program. These satisfaction data can be used to judge the value of the program. Two 30 successful examples of satisfaction-based analysis are the SERVQUAL and ACSI programs discussed below: 1. SER VQUAL: SERVQUAL, developed by Parasuraman and his colleagues, is a typical example of the performance evaluations used in service-type businesses (Parasuraman, Zeithaml, & Berry, 1985, 1988). SERVQUAL assesses performance by comparing peoples’ expectations of policy outcomes with the level , of satisfaction actually experienced. 2. American Customer Satisfaction Index (A CSI)‘: The ACSI is a national economic indicator of customer evaluation of the quality of goods and services acquired/received from companies and government agencies that produce approximately half of the US. Gross Domestic Product (GNP), plus foreign companies with substantial market shares in the United States. The index is produced and the data housed at the National Quality Research Center (N QRC) at the University of Michigan Business School. ' For a more in-depth discussion of the American Customer Satisfaction Index, see ACSI Methodology Report (Fomell, Bryant, Cha, Johnson, Anderson, and Ettlie, 1998) or visit the ACSI website at http://www.theacsi.org. 31 WW Several researchers have adopted importance-performance analysis to measure a program’s performance in the tourism and recreation fields (Hollenhorst, Olson, & Fortney, 1992; Hudson & Shephard, 1998; Siegenthaler, 1994). Importance-performance analysis uses a measurement instrument to quantify user satisfaction with performance by combining importance with satisfaction. This leads to a useful visual tool for assessing performance, in that the component scores on the two scales (importance and performance) can be plotted on a graph, as shown in Figure 3. The plot gives administrators a clear view of priorities for improvements (Henderson & Bialeschki, 2002; Veal, 2002). 5.00 5’ Indicator E o i F i e 3. Ex lanation: 1 gur p 4.00 -1 o Indicator B o Indicator D Indicator E is a most important indicator , (lmportance=5) and its performance is 3.00 J o In dicator C 1 excellent (Performance=5). 2.00 O Indicator A Indicator F is also a most important ( l= Poor; 5= Excellent) Performance indicator (Importance=5), too; however, -J r. _ ._l_ A.” Indicator F e l I I l 2.00 3.00 4.00 5.00 Importance (l= Unimportant; 5= Important) its performance is poor (Performance= ] ). g- Figure 3. An example of importance-performance analysis graphic. 32 Importance-performance analysis is an easy and popular way to evaluate the performance of policy. However, this approach has one drawback: the weights assigned to various elements affecting performance are not clear. Therefore, there is a need to evaluate the policy with a holistic approach that make the relative weights of various elements visible. The following paragraph will discuss a logical approach (the analytic hierarchy process) which will help to assign weights to a group of elements. The Analytic Hierarchy Process (AHP) What Is the Analytic Hierarchy Process? The analytic hierarchy process model was developed by Saaty in response to the scarce resources allocation challenges and planning needs of the military (Saaty, 1980). He described AHP as a multi-objective decision making approach that employs a pair- wise comparison procedure to arrive at a scale of preferences among a set of alternatives (Braunschweig & International Service for National Agricultural Research, 2000). AHP considers both qualitative and quantitative approaches to research and combines them into a single empirical inquiry. It uses a qualitative method to decompose an unstructured problem into a systematic decision hierarchy. In the quantitative sense, it 33 employs a pair-wise comparison to execute a consistency test to validate the consistency of responses. In practice, AHP focuses on assigning weights to program/policy elements. Therefore, it can help to identify the key elements in a program/policy and help to make more efficient decisions. How Does the AHP Work? The AHP procedure is based on three principles of analytic thinking: 1) constructing hierarchies, 2) establishing priorities, and 3) logical consistency (Saaty, 1995). 1. Structuring Hierarchies The first step in AHP is to decompose the decision problem into a hierarchical structure. Saaty recommended the following steps when designing a hierarchy: (1) Identify the overall goal. (2) Identify the sub-goals of the overall goal. (3) Identify criteria that must be satisfied to fulfill the sub-goals of the overall goal. (4) Identify sub-criteria under each criterion. (5) Identify the actors involved. 34 (6) Identify the actors’ goals. (7) Identify the actors’ policies. (8) Identify the options or outcomes (Saaty & Vargas, 1994). A basic hierarchical structure is illustrated in Figure 4. The above steps are the guidelines within a structured hierarchical model. Different approaches can be used to build the hierarchical structure; however, the most successful way to structure a hierarchy is brainstorming by the stakeholders. Sub-Criteria Criterion l Sub-Criterion 2] Sub-Criterion 3 I Criterion 2 _ Sub-Criterion 4 l 1 Goal Sub-Criterion 5 l , [Criterion 3 ‘ [ Sub-Criterion 6 J Sub-Goal 2 Sub-Criterion 7J Criterion 4 . --*J Sub-Criterion 8] Figure 4. An example of the basic structure of a hierarchy framework design. 2. Setting Priorities The second step in using AHP is to set the priorities and weights for each element. The elements of each level of the hierarchy are rated using the pair-wise comparison 35 approach. As Mendoza stated, “The basic principle of the procedure involves setting up a matrix consisting of observations or judgments based on pair-wise comparisons of the relative importance between and among the elements. (p. 484)” (Mendoza & Sprouse, 1989) The basic pair-wise comparison method is based on the actors’ comparative judgment between paired goals according to the importance of one goal over the other. Within goals, there are n(n-1)/2 possible paired comparisons to be made (Basarir, 2002; Torgerson, 1958). The subject is provided with the pairs and asked to define which goal in the pair is more important to him/her. Saaty’s scale of measurement uses verbal comparisons to determine the weight of criteria. Once the verbal comparisons are made, they are translated into the numerical value of the scale (Braunschweig & International Service for National Agricultural Research, 2000; Cheng & Li, 2001). The scale of measurement, which is used to elicit the comparisons recommended by Saaty, are presented in Table 5. After all elements have been compared with the priority scale pair by pair, a paired comparison matrix is formed (Saaty, 1990). The matrix is given as: p - all 012 . . aln 021 a22 . . a2n A=(a,,-)= . . . . . (i,j=1,2,...,n) _anl anl . . annd The entries are defined by the following two entry rules. 36 Rule 1: If ab- = on , then aJ-i = Na, on¢0 Rule 2: If element i is judged to be of equal important as element j, then 30’ = aj, =1 A vector of weights [w=(w1, wz,. . ., wn)] is then computed. If the judgments were perfectly consistent (ajkakj = a”) then the entire matrix would contain no error, and could be expressed as ajj=Wi/Wj. In this case, the final weights can be expressed as: n Wi= 30/ Z akj for all i=1,2,. . .,n k=l Table 5. Saaty’s scale of measurement for pair-wise comparisons. Numerical Value Verbal Scale Explanation 1.0 Equal importance 0f bOth Two elements contribute equally elements 3 0 Moderate importance of one Experience and judgment favor one ' element over another element over another 5.0 Strong importance Of one An element is strongly favored element over another 7 0 Very strong importance of one An element is very strongly ' element over another dominant 9 0 Extreme importance of one An element is favored by at least an ' element over another order of magnitude 2.0, 4.0, 6.0, 8.0 Intermediate values Used to compromise between two judgments (Forman & Selly, 2002; Saaty, 1980) 3. Logical Consistency In the evaluation process, it is important to assess the consistency of inputs provided by participants to the analyst. However, people are often inconsistent when 37 answering questions. Errors in judgment are common; therefore, the consistency ratio (CR) is used to measure the consistency in pair-wise comparisons (Cheng & Li, 2001; Saaty, 1994). Generally speaking, the smaller the value of CR, the smaller is the deviation from consistency (Ong, Koh, & Nee, 2001). Satty also recommends acceptable CR values for different matrix sizes; these CR values are (Saaty, 1995): (1) For a 3 by 3 matrix, the CR value should be equal to or less than 5% (2) For a 4 by 4 matrix, the CR value should be equal to or less than 9% (3) For a larger matrix, the CR value should be equal to or less than 10% If the CR value is more than 10 percent, the judgments are somewhat random and should be revised. There are three ways to make these revisions: (1) One way to improve the CR value is to request participants to improve the quality of their judgments in making pair-wise comparisons by providing another set of answers. (2) Another way to improve the CR value to improve consistency is the arithmetic method (compute the geometric mean of the element in each row) as suggested by Saaty (1980) or provide an algorithm to modify the given matrix as suggested by Xu and Wei (1999). However, using these methods may alter '38 the initial logic used by the respondents. Therefore, if the results of the original consistency test are too far away from the acceptable consistency, this method should not be used. (3) If the above two methods fail, then the last resort is to redevelop the decision hierarchy. The goal here is to develop a new hierarchy structure which results in more consistency in the pair-wise comparisons of elements in the decision hierarchy. Group Preference Aggregation with the AHP The procedures described above are for assessment situations involving only one decision-maker. However, a substantial number of stakeholders, interest groups, and other public entities must be involved in the program evaluation process in most cases. Hence, there is a need to develop a group program evaluation process. Forrnan and Peniwati (1998) suggest the following two major ways to aggregate information when more than one individual participates in an evaluation process: 1) aggregating the individual judgments for each set of pair-wise comparisons into an “aggregate hierarchy” [Forman and Peniwati called this method Aggregating Individual Judgments (AIJ)]; and (2) synthesizing each of the individual’s hierarchies and 39 aggregating the resulting priorities [Forman and Peniwati called this method Aggregating Individual Priorities (AIP)]. Applications of the AHP The AHP has been applied to a wide range of problem situations including: selecting among competing alternatives in a multi-objective environment, the allocation of scarce resources, and forecasting (Forman & Gass, 2001). The AHP has been widely applied in many areas, such as: prioritization (Bernadette, Krishnamurty, & Karen, 1998; Deng, King, & Bauer, 2002; Easley, Valacich, & Venkataramanan, 2000; Leung, Muraoka, Nakamoto, & Pooley, 1998; Radcliffe & Schiederjans, 2003; Swiercz & Ezzedeen, 2001; Tzeng, Teng, Chen, & Opricovic, 2002; Ye, J in, Zhang, Ling, & Barnes, 2000), resource allocation (Curry & Moutinho, 1992; Ong et al., 2001; Ridgley & Rijsberrnan, 1994; Schmoldt & Peterson, 2000), quality management(Albayrak & Erensal, 2004; Cheng & Li, 2001; Lee, Kwak, & Han, 1995; Partovi, Withers, & Brafford II, 2002; Reisinger, Cravens, & Tell, 2003; Wang, Xie, & Goh, 1998; Yurdakul, 2002), and strategic planning (Dinc, Haynes, & Tarimcilar, 2003; Kajanus, Kangas, & Kurttila, 2004; Pesonen, Kurttila, Kangas, Kajanus, & Heinonen, 2001). 40 CHAPTER 3 METHODOLOGY AND PROCEDURES Research Design “Criteria + Evidence + Judgment = Evaluation ” (Henderson & Bialeschki, 2002) The methodology employed in this study is a combination of qualitative and quantitative research involving five steps. The first step was to identify the criteria for the evaluation of leisure agriculture development policy in Taiwan and build an evaluation model using a qualitative approach. The second step was to validate criteria in the evaluation model by using Confirrnatory Factor Analysis. This step is necessary to make sure that items of an assessment instrument are relevant and representative of the intended construct for the assessment purpose. The third step was to ask stakeholders to assess performance across criteria. The fourth step was to determine the priorities and weights of the criteria by using the AHP approach. The fifth step was to calculate a performance score for leisure agriculture development policy in Taiwan. 41 The above linear step-by-step procedure is the ideal design for this research. However, population size. and time" constraints required modifications to this ideal design. Step three (assess performance across criteria) and step four (determine weights of criteria using AHP) were processed before step two (validate criteria). In other words, a more advanced assessment of the content validity for this evaluation model was made only after data were gathered from stakeholders to perform steps three and four. After the evaluation model was refined, weights of criteria were recalculated based on the refined evaluation model. The overall approach employed is presented in Figure 5 and is discussed more fully in the following section. . See page 46 for details. .. The Council of Agriculture funded this one-year project. There was not enough time to collect the data for validating criteria. 42 RIiSI-Z.~\R(‘ll I’I.()\\‘ (‘IIAR'l- lDlCAI. DESIGN "antitative A, T036" IDENTIFY CRITERIA FOR ualitatlve roach -Assess Content ValIdIty Of the EVALUATION AND , Literature Review HIerarchIcal EvaluatIon Model DEVELOP A HIERARCHI C AL , Personal Interview USMQ 9°"fi'mat0W Factor EVALUATION MODEL (IS-member Advisory Panel) } AnaIySIs , , , - Expert Judgement , (Sample from Stakeholder , J Population; i.e.. Scholars, Farm ‘ Owners, Policy Enforcers) I VALIDATE CRITERIA h \ L . I \I L i uantltatlve ASSESS PERFORMANCE DETERMINE WEIGHTS QM m ACROSS CRITERIA OF CRITERIA USING AHP Aggroach -Stakeholder l . l -(18-member p0 - I L L I 4 Advisory pulatIon * Panel) J (i.e., Scholars, CALCULATE " ____.___ Egjgyowne's' PERFORMANCE SCORE ZPIxWI Enforcers) - l . v . ”I f CONCLUSIONS h V . P ’" RICSI‘L\R(’II Fl.()\\‘ (‘IIAR'I- {\‘IUDIFIICD DESIGN" IDENTIFY CRITERIA FOR . QualitativeAgproach 1 Quantitative EVALUAT|0N AND - LIterature Revnew AOL—roach DEVELOP A HIERARCHICAL - Personal Interview -Stakeholder EVALUATION MODEL (IS-member AdVISOI'Y Panel) Population \ L - Expert Judgement (i.e., Scholars. f Farm Owners, ; 1 2°?” ASSESS PERFORMANCE DETERMINE WEIGHTS Quantitative " °’°°'5) ACROSS CRITERIA OF CRITERIA USING AHP Acumen . ., _ . , : 7 -(18-member ...... v T . Advisory uantltatlve ‘ 3 Panel) j A roach VALIDATE CRITERIA y 3:32: "'Ziiiiiiiiiliiililiiiii iliiiiiiiiiiiiifi17.7.7.7... """ RECALCULATE” , 2 Validity ofthe i...YY§!§!1T§9595'I§5'§..i‘ Hierarchical .......... . ....... J Evaluation i Model Using Confirmatory CALCULATE n Factor Analysis] PERFORMANCE SCORE ZPI’XWI , . . i=1 7 CONCLUSIONS Q Figure 5. The flow chart of this research. 43 Identify Criteria for Evaluation and Develop a Hierarchical Evaluation Model “Evaluative criteria are the specific dimensions of policy objectives that can be used to weight policy options and judge the merits of existing policies or programs. Evaluative criteria can also be thought of as justifications or rationales for a policy or government action” (Kraft & Furlong, 2004). The first step in this research design was to identify the dimensions of policy objectives to be evaluated. Identifying the evaluative criteria, categorizing them, and developing a hierarchical evaluation model for applying them are the three major tasks involved in this step. Each is discussed below. 1. Criteria identification: A number of articles have identified the purposes of leisure agriculture development policy in Taiwan (Andreoli & Tellarini, 2000; Bosshard, 2000; Cheng, 1996; Chiang, 1997; Jeng et al., 1995; J.-Y. Li, 1996; M.-H. Li, 1996; Liu, 1994, 1997; Prem & Michel, 1999; Yu, 1991). However, no article has clearly identified the criteria for evaluating this leisure agriculture development policy. Therefore, it was necessary to conduct personal interviews of experts (the advisory panel) to identify the criteria relevant for this evaluation. 44 2. Criteria categorization: Content analysis was used to capture the criteria for evaluation identified previously (interviews of the advisory panel members) and place them into categories. A three-person expert panel was employed to create the criteria response categories that were used. 3. Develop the hierarchical evaluation model: Saaty (1990) recommends limiting the number of items at any evaluation level to a maximum of nine. The reason for this is that people cannot be consistent in judging long listings of pair-wise comparisons. Therefore, it was necessary to develop a hierarchical evaluation model to limit the number of required paired comparisons at any one level of criteria comparison. Again, a three-person expert panel was employed in developing the hierarchical model used in this study. Sample “The purpose of sampling is usually to study a representative subsection of a precisely defined population in order to make inferences about the whole population” (Silverman, 2000, p. 102). The leisure agriculture policy stakeholders relevant to this 45 study belong to the following groups: scholars (population=52)', farm owners (population=198)", and policy enforcers (population=698)m. Purposive sampling was used to select the experts from each group of stakeholders to participate in this study. Selection of participants was based on their reputation, experience, and knowledge in the leisure agriculture field. Due to budget limitations, in-depth interviews of only seven experts from each group were conducted. Data Collection Procedures The in-depth interview was adopted to complete step one of the research design for the following reasons: 1) no peer pressures, 2) no potential influence or contamination by other respondents, 3) some respondents find it easier to deal with sensitive issues in a one—on-one clinical setting, 4) each respondent gets equal time, and 5) easier to schedule ° Since 2001, the Council of Agriculture has been hiring scholars to evaluate the development of leisure agriculture. The list of scholars employed by the Council of Agriculture in 2001 to 2003 was Obtained. After deleting duplicates, 52 scholars were identified. " The farm owner list was obtained from the Council of Agriculture. The total farm owner population is 198. m Policy enforcers are those who carry out leisure agriculture policy. There are two types of organizations, which carry out this policy: local government and farmers’ association. There are in total, 389 local government and 309 farmers’ associations in Taiwan. The total policy enforcer population is 698. 46 interviews at offices (Mariampolski, 2001, pp.46-54). Selected respondents were first contacted by telephone to determine their willingness to participate in this project. Those willing to participate were given the consent form that is provided in Appendix I before being interviewed. It described the purpose of the research and relevant ethical issues. Tape recorders were not used during interviews because Wolcott (2001) suggested that they distract both the respondent and the researcher during the interview. Moreover, some respondents simply do not like any kind of recorder to be used (Carson, 2001), especially when the topic of this research is related to criticizing government’s performance. Notes were taken to record responses. In order to reduce possibilities of bias and error, two interviewers taking separate notes were employed. The two interviewers met immediately after the interview to compare and finalize their notes. Data Analysis Procedures Content analysis was used to code groups of words contained in transcripts of the interviews into categories. Two steps were involved. First, category codes were assigned to words, phrases, sentences, or paragraphs initially linked to the three goals of leisure agriculture development (economic (1%zg’é‘ Sheng Chan), enjoyment (1%r‘BfSheng Huo), and ecology (flit? Sheng Tai)). Second, the coded materials were compared and 47 contrasted. The purpose of the latter was to organize the data according to the topics and sub-topics related tO the evaluation criteria. The first step is sometimes called “axial coding” and the second “selective coding” (Carson, 2001). Limitations Associated with Identifying Criteria for Evaluation and Developing a Hierarchical Evaluation Model In this section, concerns about how this research was designed to obtain the criteria for evaluation that were used and the strategies used to mitigate them will be discussed. Issue 1 .' “Awareness of being tested” and “Role selection” might be sources of errors in some measures. The respondents’ awareness of the research process might influence their responses (i.e. inaccuracy, defensiveness, or dishonesty) (Campbell & Russo, 2001). Issue 2: There is no sure way to replicate results either with the same interviewers or different interviewers. Interview limitations include the potential for distorted responses due to personal biases such as anger, anxiety, politics, and a simple lack of awareness of the topic. Interviews can also be greatly affected by the emotional state of the interviewee at the time of the interview. Interview data 48 are also potentially impacted by: recall error on the part of the interviewer, the nature of the rapport between the interviewee and the interviewer and self- serving responses (Patton, 2002). Mitigation I : Sensitive issues were avoided in the interview and tape recorders weren’t used. In order to reduce interviewer bias or miscommunications, two interviewers were used to take notes. Mitigation 2: Respondents were allowed to choose the time and place for the interview in order to reduce distractions and create optimal conditions for obtaining inputs from respondents. Assess Performance across Criteria In this step (i.e., step two), respondents were asked to assess performance across criteria based on their attitudes toward each criterion. The criteria were derived using the approach noted earlier in the first step of criteria identification. 49 Sample The population identified to be interviewed numbered 948 individual stakeholders including: 52 scholars, 198 farm owners, and 698 policy enforcers’. Questionnaires were mailed to all the stakeholders identified; hence this research plan can be described as a census rather than as a sample. Data Collection Procedures Research packets were distributed and mailed to stakeholders which consisted of an introductory cover letter, an informed consent form, the questionnaire, and a self- addressed stamped envelope (Please see Appendix II for consent form and questionnaire). One follow-up mailing was sent a week later. Limitations Associated with Assessing Performance across Criteria In this section, issues related to the approaches used to assess performance along with approaches employed to mitigate them are discussed. Issue 1: “Content validity is a necessary but not a sufficient form of evidence to support the validity of scores produced from the scale. (p.33l)” (Nugent, Sieppert, & . Please see footnote on page 46 for details. 50 Hudson, 2001). The essential concepts of content validity are that items Of an assessment instrument should be relevant and representative of the intended construct for a particular assessment purpose. Issue 2: It was not possible to test the item analysis in the pilot test because of the limited population size. According to a “rule of thumb” suggested by Nunnally, the number of responses needed for item analysis is about five responses for each item on the scale (Nunnally & Bernstein, 1994). Kline also suggested that the ideal sample size for the Confirmatory Factor Analysis (CFA) model should be assessed in terms of the ratio of subjects to free model parameters (i.e., 10:1, or even better, 20:1) (Kline, 1998, p.211). There were 33 items in the instrument used in this study. Kline’s guideline would require a sample of 330 (or even 660) to sufficiently test the goodness of fit of the CFA model by using the Analysis of Moment Structures program (AMOS). Thus, the recommended sample size for the pilot test is equal to or greater than the total population of about 900 available to conduct the overall study. Thus, it was impossible to conduct an item analysis in the pilot test. 51 Issue 3 .' Reliability refers to the extent to which a scale produces consistent results if repeated measurements are made (Malhotra, 2002, p.292). Due to time and budget limitations, it was not possible to use the test-retest procedure to assess reliability of the scales used in this study. Issue 4: Non-response bias might influence study results. Mitigation I : Experts were asked to review the questionnaire for content validity. Mitigation 2: After the survey was completed, statistical analyses were conducted to extract useful information from the data collected and to assess the quality of the outputs. Confirmatory Factor Analysis was used to assess what each item represents and whether the items measure what they were expected to measure at each content level (Ding & Hershberger, 2002). Mitigation 3: The Cronbach’s alpha was used to assess internal consistency reliability. Mitigation 4: All respondents were contacted twice to increase response rate. However, the confidential nature of the surveys made it impossible to track or analyze differences between the response and non-response groups. 52 Determine Weights of Criteria Using AHP The Analytic Hierarchy Process (AHP) was used to develop criteria weights. All of the survey responses were entered into an AHP computer implementation program called Expert Choice '. Three alternative AHP software packages are available to perform this analysis including: AutoMan, Expert Choice, and HIPRE 3+. Ossadnik and Lange (1999) conducted an “AHP-based evaluation of AHP Software” and found that Expert Choice had the best overall performance. This analysis consists of the following three- step process: 1. Collect input data (judgments) by pair-wise comparisons of the evaluation elements. These pair-wise comparisons were made by asking the question: “Which of the two elements is more important with respect to a higher level criterion, and how strong is the differences in importance, using a 1-9 scale shown in Table 5 (page 37) for the elements on the left over the element on the right of the matrix?” Pair-wise comparisons allowed the relative weight of elements to be ' Fonnan, E. H., Saaty, T. L., Selly, M.A., & Waldron, R, (1983) Expert Choice 2000, Decision Support Software, McLean, VA. 53 obtained by having decision-makers focus on a single pair of two elements at a time. 2. Check the consistency ratio (CR) of each matrix using Satty’s guideline for the acceptable CR values for different matrix sizes (See page 38.) 3. Calculate relative weights (local priorities and global priorities)" of the evaluation elements (The global weight of the evaluation elements will add up to one) Data Collection Procedures Research packets were distributed and mailed to the 18-member advisory panel (the experts who were interviewed to develop the evaluation criteria) which consisted of an introductory cover letter, an informed consent form, the AHP questionnaire (please see Appendix III for the questionnaire), and a stamped envelope for returning the .. “The priority Of a node is a numerical value represented as a percentage of one. It is derived from pair- wise comparisons with respect to the parent node. The local priorities of the children of a node add up to one. The global priorities of a node represent the portion of the parent's priority inherited by the child. The global priorities of the children also sum to the parent's global priority. The global priority of a child equals the local priority of the child times the global priority Of the parent” (abstracted from the Expert Choice software package user manual). 54 questionnaire. Follow-up reminder phone calls were made later to increase rate of response. Limitations Associated with Determining Weights of Criteria Using AHP Issues involving criteria weighting design and strategies for mitigating them are discussed below. Issue I .' Non-response bias might influence results. Issue 2: It is important to know how consistent respondents were in assigning weights to the criteria. If respondents were not consistent with their answers, then the resulting weights would not accurately reflect their true values. Mitigation 1: The AHP questionnaire used in this study was very long and complicated. In order to increase the response rate and accuracy, a detailed explanation of how to answer the AHP questionnaire was conducted earlier when the in- depth person interviews were conducted. The questionnaire also included an example to illustrate how to complete it properly. Moreover, a follow-up telephone call was used to remind panelists to complete and return the questionnaire. 55 Mitigation 2: The consistency ratio (CR) value of each paired comparison set was calculated for each respondent‘s rating. If it fell outside of the acceptable range as recommended by Satty (See page 38), respondents were asked to complete the pair-wise comparison a second time. If it still fell outside of the acceptable range, a mathematical approach (computing the geometric mean of the element in each row) was adopted to improve consistency. Validate Criteria (Refine Evaluation Model) The development of the evaluation model was based on a qualitative approach. An additional quantitative approach was used to assess and enhance the content validity of the initial evaluation model developed. This quantitative approach involved: 1) examining what each item (criterion) represents, and 2) determining whether the items (criteria) measure what they were designed to measure (Ding & Hershberger, 2002; William, Eaves, & Cox, 2002). The content validity of the evaluation model was examined based on data derived from a census of the stakeholder population through confirmatory factor analysis (CFA) using the computer software AMOS 4.0 (Arbuckle, 1996). 56 Sample The same dataset obtained from implementing step three of the research design (assessing performance across criteria) was used to examine the content validity of the evaluation model. Limitations Associated with Validating Criteria (Refining Model) Issues involving model refining and strategies for mitigating them are discussed below. Issue 1: Non-response bias might influence results. Mitigation 1: All respondents were contacted twice to increase the response rate. However, the confidential nature of the survey made it impossible to track or analyze differences between respondents and non-respondents. 57 Calculate Performance Score The equation for calculating the total performance score of peoples’ attitudes toward leisure agriculture used in this study is: n Esti stj +Eifoj xAfifj +-/I/—p;prj x Npl' Total performance score= £1 Nsi+Nfi+Npi Eguation 1 For X9:- = The sample mean of scholars for the ith indicator WSI = Scholars’ weighted mean for the ith indicator NS; = Scholars’ total number of cases for the ith indicator «7f ,- = The sample mean of farm owners for the ith indicator Wf,- = Farm owners’ weighted mean for the ith indicator Nf; = F arm owners’ total number of cases for the ith indicator 7n}; = The sample mean Of policy enforcers for the ith indicator Wpi = Policy enforcers’ weighted mean for the ith indicator Np ,- = Policy enforcers’ total number of cases for the ith indicator This equation could yield values between 0 and 10. The higher the score obtained, the more successful is leisure agriculture policy in Taiwan. 58 CHAPTER 4 RESULTS AND DISCUSSION Criteria for Evaluation and the Evaluation Model The first step in the design of this research was to identify criteria for evaluation and develop a hierarchical evaluation model. The elements in this step are to: Identify the potential criteria, categorize the criteria, and develop the hierarchical evaluation model. Results obtained for each of these elements are presented below: 1. Identify the potential criteria A total of 21 individual experts were contacted to participate in this phase of the study. Three declined to participate, and the remaining eighteen were interviewed to obtain inputs needed to develop the criteria for evaluation. They are identified in Table 6. Two interviewers took notes independently during the personal interviews. After the interview, the interviewers discussed the respondents’ answers to reach a consensus about what the subjects had said. All content (words, phrases, or sentences) related to potential 59 criteria for the evaluation were extracted. A total of 335 potential indicators‘ were identified (please see Appendix VI). Table 6. List of experts interviewed. Name Title Scholars Chao-Lang Chen, Ph.D. Professor National Taiwan University Department of Agricultural Extension Chao-Lin Tuan, Ph.D. Professor National Pingtung University of Science and Technology Department of Agribusiness Management Chien-Hsing Cheng, Ph.D. Associate Professor, Chairperson Taichung Healthcare and Management University Department of Leisure and Recreation Management Mei-Hsiu Yeh, Ph.D. Associate Professor F u-Jen Catholic University Department of Landscape Architecture Tsung-Chiung Wu, Ph.D. Associate Professor, Chairperson National Chiayi University The Graduate Institute of Leisure, Recreation, and Tourism Management Farm Owners Ch-Hung Chuo Farm Owner Toucheng Leisure Farm Ching-Lai Cheng Farm Owner Shangrilas Leisure Farm I-Fung kung Farm Owner . Da-Ann Exploration Park Project Manager (Farm owner’s son) Mike M.C. Wu Central Youth Dairy Farm Flying Cow Ranch Shih-Shih Chen Farm Manager Shin Kong Chao Feng Recreation Farm ' The 335 potential indicators were not all mutually exclusive. The same or similar indicators appeared more than one time on the initial list of 335 potential indicators (please see Appendix VI). 60 Table 6 (cont’d) Name Title Yung Ching, Chen Business Director Nan Yuan Resort Farm Policy Enforcers Ch-Hwei kung Director Hsin-Yi County Farmers’ Association F u-Cheng Kuo Director of Agricultural Division Nan-Tou County Government Hung-Cheng Cheng Vice Director Chuo-Lan County Farmers’ Association Jui-Hsiang kung Project Manager Da-Chia County Farmers’ Association Nancy Chou F armers' Service Department Council of Agriculture, Executive Yuan Tsai-Kun Lin Director I-Lan County Farmers’ Association Joseph Cheng Deputy Secretary General Taiwan Leisure Farming Development Association 2. Categorize the potential indicators-content anghlsis After identifying the potential indicators, they had to be categorized. Three hundred and thirty five indicators are not a small number for categorizing. Therefore, the first task was to determine how to categorize them in a systematic way. “Leisure agriculture is an industry that combines economic (if; Sheng Chan), enjoyment (532% Sheng Huo), and ecology ($112 Sheng Tai) together” (Chen, 2002). This concept was very popular in the agriculture field, and the government, scholar, and farm owner stakeholders engaged in this study all accepted this three faceted concept of the purposes of leisure agriculture. Therefore, the potential indicators were first grouped into these three categories: “economic, enjoyment, and ecology.“ Judgments of which 61 categories the indicators belonged to were based on the opinions of a three-person expert panel. The three-person panel determined that the 335 indicators should be grouped as follows: 209 economic indicators, 69 enjoyment indicators, and 57 ecology indicators. 3. Develop the hierarchical evaluation model After grouping the 335 indicators into the three groups, it was Observed that the indicators under each goal were very diverse’. Therefore, it is necessary to develop a hierarchy of sub-categories by sorting these indicators into different dimensions. The three-person expert panel recoded: the 209 economic indicators into six dimensions (Assist Farm Management, Educate Farmers, Improve Farmers’ Economic, Use Farm Resources Wisely, Diversify Farm, and Business Make Farming Attractive,) the 69 enjoyment indicators into four dimensions (Retain Traditional Culture, Making Recreational Opportunities Available, Improve Quality of Life, Maintain Community Structure,) and the 57 ecology indicators into three dimensions (Protect Environment, Maintain Agricultural Environment, Educate Environmental Protection.) 'Saaty recommended limiting the number of pair-wise comparison to a maximum of nine (Saaty, 1990). 62 The above step sorted the indicators into a hierarchy of interrelated elements, which can be described as a tree containing the overall goal at its top with many levels of dimensions in between and the indicators at the bottom. The indicators under each dimension are discussed in detail below. Economic Indicators The economic dimension “Assist Farm Management” consists of the following three indicators: assist marketing, assist cooperation, and assist farm operation; the dimension “Educate Farmers” consists of the following five indicators: adjust temperament, increase receptiveness, assist farmers’ interpretative ability, develop farmers’ creativity, and change farmers’ thinking; the dimension “Improve Farmers’ Economic” consists of two indicators: increase farmers’ income resources, and increase farmers’ income; the dimension “Use Farm Resources Wisely” also consists of two indicators: reveal agricultural uniqueness, and maintain current agriculture; the dimension “Diversify Farm Business” consists of two indicators: attract new investments, and expand traditional agriculture; the dimension “Make Farming Attractive” consists of the following three indicators: increase number of tourists, increase tourists’ satisfaction, and promote the image of leisure agriculture. 63 Enjoyment Indicators The enjoyment dimension “Retain Traditional Culture” consists of two indicators: preserve the current culture, and educate the current culture; the dimension “Making Recreational Opportunities Available” consists of two indicators: supply recreational activities, and supply recreational locations; the dimension “Improve Quality of Life” consists of two indicators: improve infrastructure of rural areas, and improve spiritual life of farmers; the dimension “Maintain Community Structure” consists of three indicators: improve demographic composition of rural areas, increase community vitality, and increase interaction between rural and urban areas. Ecology Indicators The dimension “Protect Environment” consists of three indicators: preserve environment, repair environmental damages, and reduce negative impacts of development; the dimension “Maintain Agricultural Environment” consists of two indicators: preserve agricultural landscape, and preserve rural community; the dimension “Educate Environmental Protection” also consists of two indicators: environmental education of farm owner, and environmental education of tourists. 64 This structure of the evaluation hierarchy was also presented to several scholars for their feedback. Figure 6 shows the final structure of the hierarchical evaluation model that was developed. 65 ——+i7 Assist Marketing —->l7 Assist Farm Management kal’ Assist Cooperation ] I Assist Farm Operation l I—Pl’f Adjust Temperament J 4 Increase Receptiveness J —>I Educate Farmers f—ahliAssist Farmers' Interpretative Ability I I Develop Farmers' Creativity J ‘4 Change Farmers' Thinking! Increase Farrners' Income SourcesJ 99 Economic —>l Improve Farmers' Economic Increase Farmers' Income l Reveal Agricultural Uniqueness -——>l Use Farm Resources Wisely L+l Maintain Current Agriculture I ~——§l Diversify Leisure Farm Business L—bl Make Farming Attractive Attract New Investments 1 Expand Traditional Agriculture Increase Number of Tourists l Increase Tourists' Satisfaction l Eh Promote the Image of Leisure Agriculture J Preserve the Current Culture l Leisure 4 Retain Traditional Culture Agriculture -——— Educate the Current Culture ‘ Development Supply Recreational Activities ——>l Making Rec. Opportunities Available Supply Recreational Locations Enjoyment ~—>l Improve Quality of Life Improve Infrastructure of Rural Areas J Improve Spiritual Life of Farmers J Improvel , r t of Rural Areas ~—>L Maintain Community Structure Increase Community Vitality l Increase Interaction between Rural and Urban Areas £131.31 Preserve Environment Protect Environment Repair Environmental Damages Reduce Negative Impacts of Development Ecology Maintain Agricultural Environment fl Preserve Rural Community I I Preserve Agricultural Landscape ‘ I I Environmental Education of Farm Owners Educate Environmental Protection Environmental Education of Tourists l Figure 6. The hierarchical evaluation model. for eraf respons Table (111116 I Ihepc ~ll'llS] Emu; mean Results of Assessing Performance across Criteria Nine hundred and forty eight questionnaires were mailed out to the stakeholders for evaluating the Council of Agriculture’s performance of leisure agriculture. The valid response rate was 53.7% as can be seen in Table 7. Table 7. The response rate to the performance assessment survey by stakeholder groups. Farm Policy Scholars Owners Enforcers Total # of Questionnaire Mailed 52 198 698 948 # of Response 33 68 440 541 Response Rate 63.5% 34.3% 63.0% 57.1% # Of Valid Response 25 62 422 509 Valid Response Rate 48.1% 31.3% 60.5% 53.7% Table 8 shows the results of the performance rating by stakeholder groups across criteria. Generally speaking, the participants’ ratings in each stakeholder group. were quite consistent. However, these initial results showed that farm owners tended to assess the performance of Council of Agriculture’s leisure agriculture policy lower than did ' This research used coefi'rcient of variation (CV=standard deviation/mean) to indicate the consensus of the group of performance assessments by the participants. The CV value of the criteria is small, which means the responses within the group were consistent. 67 policy enforcers and scholars did. In order to evaluate the differences, one-way ANOVA was used to assess the statistical significant of the mean differences across the three stakeholder groups. The results are presented in Table 8 and indicate significant ,9 $6 differences for the following nine indicators: “assist cooperation, assist farm 99 66' 99 ‘6' adjust temperament, Increase farmers’ income sources, Increase 99 ‘6 Operation, 9’ ‘6 farmers’ income,” and “reveal agricultural uniqueness, maintain current agriculture,” ’9 6‘ “increase number of tourists, supply of recreational locations.” Eight of the nine are indicators linked to the economic goal. The mean rating for scholars is more frequently higher than for the other two groups, especially for the indicators under the general economic category. Farmers, those most directly economically impacted by the Council of Agriculture’s leisure agriculture policy, typically rate its economic indicators lower than do either of the other two stakeholder groups. However, it should noted that the nominal mean performance rating differences proved to be statistically significant for only 9 of the 33 indicators that were evaluated. 68 Table 8. Mean performance evaluation ratings by stakeholder group for 33 program performance indicators. kneiated Goal) N Mm s ANOVA Kruskal Wallis Test Indicator F Sig. Chi2 Sig. Economic) Farm Owners 62 5.15 1.91 1. Assist Marketing Policy Enforcers 420 5.62 1.72 Scholars 25 5.96 1.65 2.645 0.072 7.334 0.041 Total 507 5.58 1.75 (Economic) Farm Owners 62 5.10 2.06 2. Assist Cooperation Policy Enforcers 421 5.55 1.61 Scholars 25 6.20 1.73 4.104 0.017 9.491 0.009 Total 508 5.53 1.69 (Economic) Farm Owners 62 4.90 2.01 3. ASSiSt Farm Operation Policy Enforcers 421 5.56 1.65 Scholars 25 5.84 1.95 4.467 0.012 7.945 0.019 Total 508 5.49 1.73 (Economic) Farm Owners 62 4.74 2.14 4. Adjust Temperament Polic Enforcers 419 5.40 1.76 Scholars 25 5.04 1.86 3.868 0.022 7.875 0.019 Total 506 5.30 1.82 (Economic) Farm Owners 61 5.44 1.95 5. Increase Receptiveness Policy Enforcers 417 5.56 1.74 Scholars 25 5.44 1.89 0.149 0.862 0.075 0.963 Total 503 5.54 1.77 (Economic) Farm Owners 62 5.60 2.22 6. Assist Farmers’ Policy Enforcers 421 5.73 1.74 ___ ___ 0 819 0 664 Interpretative Ability Scholars 25 6.04 1.70 ' ' Total 508 5.73 1.80 éEconomic) Farm Owners 62 5.15 2.27 . Develop Farmers’ Policy Enforcers 421 5.58 1.80 Creativity Scholars 25 6.04 1.65 2.399 0.092 5.326 0.070 Total 508 5.55 1.86 (Economic) Farm Owners 62 5.65 2.10 8. Change Farmers’ Policy Enforcers 422 5.87 1.79 Thinking Scholars 25 6.24 1.81 0.979 0.377 0.173 0.173 Total 509 5.86 1.83 (Economic) Farm Owners 6 5.89 2.07 9. Increase Farmers’ Policy Enforcers 422 5.54 1.83 Income Sources Scholars 25 6.36 1.35 3.060 0.048 8.185 0.017 Total 509 5.62 1.85 (Footnote 1. Scale: 10=Superior; 0= Failing) (Footnote 2. The Nonparametric test (Kruskal Wallis Test) was used if the variable did not meet the assumption of “homogeneity-of-variance” when running the One-Way ANOVA test.) 69 Table 8 (cont’d) Goal) N Mean S ANOVA KruskalWaIlisT ' tor F Sik c1112 Sgg' . (Economic) Farm Owners 61 5.90 2.21 10. Increase Farmers’ Policy Enforcers 420 5.55 1.81 Income Scholars 25 6.44 1.50 3.411 0.034 9.63 0.008 Total 506 5.64 l .86 (Economic) Farm Owners 62 5.95 2.1 1 11. Reveal Agricultural Policy Enforcers 419 6.24 1.55 Uniqueness Scholars 241 6.88 1.45 6569 0'0” Total 505 6.241 1.63 (Economic) Farm Owners 62 5.39 2.06 12. Maintain Current Policy Enforcers 419 5.88 1.61 , Agriculture Scholars 25 6.24 1.5 3.091 0.046 7.013 0.030 Total 506 5.83 1.681 (Economic) Farm Owners 62 5.19 2.30 13. Attract New Policy Enforcers 418I 5.55 1.75 Investment Scholars 25 6.00 1.41 "' 3218 0200 Total 505 5.53 1.82 (Economic) Farm Owners 62 6.10 2.0 14. Expand Traditional olicy Enforcers 417 5.88I 1.83 Agriculture Scholars 25 6. 40 1531.229 0.294 3.565 0.168 Total 504 5.93 1.85 (Economic) Farm Owners 61 6.52 2.071 15. Increase Number Of Policy Enforcers 419 6.23 1.71 Tourists Scholars 241 7.011 1.25 8'78° 0'0” Total 5041 6.31 1.75 (Economic) arm Owners 61 6.08] 2.00 16. Increase Tourists’ Policy Enforcers 420 5.9 1.53 Satisfaction Scholars 25 6.40 1.41 2544 0280 ITotal 506 6.01 1.60 (Economic) Farm Owners 62 6.1 2.06 17. Promote the Image of Policy Enforcers 421 6.11 1.63 Leisure Agriculture Scholars 25 6.641 1521.164 0.313 3.531 0.171 Total 50 6.15 1.681 KEnjoyment) Farm Owners 62 5.65 2.07 18. Preserve the Current Policy Enforcers 421 6.06 1.67 --- ___ 3 701 0 157 Culture Scholars 25 5.84 1.80 ' ' Total 5081 6.00 1.73 70 Table 8 (cont’d) ted Goal) N Mm S ANOVA Kruskal Wall's Testl ' tor F gig. (31112 Sig.— (Enjoyment) Fann Owners 62 5.56 2.06 19. Educate the Current PoligyI Enforcers 420 6.00 1.7 Culture Scholars 25 5.64 1.75 1.961 0.142 3.736 0.154 Total 507 5.93 1.7 (Enjoyment) Farm Owners 62 5.71 2.1 l 20. Supply of Polic Enforcers 421 6.08 1.67 Recreational Activitiesscholsrs 25 6,16 1,77 1276 0'280 33% 0'192 Total 508 6.04 1.73 (Enjoyment) Farm Owners 62 4.79 2.41 21. Supply of Polic Enforcers 421 5.76 1.63 Recreational Locationsschoérs 24 6.08 1,25 --- --- 12'845 0'0” Total 507 5.65 1.75 (Enjoyment) Farm Owners 62 5.85 2.10 22. Improve Infrastructure Policy Enforcers 421 5.89 1.77 of Rural Areas Scholars 25 5.6 1.65 0.161 0.851 0.160 0.923 Total 508 5.87 1.80 (Enjoyment) Farm Owners 62 5.63 2.26 23. Improve Spiritual Life Policy Enforcers 421 5.41 1.76 ___ ___ 1 744 0 418 of Farmers Scholars 24 5.13 l .57 ' ' Total 507 5.4 1.82 (Enjoyment) Farm Owners 62 5.56 2.09 24.1mprove Demographic Policy Enforcers 421 5.30 1.73 ..- --- 4 270 0 118 Composition ofRuraI Scholars 25 5 .92 1 .29 ' ' Areas Total 508 5.36 1.76 (Enjoyment) Farm Owners 62 5.84 2.14 25. Increases Community Policy Enforcers 420 5.96 1.67 ___ --- 2 20] 0 333 Vitality Scholars 25 6.36 1.52 ' ' Total 507 5.96 1.73 (Enjoyment) Farm Owners 62 6.29 2.15 26. Increases Interaction Policy Enforcers 421 6.35 1.61 --- --- 1 657 0 437 between Rural and Scholars 25 6.68 1.49 ' ' Urban Areas Total 508 6.36 1.68 (Ecology) Farm Owners 62 5.94 2.25 27. Preserve Environment Policy Enforcers 420 6.10 1.73 Scholars 25 5.52 1.66 3'28 0'20] Total 507 6.06 1.80 71 Table 8 (cont’d) ted Goa.) N Mean 5 ANOVA lKruskal Wallis dicator F Sii Chi2 Sig.__ (Ecology) Farm Owners 62 5.92 2.25 28. Repair Environmental Policy Enforcers 420 6.00 1.77 --- ___ 2 570 0 277 Damages Scholars 25 5.44 1 .69 ' ' Total 507 5.96 1.83 (Ecology) Farm Owners 60 5.28 2.32 29. Reduce Negative Policy Enforcers 420 5.50 1.69 ___ ___ l 810 0 405 Impacts of Scholars 24 4.92 1.86 ' ' Develorament Total 504 5.44 1.78 (Ecology) Farm Owners 61 5.97 2.28 30. Preserve Agricultural Policy Enforcers 420 6.11 1.63 ___ --- 1 776 0 411 Landscape Scholars 25 6.36 1.58 ' ‘ Total 506 6.10 1.71 (Ecology) Farm Owners 62 5.65 2.11 31. Preserve Rural Policy Enforcers 420 6.20 1.74 Community Scholars 25 5.841 1.84 2.897 0.056 5.641 0.06 Total 507 6.1 l 1.80 (Ecology) Farm Owners 62 6.02 2.20 32. Egvironmental Policy Enforcers 419 6.05 1.59 --- --- 0.247 0.884 ucatlon of Farm Scholars 25 5.84 1.75 Owners Total 506 6.03 1.68 (Ecology) arm Owners 62 5.19 2.33 33. Environmental Policy Enforcers 421 5.82 1.71 --- ___ 4 950 0 084 Education Of Tourists Scholars 25 5.56 1.66 ' ' Total 508 5.73 1.80 72 Results of Determining Weights of Criteria using AHP The 18-member advisory panel interviewed initially to collect data for implementing step 1 of the study design was contacted again and invited to judge the pair-wise comparison results for each evaluation element. Fourteen agreed to participate including: five scholars, four farm owners, and five policy enforcers. All of their responses were entered into AHP computer implementation software called Expert Choice. This analysis consists of the following three-step process: 1. Collecting input data (judgments) by pair-wise comparisons of the evaluation elements. 2. Checking the consistency ratio (CR) of each matrix. 3. Calculating relative weights (local priorities and global priorities) of the evaluation elements. After collecting responses from the experts, the CR value of each matrix and each respondent was first examined to determine the consistency of his/her responses. The respondents were asked to provide new sets of pair-wise comparison answers if their responses failed to meet the criteria recommended by Saaty (See page 38). After calculating the CR values, each of these fourteen respondents was found to have provided 73 at least one CR value which failed to meet the criteria. For these cases, respondents were asked to provide another and more consistent set of answers. Most problem cases were resolved by respondents in their second time through the process. Only two respondents, with one case each, failed to meet the Saaty’s criteria on their second attempts. Since the deviation from Satty’s criteria was minimal in both cases, no action was taken to further refine them. The experts in the 18-member advisory panel were from three different stakeholder groups with different value systems. However, this research is not concerned with each individual’s resulting alternative priorities. Hence, aggregating individual priorities (AIP) was deemed appropriate in this case. The arithmetic mean was used to aggregate individual priorities (for more details about the mathematical procedure, see references below to Aczel and Roberts (1989); Ramanathan and Ganesh (1994); Forrnan and Peniwait (1998); Chwolka and Raith (2001) . The results of the relative weights (local priorities and global priorities) of the evaluation elements are Shown in Table 9 and Table 10. Since the evaluation criteria still need to be validated, the weightings of criteria will be presented after discussion of validation of the evaluation model. 74 Table 9. Relative weights (local priorities) of evaluation elements by stakeholder group. (S: Scholars, F0: Farm Owners, PE: Policy Enforcers) . . . S FO PE Total WWW) Mean Mean Mean Mean Economic 0503 0236 0236 0.331 Enjoyment 0263 0282 0.413 0.32 Ecology 0234 0.482 0351 0347 Total 1.000 1.000 1.000 1.000 . . . . . S FO PE Total DlmenSIOn (Local PriorItles) Mean Mean Mean Mean ' )AssistFarmMamgement 0.136 0.131 0.110 0.125 (Economic)EdtmteFarmerS 0.081 0.099 0265 0.152 (Economic) Improve Fanners'Economic 0228 0.12 0.094 0.150 (Economic) Use FannReeowomWisely 0233 0242 0255 0243 'cLIchsinFannBtBirms 0.106 0.21 0.156 0.157 (Economic)MakeFarmingA11ractive 0216 0.185 0.120 0.173 Total 1.000 1.000 1.000 1.000 )RetainTladitiolelltme 0304 0283 0256 0281 (Ejoyment)MaldngRec. Opportunities Available 0.192 0.273 0218 0.24 (FnjoymmtflmplovthnlityofIife 0228 0289 0396 0306 ' )MaintainCommrmityStrucune 0276 0.155 0.130 0.189 Total 1.000 1.000 1.000 1.000 logy)PlotectEnvirom1ent 0.486 0330 0354 0394 (Ecology)MaintainAgricultmalEnviromrent 0218 0284 0288 0262 )EdtmtanvironmentalProtection 0296 0386 0.358 0344 Total 1.000 1.000 1.000 1.000 75 Table 9 (cont’d) (S: Scholars, F0: Farm Owners, PE: Policy Enforcers) hndicator (Local Priorities) S FD PE Total Mean Mean Mean Mean (Amist Farm Mamganent) 1. AssistMad10, then the variable may be redundant with others in the data set. As can be seen in Table 11, all of the calculated VIF values are small; therefore, multicollinearity is not a relevant issue in the results from this study. 80 3. Outliers: There is no absolute definition of what is an extremely large or small entry in a given data set. A common rule of thumb is that if the datum is more than three standard deviations away from the mean, then that datum may be an outlier. In this data set, all data items are within three standard deviations of the mean. In other words, there are no extremely high or low data in this data set. 4. Normality: The absolute values of skew and kurtosis can be used to interpret the distribution of the individual variables. “There are few clear guidelines about how much non-normality is problematic. Data sets with absolute values of skew indexes greater than 3.0 seem to be described as extremely skewed by some authors..., absolute values of the kurtosis index form 8. 0 to 20.0 have been described as indicating extreme kurtosis” (Kline, 1998, p 82). The values of skewness and kurtosis presented in Table 12 indicate that the distributions of all the variables are not perfectly normal; however, the absolute value of all variables’ skewness and kurtosis are below some scholars’ recommended values. This suggests that the distributions of the data analyzed should not significantly, negatively impact the quality of results obtained from the analyses performed. 81 Table 11. Table of variance inflation factor (VIF*) values. [* VIF=1/ (1-R2)] (Kline, 1998, p 78) (“C” is the abridgment of Criteria e.g. C1= Criteria 1) 82 ‘ 73571:: 7‘: I124 I" ..‘ Had [fu‘l'firIS/S7fi‘fL-Elfll':l Table 12. Examination of data for normality. Mean S Skewness Kurtosis (Feornnic)1.AssistMarketing 5.58 1.75 -0.82 1.34 (Economic)2.AssistCooperation 5.53 1.69 -0.81 1.06 (Eoornnic)3.A$istFarmOperation 5.49 1.73 -0-81 0.91 (Fmrrxnic)4.AdjmtTemperament 5.30 1.82 074 1.00 (Feamric)5.lrneaseReoeptiveress 5.54 1.77 -0.70 0.97 (Econrxnic)6.AssistFanrrers’InterrxetativeAbility 5.73 1.80 -0.67 0.85 (Ecornnic)7.DevelopFanners’Oeativity 5.55 1.86 -0.71 0.81 (Boormric)8.ChangeFarmers’Thinking 5.86 1.83 -0.78 1.25 (Eocxmic)9.lrneaseFanners’IrmneSomees 5.62 1.85 -0-61 0.84 (Eocrmic)lO.IruerseFanners’Irmne 5.64 1.86 -0.70 1.06 (Eoonornic)ll.RevealAgriculurralUniquenels 6.24 1.63 -0.70 1.53 (Eommric)12.MaintainCrmentAgriculune 5.83 1.68 -0.79 1.85 (Ecamric)l3.AttractNewInvestmart 5.53 1.82 -0.74 0.88 (Ecmnrnic)l4.ExpandTraditiomlAgriarlurre 5.93 1.85 -0.85 1.82 (Eeornnic)15.1rx:rerseNmnberofI‘omists 6.31 1.75 -0.82 1.61 (annic)l6.lrx:reaseTomists’Satistaction 6.01 1.60 -0.75 1.74 (Eoonornic)17.Promotetl'eImageoerisureAgricultme 6.15 1.68 -0.75 1.77 (Enjoymart)18.PreservetheC\nrentQIlture 6.00 1.73 -0-76 1.37 (Bjoymart)19.ErhxatetheOmthulune 5.93 1.78 -0.69 1.09 (Errioyment)20.SupplyofRecreatiomlActivities 6.04 1.73 -0.67 1.60 (Bjoynm)21.SupplyofRemeafianlIncafiorrs 5.65 1.75 -0.90 1.34 (BjoymerrtflllmpovelnfiastnmeomealArers 5.87 1.80 -0.65 1.14 (Errioyrnent)23.lrnp'oveSpiritualLifeofFarmers 5.42 1.82 -0.72 1.07 Wfl4hnpoveDamgraplficComposifimomealArms 5.36 1.76 -0.84 1.29 (Frrioyrnent)25.lrneasesCommunitertality 5.96 1.73 -0.85 1.63 (Emmfldlrmasslmflacfimbammmlandum 6.36 1.68 -0.84 1.89 (Eoology)27.PreserveEnvirmment 6.06 1.80 -0.74 1.16 (Ecology)28.RepairanvirmmartalDamage 5.96 1.83 -0.67 1.10 (Eoolog)29.ReduceNegativelrrrpactsofDeveloprnent 5.44 1.78 -0.88 1.20 (Foolog)30.PreserveAgriculturalIarxiscape 6.10 1.71 080 1.57 (Ecology)31.PreserveRuralCommmity 6.11 1.80 -0.87 1.34 (Eoolog)32animnmentalEdrmtionofFarmOwners 6.03 1.68 -0.83 1.70 (Eoology)33.EnvironrmrtalermtionofTomi§s 5.73 1.80 -0.86 1.29 83 Data screening indicated that this data set is acceptable for structure equation modeling analysis. In order to test the factorial validity of the evaluation model, a Confirmatory Factor Analysis (CFA), utilizing maximum likelihood procedures with the covariance matrix as input, was performed using AMOS 4.0 (Arbuckle, 1996). In order for a Confirmatory Factor Analysis (CF A) model to be identified, there must be at least three factors. Also, each first—order factor should have at least two indicators (Kline, 1998, pp. 203-207). The model shown in Figure 6 (page 62) satisfies both of these requirements. To evaluate the fit of the model to the data, the chi-square statistic ( 2’2 ), the Normed Fit Index (N F I), the Comparative Fit Index (CFI), and the Root Mean Square Error of Approximation (RMSEA) were examined. These indices were chosen because some adjust for sample size (CF I), and all are appropriate for CF As with maximum- likelihood procedures (Tabachnick & F idell, 2001). Additionally, the Goodness of Fit Index (GFI) and the Root Mean Residual (RMR) were assessed. The results revealed a generally satisfactory fit of the evaluation model to the data. The chi-square for the initial evaluation model was, x2 (216) = 1730.64, p < .01. (xz/df =3.613), and the following other goodness of fit statistics were found to be satisfactory : NFI = .902, CFI = .927, RMSEA = .072, GFI = .821, RMR = .138 (The 84 acceptable level of fit index for xz/df is less than 3. The acceptable level of fit indices for NFI, CPI, and GF I is .90. The acceptable value of the RMSEA is about .08 or less. An RMR of zero indicates a perfect fit. The smaller the RMR is, the better (Arbuckle & Wothke, 1999; Byme, 2001; Kline, 1998) Modification indices and residual covariance from the analysis indicated that the following 12 indicators were correlated with a subscale: Adjust Temperament, Increase Receptiveness, Change Farmers’ Thinking, Attract New Investment, Expand Traditional Agriculture, Increase Number of Tourists, Preserve the Current Culture, Educate the Current Culture, Supply Recreational Activities, Supply Recreational Locations, Improve Demographic Composition of Rural Areas, and Reduce Negative Impacts of Development. It seemed desirable to remove these items from the evaluation model to obtain a better model-data fit. When they were removed, CF A was estimated with the remaining 21 indicators resulting in a good fit of the model to the data, as all the fit indices met statistical criteria, x2 (.76, = 463.43, p < .01. (xz/df =2.633), NFI = .954, CFI = .971, RMSEA = .057, GFI = .922, RMR = .082. In other words, the item-content structure and relations between each content area for the evaluation is better than in the original model. Figure 7 displays the structure of the refined evaluation model. 85 .25 t/ "l Assist Marketing .24 . :/ .87~>l Encourage Community Cooperation 21 t/ 7" Personalize Assistance J .29 Assist Farmers” Interpretative Ability .25 Assist Farm Management 98 .84 Continue Farmers' Education .87 . . *l Develop Farmers’ CreatIVIty 17 9114 Increase Farmers' Income Sources '13 Improve Farmers‘ Economic .93 . *{ Increase Farmers Income 36 .804 Reveal Agricultural Uniqueness .33 .82 ‘l Maintain Current Agriculture 24 Wrincrease Tourists' Satisfaction J _17 .91 Make Farming Attractive Promote the Image of Leisure Agriculture .20 .11 / b/ 4 Improve Infrastructure of Rural Areas 04 mi” 20 ‘ Im rove Quali of Life 0 ./ a: p ty ,, .sou M- _ a l/ _ -~V V "I Ilr'lplUVC spit-tum LII m r airriets J Agriculture .98 w .12 .14 Development .94 93" Increase Community Vitality K40 ‘78 Increase Interaction between Rural and L Urban Areas Maintain Community Structure .19 ~15 ,7{ Preserve Environment .12 .92 / 34% Repair Environmental Damages } Protect Environment .24 PreserveA ricult ral Landsca 6 Maintain Agricultural 87" 9 U P .29 Environment 34" Preserve Rural Community '23 884 Environmental Education of Farm Owners r23 .88 Educate Environmental Protection Environmental Education of Tourists Note: The unstandardized versions of the above estimates are significant at the .01 level except for those designated “ns,” which means not significant. The standaridized values for the disturbances of the unobserved (latent) variables and for the measurement errors of the indicators are proportions of unexplained variance. Figure 7. The modified evaluation model and the standardized solution. Crc CCOI indi mod proc mod rema glob, 101101 ecolc pTOgI mini; Welgl The internal consistency reliability for each of the factors was calculated using Cronbach’s coefficient alpha (Cronbach, 1951). To be acceptable, alpha values showed be greater than .70; the a value of all items (21 indicators) is .97; the a value of economic goal indicators (11 indicators) is .95; the a value of enjoyment indicators (6 indicators) is .90; and the a value of ecology goal indicators (6 indicators) is .94. Since several indicators and dimensions were deleted from the initial evaluation model based on the analysis of confirmatory factor analysis, the weights assessment procedure needs to be recalculated using the Expert Choice software one more time. The modified hierarchy in Figure 7 was used to determine refined weightings of the remaining criteria. The results of the recalculated relative weights (local priorities and global priorities) of the evaluation elements are shown in Table 13 and Table 14. The ranking by goal weights (“Total” global priorities of goal in Table 14) is as follows: ecology (.347), economic (.331), and enjoyment (.322). This indicates that ecology is considered to be the most important goal for Taiwan’s leisure agriculture program and enjoyment is the least important goal. The differences across goals in minimal, so one can conclude that each of the three is equally important. However, goal weight differences exist across the three stakeholder groups queried in this study. 87 Scholars feel that the economic goal is the most important goal; farm owners place greatest importance on the ecology goal; and policy enforcers think that the enjoyment goal is the most important goal. Global priorities of the 21 indicators are also shown in Table 14. The results show the top five most important indicators are: (Enjoyment) Improve Spiritual Life of Farmers (weight= 0.126); (Enjoyment) Improve Infrastructure of Rural Areas (weight= 0.080); (Ecology) Preserve Environment (weight= 0.078); (Ecology) Environmental Education of Farm Owners (weight= 0.077); and (Enjoyment) Increase Community Vitality (weight= 0.069). 88 Table 13. Recalculated relative weights (local priorities) of evaluation elements by stakeholder group. (S: Scholars, F0: Farm Owners, PE: Policy Enforcers) Economic 0.503 0236 0236 0.331 Enjoyment 0263 0282 0.413 0.322 Ecology 0234 0.482 0351 0.347 Sum 1000 1.000 1.000 1.000 . . . .. S FO PE Total D1mensron (Local Pnorltres) Mean Mean Mean Mean (Eoontmic)AssistFarml\/krnagermrt 0.157 0.165 0.131 0.150 ')EducateFarmers 0.096 0.122 0.312 0.181 (EconomicflnpriveFamrers'Eomnnic 0.251 0.158 0.101 0.171 (Ecorxxnic)UseFarmResourcesWrse§r 0.256 0.322 0.312 0.294 (Eoornnic)MakeFanningAttractive 0.240 0.233 0.144 0.204 Sum 1.000 1.000 1.000 1.000 (EnjoymentflrgproveQualityofIife 0.440 0.646 0.686 0.587 (Bijoymerrt)MaintainCommunityStnmrre 0.560 0.354 0.314 0.413 Sum 1.000 1.000 1.000 1.000 (Ecology/)ProtectEnvironment 0.486 0.330 0.354 0.394 logy/)MairrtainAgriarlunaanvironment 0.218 0.284 0.288 0.262 logy)Edu:ateFrrvironmentalProtectim 0.296 0.386 0.358 0.344 Sum 1.000 1.000 1.000 1.000 89 Table 13 (cont’d) (S: Scholars, F0: Farm Owners, PE: Policy Enforcers) Indicator (Local Priorities) S no PE T0“ Mean Mean Mean Mean (AssistFannManagerrrent)l.AssistMarketing 0.362 0.497 0.335 0.391 (AssiaFarmManagementflAssistCooperation 0.150 0.214 0.352 0.241 (AssistFannMarragernmt)3.A$istFarrnOperation 0.488 0.289 0.293 0.368 Sum 1.000 1.000 1.000 1.000 (EduateFarmers)6.AssistFanners’InterpretativeAbility 0.507 0.173 0.233 0.314 (EdrmteFarmers)7.DevelopFanners’Creativity 0.493 0.827 0.767 0.686 Sum 1.000 1.000 1.000 1.000 (hrqxoveFanrers’Ecmornic)9.hneaseFanners’hroomeSomoes 0.423 0.250 0.583 0.431 (hnpoveFanners’Eoommic)10.ImsmeFarmers’Income 0.577 0.750 0.417 0.569 Sum 1.000 1.000 1.000 1.000 seFannResouroesWrsely)11.RevealAgriculunalUniqueness 0.701 0.766 0.692 0.716 seFarmResomeesWrsely)12.MaintainCunentAgiiculture 0.299 0.234 0.308 0.284 Sum 1.000 1.000 1.000 1.000 NakeFarmingAttractive)l6.IncreaseTourists’Satisfaction 0.292 0.542 0.583 0.467 WFWWWWWWOW 0.708 0.458 0.417 0.533 Agnorlture Sum 1.000 1.000 1.000 1.000 90 Table I3 (cont’d) (S: Scholars, F0: Farm Owners, PE: Policy Enforcers) S F0 PE Total Mean Mean Mean Mean (ImpoveQualityoflife)22.ImgoveInfiastnx:uneomealAm 0.500 0.562 0.378 0.474 (IrrqaroveQualityofLife)23.IrnproveSpiritualLifeofFarmers 0.500 0.438 0.622 0.526 Sum 1.000 1.000 1.000 1.000 11ndicator (Local Priorities) (MairnainCommmitySmmRflSJncreasesCommmityVitality 0.594 0.625 0.560 0.591 G'mm‘cmmmtysumemmmmmmba‘m‘ 0.406 0.375 0.440 0.409 RmalandUrbanAreos Sum 1.000 1.000 1.000 1.000 (ProtectEnvironment)27.Preservanvironment 0.658 0.494 0.617 0.596 anirorrrnent128.quairEnvironrnentalenages 0.342 0.506 0.383 0.404 Sum 1.000 1.000 1.000 1.000 (MW/www‘l0‘“"““)30'“m’°Agm‘l“al 0.527 0.500 0.300 0.438 landscape (l\4airrtainAgriculunalEnvnomnent)31.PreserveRmalCommrmity 0.473 0.500 0.700 0.562 Sum 1.000 1.000 1.000 1.000 (Eclumte Environmental Protection) 32. Environmental Education of Farm Owners (Eduwe Environmental Protection) 33. Envimnental Eduwion of Tomists 0.633 0.562 0.750 0.655 0.367 0.438 0.250 0.345 Sum 1.000 1.000 1.000 1.000 91 Table 14. Recalculated weights (global priorities) of the evaluation elements by stakeholder group. (S: Scholars, F0: Farm Owners, PE: Policy Enforcers) S FO PE Total GI l lP . 'l' Goal( ) Mean r Mean r Mean r Mean r Eoommic 0.503 I 0236 3 0236 3 0331 2 Enjoymn 0263 2 0282 2 0.413 I 0.322 3 Ecology 0234 3 0.482 1 0351 2 0347 1 Sum 1.000 1.000 1.000 1.000 . . . .. S FO PE Total D1mensron (Global Priorities) Mean r Mean r Mean r Mean r (Eoornnic)A$istFannMamgement 0.077 8 0.040 8 0.032 9 0.050 10 (Eeornmic)qucateFarmers 0.055 9 0.031 10 0.067 6 0.053 9 (EomnnicflmpoveFanners'I—Jconomic 0.100 6 0.034 9 0.029 10 0.056 8 (Eoormric)UseFarmReswrcesWrsely 0.141 I 0.074 6 0.054 8 0.091 5 'c)MakeFarmingAttradive 0.130 3 0.057 7 0.055 7 0.083 7 (BjoymentflmproveQralityofIife 0.134 2 0.184 2 0.298 1 0.206 1 ' )MaintainCmrrmmitySmmne 0.129 4 0.099 5 0.115 3 0.115 4 log)Protectanironment 0.107 5 0.141 3 0.142 2 0.129 2 logy/)MairrtainAgriwlunaanvironmmt 0.047 10 0.138 4 0.094 5 0.090 6 (Eoology)EdrxateEnvirmnentalProtectim 0.080 7 0.202 I 0.114 4 0.127 3 Sum 1.000 1.000 1.000 1.000 (r denotes the rank according to its weight) 92 Table 14 (cont’d) (S: Scholars, F0: Farm Owners, PE: Policy Enforcers) . . .. S FO PE Total Indicator (Global Priorities) Mean Mean Mean Mean (Earmric)l.A$istMadteting 0.033 0.020 0.011 0.021 (FouunicflAssiaCooperation 0.014 0.008 0.012 0.010 (Eoornnic)3.AssistFarrnOperation 0.029 0.012 0.008 0.017 (Eoonomic)6.AssistFanners’InterpretativeAbility 0.027 0.006 0.017 0.018 (Eoommic)7.DeveIopFanners’Creativity 0.028 0.026 0.048 0.035 (Eoonrxnic)9.lmeaseFanners’IncomeSomoes 0.031 0.008 0.010 0.017 (Eoornnic)10.lrnerseFanrers’Inoome 0.069 0.026 0.018 0.038 (Ecumrric)ll.RevealAgriculmralUniqueress 0.098 0.063 0.034 0.065 (Eommnic)12.MaintainC1mentAgriwltme 0.043 0.011 0.019 0.025 (EwanicfldlrneaseTomists’Satislhctim 0.048 0.029 0.021 0.033 (Econmnic)17.PrormtetheIrmgeofleismeAgricultme 0.083 0.029 0.034 0.050 (Enjoymmt)22.lmrxovelnfimmneomealAm 0.047 0.105 0.093 0.080 (Fry'oyment)23.1mpoveSpirinnlefeofFanners 0.087 0.078 0.204 0.126 (Enjoyrrent)25.1measesCmnmmitertality 0.080 0.057 0.067 0.069 (Enjoymem)26.hnereeslrmeracdonbetwearRmalarrlUrbarrArers 0.049 0.041 0.056 0.049 (Ecology)27.PreserveEnviromrent 0.067 0.074 0.092 0.078 (EoologdZBanirEnvironmentalDamages 0.039 0.067 0.049 0.051 (Fooleg)30.PreserveAgfiarltmaIIarxlseape 0.025 0.077 0.028 0.041 (Eoobgl)31.PreserveRmalCamnunity 0.024 0.061 0.066 0.050 (Eoology)32EnvironmmtalEdrmtionofFannOMm 0.040 0.107 0.089 0.077 (Eoology)33.anirmmentaletmtionofl"ourists 0.039 0.095 0.024 0.050 Sum 1.000 1.000 1.000 1.000 93 The weights’ mean differences within the three stakeholder groups (scholars, farm owners, and policy enforcer) were also examined for statistical significance. Table 15 shows that the observed differences are statistically significant for only one goal (Ecology, F (2)=4.066, p=.048; X2(2)=7.346, p=.025) and one indicator (Increase Farmers’ Income Sources, X2(2)=6.615, p=.037) at the .05 level of significance. This indicates that scholars, farm owners, and policy enforcers on the 18-member advisory panel are in agreement in how they view the importance of each indicator. Table 15. Results of recalculated weights’ mean comparisons within the three stakeholder groups. One-Way Kruskal ANOVA Wallis Test F Sig. x2 srL conomic 3.002 0.091 4.897 0.086 njoyment 1.261 0.321 3.056 0.217 Ecology 4.066 0.048 7.346 0.025 (Economic) Assist Farm Management --- --- 1.003 0.606 (Economic) Educate Farmers 0.825 0.463 2.006 0.367 (Economic) Improve Farmers' Economic --- --- 4.043 0.132 (Economic) Use Farm Resources Wisely 1.128 0.358 1.003 0.606 (Economic) Make Farming Attractive 0.904 0.433 2.306 0.316 (Enjoyment) Improve Quality of Life --- --- 4.376 0.112 (Enjoyment) Maintain Community Structure 0.262 0.774 0.522 0.770 (Ecology) Protect Envirornnent 0.230 0.798 0.642 0.725 (Ecology) Maintain Agricultural Environment 3.114 0.085 5.773 0.056 Ecology) Educate Environmental Protection 1.839 0.205 3.105 0.212 (Note: Nonparametric test (Kruskal Wallis Test) was adopted if the variable did not meet the assumption of “normality” or “homogeneity-of-variance” when running the One-Way ANOVA test.) 94 Table 15 (cont’d) One-Way Kruskal ANOVA Wallis Test F Sig. x2 SAL Economic) 1. Assist Marketing -- —-- 1.182 0.554 (Economic) 2. Assist Cooperation 0.134 0.876 0.052 0.974 (Economic) 3. Assist Farm Operation 1.452 0.276 3.038 0.219 (Economic) 6. Assist Farmers’ Interpretative Ability --- --- 1.913 0.384 (Economic) 7. Develop Farmers’ Creativity 0.959 0.413 1.501 0.472 Economic) 9. Increase Farmers’ Income Sources --- -—- 6.615 0.037 (Economic) 10. Increase Farmers’ Income -- --- 3.036 0.219 Economic) 11. Reveal Agricultural Uniqueness --- --- 0.043 0.979 (Economic) 12. Maintain Current Agriculture 0.979 0.406 0.274 0.872 Economic) 16. Increase Tourists’ Satisfaction -- -- 1.140 0.565 (Economic) l7. Promote the Image of Leisure Agriculture -- --- 3.215 0.200 (Enjoyment) 22. Improve Infrastructure of Rural Areas 1.201 0.338 2.853 0.240 (Enjoyment) 23. Improve Spiritual Life of Farmers --- —-- 2.951 0.229 Enjoyment) 25. Increases Community Vitality 0.273 0.766 0.758 0.685 (Enjoyment) 26. ms Interaction between Rural and Urban 0.187 0.832 0.465 0793 Ecology) 27. Preserve Environment --- --- 0.463 0.793 (Ecology) 28. Repair Environmental Damages --- --- 1.560 0.458 (Ecology) 30. Preserve Agricultural Landscape --- --- 3.133 0.209 (Ecology) 31. Preserve Rural Community --- --- 4.456 0.108 (Ecology) 32. Environmental Education of Farm Owners 1.939 0.190 3.987 0.136 (Ecology) 33. Environmental Education of Tourists --- --- 4.250 0.119 (Note: Nonparametric test (Kruskal Wallis Test) was adopted if the variable did not meet the assumption of “normality” or “homogeneity-of-variance” when running the One-Way ANOVA test.) 95 Performance Score Results The total performance score was calculated by using Equation 1 as introduced on page 58. Results are presented in Table 20. The performance score for scholars is 6.106‘ (The probability is 0.95 that the interval 5.449 to 6.761 includes the true mean for the performance score) which is the highest score within the three stakeholder groups. The performance score for the farm owners is 5.782 (The probability is 0.95 that the interval 5.226 to 6.337 includes the true mean for the performance score) which is the lowest score within three groups. The performance score for the policy enforcers is 5.924 (The probability is 0.95 that the interval 5.759 to 6.090 includes the true mean for the performance score). The overall performance score for the three stakeholder groups combined is 5.916 (The probability is 0.95 that the interval 5.679 to 6.153 includes the true mean for the performance score). These results indicate that the stakeholders’ attitudes toward the performance of leisure agriculture policy are marginally positive. However, there are also clearly weakness in many dimensions of the policy and considerable room for improvement. , ¥ . Scale: lO=Superior; 0= Failing 96 Table 16. Accumulative weighted performance score by stakeholder group. Scholar FarmOwner Policyanoroer Total is Ws Ws-Sk—f Wf Wf-SYP l’Vp Wp-S Weighed/’5. 1.AssistMarketing 5.96 0.033 0.1981 5.15 0.020 0.100 5.62 0.011 0.063 0.074 .2.me 6.20 0.013 0.082 5.10 0.008 0.042 5.55 0.012 0.068 0.06 3.AssistFaerperatim 5.84 0.029 0.172 4.90 0.012 0.059 5.56 000810.046 0.053 4.AdjustTemperament 5.04 0.000 4.74 0.000 5.40 0.000 0.000] 5me 5.44 0.000 5.44 0.000 5.56 0.00 0.0001 6.AssistFarmers‘lntapretativeAbility 6.04 0.027 0.165 5.60 0.006 0.035 5.73 0.017 0.099 0.094 7. DevelopFarmers’Creativity 6.04 0.028' 0.168 5.15 0.026 0.133 5.58 0.049 0.276 0.253 .GrangeFarmers’Thinking 6.24 0.000 5.65 0.000 5.87 0.000 0.00 .InaerseFarmers’lnoomeSomees 6.36 0.031 0.197 5.89 0.008 0.04 5.54 0.010 0.058 0.063 10.11maseratmers’rnootm 6.44 0.069 0.442 5.90 0.026 0.152 5.55 0.018 0.100 0.123 ll.RevealAgriarlunalUniqueness 6.8810098 0.677 5.95 0.063 0.372 6.24 0.034 0.215 0.25 12.MaintainCtmmtAg1-iculurre 6.24 0.043 0.271 5.39 0.011 0.057 5.8810019 0.113 0.114 13.111118111912me 6.00 0.000 5.19 0.000 5.55 0.000 0.000] 14.ExpandTraditionalAgriculun'e 6.40 0.000 6.10 0.000 5.8 --- 0.000 0.0011 15.1tueaseNmnberorromists 7.081 0.000 6.52 0.000 6.23 0.000 0.000] l6.IncreaseTourists’ Satisfaction 6.40 004810306 6.0810029 0.1781 5.981 0.021 0.123 0.13 17.Prm10tethelmageofleisure 6.64 0.083 0.550 6.1810029 0.176 6.11 003410.210 0.22 (Economic) Subtotal 3. 22 78 1.352 1.369 1.458 l8.PreservetheQnrentCulune 5.84 0.000 5.65 0.000 6.06 0.000 0.000] l9.Edrx:atetheC1.rnentOlltme 5.64 0.000 5.56 0.000 6.00 0.000 0.0011 20.&mplyofRecrmtionalActivities 6.16 0.000 5.71 0.000 6.081 0.000 0.000] 21.3tpplyofReueationalLocatiom 6.08 0.000 4.79 0.000 5.76 --- 0.000 0.00 22.1mprovelnfimtrucmreomealAreas 5.6810047 0.266 5.85 0.10 0.617 5.89 0.093 0.550 0.544 23.1mrxoveSpiriuralLifeofFanners 5.13 0.087 0.445 5.63 0.078 0.441 5.41 0.205 1.111 0.99 24. ve ' ition Emawcm 5.92 0.000 5.56 0.000 5.30 0.000 0.000' 25.IncreasesCorm1unityVitality 6.36 0.080 0.511 5.84 0.057 0.334 5.96 006810.404 0.401 26.1m'emeslnteractimbetwearRmal g mumm 6.68 0.049 0.327 6.29 0.041 0.259 6.35 0.056 0.353 0.3401 rot/mew) Subtotal 1.550 1. 652 2.418 2.283 lyriesaveawiomm 5.52 0.067 0.371 5.94 0.074 0.437 6.10 0.093 0.567 0.542 EaRepairEnvirmmemalDarmges 5.44 0.039 0.214 5.92 0.067 0.397 6.00 0.050 0.299 0.307 .ReouceN ' of Dew] mm 4.92 --- 0.000 5.281 --- 0.000 5.50 --- 0.000 0.0001 0.PreserveAgriculnlalLarxiscape 6.36 0.025 0.156 5.97 0.077 0.461 6.11 0.028 0.169 0.203 l.PreserveRuralCorrrmmity 5.8410023 0.134 5.65 0.061 0.342 6.20 0.067 0.417 0.394 2'Wmm0me 5.84 0.040 0.235 6.02 0.10 0.652 6.05 0.090 0.545 0.542 3. E‘nvirmmerrtalEdrmtionofTomists 5.5 0.039 0.218 5.19 0.095 0.490 5.82 002410141 0.187 (Eco/om Subtotal 1.329 2.778 2.137 2175 (Scale: 10=Sweriorz0=Failing1 6.106 5.782 5.924 5.916 95% CI: (5.4496761) (5.2266337) (5.759~6.090) (5.679~6.153 97 CHAPTER 5 CONCLUSIONS This chapter is divided into three parts. The first part presents a summary of study results. The second part contains conclusions and their implications. The last part includes recommendations for future research. Summary of Study Results The main purpose of this research was to develop and apply a systematic process for collecting and analyzing information from stakeholders that were needed to evaluate the performance of Taiwan’s leisure agriculture development policy. The following three specific questions were addressed. What are the goals of leisure agriculture development in Taiwan? What are the relative priorities of these goals? And, how effective do stakeholders think the Council of Agriculture has been in developing leisure agriculture? In this first section, findings pertaining to these-three research questions are summarized and discussed. 1. What are the goals of leisure agriculture development in Taiwan? An 18-member advisory panel, including scholars, farm owners, and policy enforcers, were interviewed in order to identify the goals of leisure agriculture development in Taiwan. The panelists identified 33 relevant performance indicators 98 including: seventeen economic indicators, nine enjoyment indicators, and seven ecology indicators. Confirmatory Factor Analysis (CF A) was used to refine the content validity of the assessment instrument. The refined performance evaluation instrument included 21 performance indicators including: eleven economic indicators, four enjoyment indicators, and six ecology indicators. 2. What are the relative priorities of these goals? The Analytic Hierarchy Process (AHP) was used to determine the priorities of the evaluation criteria. The ranking by broad goal weights is as follows: ecology (.347), economic (.331), and enjoyment (.322). Thus, these three broad goals are considered to be almost equally important. The five most heavily weighted/important performance indicators and their related broad goals are: Related Broad Goal Performance Indicator Enjoyment Improve Spiritual Life of Farmers (weight= 0.126) Enjoyment Improve Infrastructure of Rural Areas (weight= 0.080) Ecology Preserve Environment (weight= 0.078) Ecology Environmental Education of Farm Owners (weight= 0.077) Enjoyment Increase Community Vitality (weight= 0.069) The weighted mean differences for the indicators across the scholars, farm owners, and policy enforcers were also analyzed for statistical significance. The results 99 indicate that all three groups of participants were quite consistent in how they viewed the importance of each indicator. 3. How effective do stakeholders think the Council of Agriculture has been in developing leisure agriculture? The overall performance score assigned by scholars is 6.106., which is the highest score within the three stakeholder groups. The lowest performance score 5.782 was assigned by the farm owners. The performance score assigned by policy enforcers is 5.924. The overall performance score from the three groups is 5.916. These results indicate that stakeholders’ attitudes toward the performance of leisure agriculture policy were marginally positive, but there is clearly considerable room to make improvements in the program. For the broad economic goal, the Council of Agriculture should focus more on assisting farm management, continuing farmers’ education, and improving farmers’ economic well-being. For the broad enjoyment goal, the Council of Agriculture should concentrate on improving quality of life. For the broad ecology goal, the Council of Agriculture should focus more on environmental protection. . Scale: lO=Superior; 0= Failing 100 1. Limitations of the Study The Council of Agriculture provided the list of farm owners and scholars for this study. However, there is no guarantee that these two lists will include all the farm owners and scholars who were in the leisure agriculture enterprise in Taiwan. Non-response bias might influence results. Especially, the response rate of the farm owners was only 34%. However, the confidential nature of the survey made it impossible to track or analyze differences between respondents and non- respondents. The AHP needs to do the pair-wise comparisons of the indicators. This is a very complex questionnaire and the respondents need patience and attention to complete the survey. It was very likely to get the inconsistency responses, if the respondents did not pay full attention to answer the questions. Moreover, the long questionnaire would also cause a low response rate. Therefore, this study just asked the members in advisory panel to answer the AHP questionnaire. Although the members in advisory panel are experts in the leisure agriculture filed and their opinions are very valuable, this may still limit the generalizability of research findings. 101 Conclusions and Implications This research has demonstrated a systematic process for collecting and analyzing information about the stakeholders’ perceptions of leisure agriculture development in Taiwan. Results provide insight to the Council of Agriculture concerning how well the current policy is perceived to be working; and focusing on the detailed results reveals information potentially useful in guiding future policy development. From a macro view, the overall performance score (5.916) indicates that stakeholders believe that the current policy for leisure agriculture development is only marginally positive and they were definitely not satisfied with specific outcomes of the policy. Moreover, scholars, farm owners, and policy enforcers all hold very similar attitudes about program performance. From a micro view, one can check the performance assessment of each indicator. No indicator’s assessment was higher than seven, which might be deemed fair to good performance; and several were assigned values less than five which might be deemed to equal to poor performance. Generally, the three groups’ assessments for most of the criteria were quite close; although statistical tests showed that some scholars’ score were significantly higher, especially for some economic indicators. This means the Council of 102 Agriculture needs to work on improving policy performance across most of the indicators identified in this study as being important. Finally, scholars emphasized economic goals; farm owners and the policy enforcers focused more on ecology and enjoyment goals. Based on the results of this study, the following recommendations should be considered by the Council of Agriculture: 1. From the priority analysis, the outcomes show the importance weightings for the broad economic (iii), enjoyment ($76), and ecology (53%,) goals are essentially equal. That means that future leisure agriculture development should focus equally on these goals. 2. The performance scores, especially the economic indicators, show dissatisfaction with the council’s performance. This indicates that the Council of Agriculture still needs to work more on assisting farmers to operate this service type of business (e. g. identifying the potential market/customer, providing educational program for farmers to learn how to operate a leisure agriculture farm, improve service quality.) 103 Recommendations for Future Research This study was the first to evaluate the Taiwanese government’s performance with respect to leisure agriculture development policy. As with any first effort, there is room for improvement. Some recommendations for future research on this topic follow. First, this research confirmed that there are three major goals (Economic, Enjoyment, and Ecology) for evaluating leisure agriculture in Taiwan. If the Council of Agriculture wants to monitor performance annually, the number of indicators tracked could reduce in order to shorten the time required to complete the evaluation questionnaire and thereby increase response rate. Moreover, if a shorter instrument were developed, it would result in a shorter AHP questionnaire. Therefore, a more advanced analysis of the instrument is suggested. The dimensions under each broad goal built in this research would be a good place to start to develop a briefer instrument. Second, the focus of this study was on measuring the satisfaction of stakeholders who were most directly impacted by leisure agriculture policy. However, other groups such as consumers are impacted as well. Future studies could include inputs from a broader range of stakeholders. 104 Third, future studies could try to evaluate the efficiency of this policy to determine if what the government has been doing is being accomplished at an acceptable cost. Cost-benefit analysis and economic impact analysis are the techniques that could be used for future studies. Fourth, this study used a panel of content experts to design the assessment instrument. Subjective judgments are involved in deciding what each content area measures and what items should be used for that content. Although this approach takes a lot of effort to implement this work, it does help to verify the criteria for evaluation. Fifth, this study used Confirmatory Factor Analysis to assess content validity and content equivalence in terms of item-content structures and content area constructs. This approach offers an alternative way to assess content validity and support the findings from the qualitative research element. A combination of qualitative and quantitative approach is suggested if future research was going to develop a new instrument. Finally, Analytic Hierarchy Process decomposes the complex evaluation criteria into smaller sub-dimension that can be better managed in terms of scaling, weighting, and combining the scores obtained from each criterion. This allows the policy decision- maker or policy evaluator to satisfactorily aggregate each of the various attributes into a 105 single measure of overall performance score. The AHP can be a valuable tool for further applications in policy evaluation. 106 APPENDICES 107 APPENDIX I: Consent Form I Leisure Agriculture Performance Evaluation Survey Date Name Address City, State Zip Dear , The purpose of this letter/phone call is to ask for your participation in my study “Analytic Hierarchy Process (AHP): A Method of Quantifying the Performance Indicators of a Tourism-Based Industry”. This study is being conducted to evaluate the performance of leisure agriculture development in Taiwan. You are being asked to participate in this study because of your expertise and experience regarding leisure agriculture development. If you agree to participate, you will be contacted twice: First for a personal Interview This will be a one-hour personal interview (the date and time of the meeting will be arranged). The topics will cover: (1) what are the objectives of leisure agriculture development? (2) what is/are the performance indicator(s) of each objective (3) What is the priority of the objectives? F ollow-up survey by mail order You will be mailed the questionnaire survey and be asked to assess the importance of a set of performance indicators. It will take about 30 minutes to complete this mail survey. All your data gathered during this study will be treated with strict confidence. Your confidentiality will be protected to maximum extent allowable by law, and to ensure confidentiality, your identity will only be known to my advising professor and me. Any reports of research findings that result from this study will not associate your identity with‘specific responses. Participation in this study is voluntary. You may choose not to participate at all, may refuse to answer certain questions, or may discontinue participation at any time. 108 If you have any questions or concerns regarding your participation in this study, please contact: Sheng-Jung 0n Hung-Hsu Yen Donald F. Holecek 250 Kuo-Kuang Rd. 250 Kuo-Kuang Rd. 172 Natural Resources Department of Horticulture Department of Horticulture Building National Chung-Hsing National Chung-Hsing East Lansing, MI, 48824 University University PHONE: 002-1-5 1 7- PHONE: (04)22850395 PHONE: (04)22850395 3530793 E-Mail: Sjou@nchu.edu.tw E-Mail: yenhungh@msu.edu E-Mail: dholecek@msu.edu If you have any questions about your right as a human subject of research, please contact: University Committee on Research Involving Human Subjects Ashir Kumar, MD, Chair 202 Olds Hall, Michigan State University East Lansing, MI 48824-1046 PHONE: 002-1-517-3552180 FAX: 002-1-517-4324503 E-Mail: UCRIHS@msu.edu If you freely consent to participate, please sign below and mail this entire document to me in the envelope provide. Signature Date Thank you, Sheng-Jung 0u Department of Horticulture, National Chung-Hsing University Hung-Hsu Yen Michigan State University 109 APPENDIX II: Consent Form II and Performance Assessment Questionnaire Leisure Agriculture Performance Evaluation Survey Date Name Address City, State Zip Dear , You have received this survey because you have been identified as an expert, who can provide valuable information for my study “Analytic Hierarchy Process (AHP): A Method of Quantifying the Performance Indicators of a Tourism-Based Industry”. My study is being conducted to evaluate the performance of leisure agriculture development in Taiwan. Over the past two months, experts form Council of Agriculture, universities/colleges, local governments, farmers’ associations, farmers have been interviewed to identified the performance indicators of leisure agriculture development. These performance indicators have been incorporated into the attached survey and distributed to a broader group of experts from the stakeholder groups, of which you are apart. I would appreciate your taking the next 15 minutes to complete the attached questionnaire. Participation in this study is voluntary. You may choose not to participate at all, may refuse to answer certain questions, or may discontinue participation at any time. You indicate your voluntary agreement to participate by completing and returning the questionnaire. The survey is anonymity. It means that no one is able to associate responses or other data with individual subjects. Moreover, only aggregate data will be shown in the reports. All your data gathered during my study will be treated with strict confidence and your'right TO PRIVACY will be protected to maximum extent allowable by law. 110 If you have any questions or concerns regarding your participation in this study, please contact: Sheng-Jung 0u Hung-Hsu Yen Donald F. Holecek 250 Kuo-Kuang Rd. 250 Kuo-Kuang Rd. 172 Natural Resources Department of Horticulture Department of Horticulture Building National Chung-Hsing National Chung-Hsing East Lansing, MI, 48824 University University PHONE: 002-1-5 l 7- PHONE: (04)22850395 PHONE: (04)22850395 3530793 E-Mail: Sjou@nchu.edu.tw E-Mail: E-Mail: yenhungh@msu.edu dholecek@msu.edu If you have any questions about your right as a human subject of research, please contact: University Committee on Research Involving Human Subjects Ashir Kumar, MD, Chair 202 Olds Hall, Michigan State University East Lansing, MI 48824-1046 PHONE: 002-1-517-3552180 FAX: 002-1-517-4324503 E-Mail: UCRIHS@msu.edu Your assistance in this research is very much appreciated. Sheng-Jung 0u Department of Horticulture, National Chung-Hsing University Hung-Hsu Yen Michigan State University 111 l Leisure Agriculture Performance Evaluation Survey Using a scale of 0 to 10, please evaluate the government’s leisure agriculture policy. (10=”Superior 01' Outstanding”; 6=“Passing”, 0=”Failing or Unacceptable”) Performance Evaluation Evaluation Indicators Grade '5; a m 0 10 1. How well do you thrnk the government “Assrsts © 0) ® ® ® 6) © C7) © ® Marketing ? 2.01:1;3-22133’XPOU think the government Assrsts © (D ® (3) © ® 6) ® (9 ® 3. How well do you thrnkthe gqvernment prov1des © 0) ® ® @ G) © (7) (9 ® Farm Operation Assrstant ? 4. How well do you thrnk the government “Adjust © CD ® ® ® @ © Q) Q) ® Temperament ? 5.“How well do you think thigovemment © ® ® ® @ (9 © ® © ® Increases Receptiveness ? 6. How well do you think the government “Assists Farmers’ Interpretative Ability”? © ® ® ® @ © © ® © ® 7.“How well do you think the government © 0) ® ® (4) © © ® ® ® Develops Farmers Creatrvrty ? 8. How well do you think the government “Changes Farmers’ Thinking”? © CD ® ® ® 6) © ® C9) ® 9.3{0w well do you thrpk the government, © (D ® ® ® 6') © (3 ® ® Increases Farmers Income Sources ? 10. How well do you think the government © CD (3 ® @ (5) © ® © ® “Increases Farmers’ Income ”? 112 Evaluation Indicators Grade 9’ 0 10 .99" 9 9 9 9 9 9 9 9 9 9 1290211.311:‘8uy3‘éi’1‘223‘fuiftféf38’m 9 9 9 9 9 9 9 9 9 9 1395.21.8188:2218.13.99“: 9 9 9 9 9 9 9 9 9 9 159113311223 filfiflikrtffrififwmt 9 9 9 9 9 9 9 9 9 9 16‘.‘Ir}rlcor‘:a‘:eesll'IsirougslistPSrhlliglicgtidfigmem © ® ® ® ® 6) © ® ® ® 17‘211’1rbfir1‘fi22 firoeficihlatlglbnfftlloiigz‘grxrgfiihlture”? © ® ® ® @ © © ® @ ® “9...“: 9 9 9 9 9 9 9 9 9 9 20598851121881.3338.33%?5112‘3353’ 9 9 9 9 9 9 9 9 9 9 2’ 93.831128231832881.3118118388 9 9 9 9 9 9 9 9 9 9 22953313123ifiififiifii’é’fi‘fififififiae 9 9 9 9 9 9 9 9 9 9 33.131.11‘2’112h51g8m35 9 9 9 9 9 9 9 9 9 9 “km“: 9 9 9 9 9 99 9 9 9 25. How well do you think the government © CD ® ® (4) 6) © 9) ® ® “Increases Community Vitality”? 113 Evaluation Indicators Grade Li? 9 m 0 10 26. How well do you think the government “Increases Interaction between Rural and © 0) ® ® @ 6) (6) ® (9) 00 Urban Areas”? 27‘.‘ How well do you think the government © 0) ® ® @ C5) @ ® @ ® Preserves Envrronment ? 28‘.‘How.well dooyou think the government © ® ® (3) @ © (19 ® ® ® Repairs Envrronmental Damages ? 29. How well do you think the government “Reduces Negative Impacts of Development”? © ® ® ® @ ® © ® ® ® 30. How well do you think the government “Preserves Agricultural Landscape”? © 0) ® ® @ © © ® © ® 31. Howwell do you think the government © (D ® ® ® (9 © ® (9 ® “Preserves Rural Community”? 32. How well do you think the government promotes “Environmental Education of Farm © ® ® ® ® 6) © ® © ® 33. How well do you think the government encourages “Environmental Education of © CD ® ® @ © © ® © ® A Few Questions to Help Us Classify Your Answers 1.Your Age is El Below 25 years old Cl 26~35 years old 13 36~45 years old El 46~55 years old Cl 56~65 years old El Above 65 years old 2.Where is your farm located? ElNorth: Ilan County, Keelung City ~ Taipei County(City), Taoyuan County(City), Hsinchu County(City) ClCentral: Miaoli County, Taichung County(City), Naitou County(City) ~ Changhua County(City), Yunlin County(City) ClSouth: Chiyi County(City), Tainan County(City), Kaohsiung County(City), Pingtung County(City) ElEast: Tatung County(City), Hwalien County(City) --~~--~~~~~ Thank you for your participation! ~~~~~~~~~~~~~~~ 114 APPENDIX III: Consent Form III and AHP Questionnaire Leisure Agriculture Performance Evaluation Survey Date Name Address City, State Zip Dear , You have received this survey because you have shown your willing to participate my study “Analytic Hierarchy Process (AHP): A Method of Quantifying the Performance Indicators of a Tourism-Based Industry”. My study is being conducted to evaluate the performance of leisure agriculture development in Taiwan. The purpose of my survey is to get the relative importance you place on each performance indicator when evaluating different performance indicators. Once the data collected, the information will be analyzed to identify the priority and weight of each indicator. I would appreciate your taking the 30 minutes to complete the attached questionnaire. Participation in this study is voluntary. You may choose not to participate at all, may refuse to answer certain questions, or may discontinue participation at any time. You indicate your voluntary agreement to participate by completing and returning the questionnaire. The survey is anonymous. It means that no one is able to associate responses or other data with individual subjects. Moreover, only aggregate data will be shown in the reports. All your data gathered during my study will be treated with strict confidence and your right TO PRIVACY will be protected to maximum extent allowable by law. 115 If you have any questions or concerns regarding your participation in this study, please contact: Sheng-Jung Ou Hung-Hsu Yen Donald F. Holecek 250 Kuo-Kuang Rd. 250 Kuo-Kuang Rd. 172 Natural Resources Department of Horticulture Department of Building National Chung-Hsing Horticulture East Lansing, MI, 48824 University National Chung-Hsing PHONE: 002-1-5 17- PHONE: (04)22850395 :rsity 3530793 E-Mail: Sjou@nchu.edu.tw PHONE: (04)22850395 E-Mail: E-Mail: dholecek@msu.edu yenhungh@msu.edu If you have any questions about your right as a human subject of research, please contact: University Committee on Research Involving Human Subjects Ashir Kumar, MD, Chair 202 Olds Hall, Michigan State University East Lansing, MI 48824-1046 PHONE: 002-1-517-3552180 FAX: 002-1-517—4324503 E-Mail: UCRIHS@msu.edu Your assistance in this research is very much appreciated. Sheng-Jung Ou Department of Horticulture, National Chung-Hsing University Hung-Hsu Yen Michigan State University 116 Aaom8oooom: mo 00% 05 o. .0028 020 .00. 05 ..o 0 05 03.8 3.5.... so» 005 220848.80: 00.8 00:888.... 0.9.0.00... \o a... ..0..Eo=oom.. 01.3. no» 2.0 n....08.no.8m: 9.0 ..0_8o..o0m: 03:80.0 8.. 08$ 0.888 o. so.» .8 02.0.2.0 o3. 0.0 0.05 8055 888888 0051...... 0 Mo 08880 8. mm 8.323.. 05. BEESQ @0029. 08608.28 .808 w 2.0 c .v .N .0052... .02. 8080—0 0.... no 0083.898. 080.5..m "a £050.... .95 8080—0 08 .«o 0085.895 mocha 80> no 8050.... 00.3 8080—0 0.... mo 0083.898: wuohum "m 8050.... ..0>o 8080.0 0.8 00 00808098. 0.9.0602 um ”88080—0 52. .«o 00.8.0098 8:37.. $0800 0005 .5 8.8.8888 00.31.80 00 852.2 mm .33 .3 828% Box 080008 81.00 0:... $03.... 03.. 8 00.000 0:. movie... .3 0.8.08 mo $00 moot? mo 0083.38 05 0.3800 0803 $80.68 .05 no 0.00%.. 26...? mo 0053.898: 02.20.. 05 0.0290 o. no» 005 2.53 _ £080 02. 5 0.33050 0.35 830 so.» o. 000:8 a £088.... .23 germanium .0830 8.0.,— D 82.00830. .8088“. D .8809. 80880.60 D b.8383 D 0.8 0.00:8 8.20s so.» 880082808080 £023 o. 0.0068 00.05 50?...» 3... 0.0.9800 o. 083 0... mafia. .8 .8.» 0.80.; _ RSV 0002....— »Aoufifll 03.5.0.2 0.: was—33 0.85m notes—03— 0oaaeot0m 005_=0_..u< 00.501— _ 117 ooueuodwl awanxg aoueuoduq Suons £19 A aoueuodml fluons aoueuoduq mmapow aouuuoduil [nnbg aoueuodull aimOpow aoueuodwl Suons aoueuoduq fluons MaA aouwodui] owanxg V Economic©®®©®©O®L®®®©©©®®©Enjoyment Similarly, if you think “Enjoyment” is of moderate importance over “Economic, ” then you should mark (3) on the right side, as shown in the following table: aoueuoduu owanxg aouuuoduq Suons £19 A aouauoduq fiuons aommodui] amapow aouwodm] lenba aouwodui] owopow aouuuoduq Suons ooummduq Buons £19 A aouuuoduil amanxg V Economic©®®©®©®®®®.@®©®®©Enjoyment 118 289.com ® aaucuodwl Suons aou'euodui] lenbg aoueuoduu Buons @ @ k 4 3 X n 9 w 9 w d 0 U D. u 3 9 aoueuodwl Buons [(19 A aoueuoduu SIUJQPOW aoueuodw] amiapow aaueuodwl Suons [(19 A saueuoduu awanxg V 0.80.8— 000... 00.0.: .8 09.8.0. 0... 02.05.". 0.33030 0.80.0— .8 Row 380888380 .05 0. £033 .QoNoom o 0.833%... 08.82 .8 How 308 0:. 0. £023 .80588m o 0.88030 0......2 .8 Row 389800 05 fl :02? 6,8080% o HEHmOm—E HwO—z 0... 0.8.08 :7. m1. .3 .48» 0.5.8.0?“ 00.50. 00...: 000... .8 0083.898. 0.: 1...... 00,003.». "£8” 0.3—50.0w“ 080.0. 00...: .8 831.2380 0.5.. 119 0.8.1.050 0822 .8 0w082 0:. 080 .82.00..2.00 20:. .0.o22> ..o 8:88 0:. 0000.08. :23 :2:3 00208.89 MFEER 0.33. 0 808.0022 .08.... o. .080 009. 000808: 0808 o. 0.08.0. 22.5 080 .2000 o. £002.55 8.8.8. .30..0>..Q 9 82.08.88 3.8.8030 820.8208 .080 02.020.00.05 .08—82.80 .008. o. 2 :2:3 500...: 000.88% £08k 003 o .0888 880 02288888 8.. 0.08.0. 0000.08 o. .008 2 25080.2 . 0.088% 08.85 o 82.0.8.8 080800.80 880.8080 080 025080.08 20:. 928.88. 5.2.00.0 00 :080 000.0 2 0.0.8.32 0.8.0th 0 0.0800,. .0:.o :23 0.0.3280 o. 80:. 0w0.8o080 o. .080 3:50 2.2.0.080 .80 82.02.08 20:. 0000.02 o. 0.0.8.0.. 922% 2.0 9.2.2000 2 :2:3 808.0850: 5.8% .200“. o $258.2. .8: 20 2008 L... 02. .3 280 08880:... 8282:... x... 3.20 .o 85:88. 20 .5. 38...... :0ow ..28o8oom: mo 08208082.. 20 .8 82.0882. 0:... 120 aoueuoduil awenxg aoueuodui] Euons £19 A aaueuodw] fiuons aoueuoduu azuepow aoueuoduil [12an aoueuodwl azeJapow of six dimensions of “Economic” aoueuoduil fiuons aoueuodlul 8110113 ,0; A aoueuodui] swanxg the relative Please 0 0.0 121 8.0“. 8.0.: .200< .2090. A aoueuodui] awanxa aaueuodw] fiuous £19 A aoueuoduil Suous soueuoduu GIBJQPOW aoueuoduil [enbg aoueuoduxl amiapow aaueuodluI fiuoug aaueuoduu Euons £19 A aoueuodw] awanxg V 82080820 8.0m .200»... (8 08.02.02 00.:. mo 02.20. 0:. 00005... 00020:: 20:. .0 82.0.08 0:. 2 0.08.0. 0.2000 22:3 0.28.006 58k .200... o 0220.62 2.0 00000205: .080 :23 0.08:0:00 0. 088.0,. 00880080 :22? 62.8.0820 200.. o 0.02.08 20:. w2u20>20 2 088.0,. 02% 2.0 .2000 o. 2 :2:3 $2.028: .202.V o 72,550.22 .502 0:. 08008 a... mi. .3 8208082. :8080w0q02 8.0.: 200.? .0 08.0202 00.2. 000:. .8 0080.892 0:. 2:0. 0000i 0. 82080820 :8080w0802 8.0.”: 200...: .8 08.0222 00.:. .8 82.8800 0:... 122 0.0.0.030 0.022 $20.00.. .0 002.0082. 20:. m0w00:0 :2:3 $03235 .20500k 0M§0U o 0.0.60.0 00.0.0.-_0..E02..w0 @2802: o. 0.0:. 020: :2:3 5.5.0000U . 0.0.0.5.... m0~0>0Q 0 002.030. 000 022:2. 000:. 0.0.80.0. 000 0000.22.00 .0.00_ 0. 0.0000. 020: :2:3 REBV 02.80.325 0.05.80. 300V 9 0.0.3030 0.022 .0 02.030. 000 022—00 0:. 2.0.0.0000 0.0.0.0.. 0:. 020: :2:3 .0000023000... 0000.00~ o 0:20 000 0000...... 20:. .00 _00 0. 0.0.0.0.. m0w005000 :2:3 ..=050..0QE0.~ .03....0V o HEHKOE— PmO—Z 0:. 00000. :7. m2. a: 020,000.... :E0E0m 0.00.59. .«0 0.30202 02... 000:. .0 0000.02.02 0:. 200.. 00005.". "020005,... £00050.“ 0.0003—8 ..0 0.30202 0.»... .0. 000209000 23.. 123 aoueuoduu awanxg aaueuodwl Buons be A aaueuodw] Buons aoueuodwl amapow aoueuodm] [enbg aoueuoduu awnpow "020000.... $000.00.. 0.0003... .0 0.0.00.0... 0>.u .0 sou‘euodtu] Suong aauvuoduq Buons [Us A aaueuoduq aulwxg 00.00.00 0:. 0.00... .v 124 00.000. 0000600.». 0.0000. .a0gnm 030.05 @ © 6 9 6 ® a a e 6 6 Q 6 g G 9 @ .mu0gnm 080.05 A w , , ¢ 3 A S W 3 W S A H x n. b n. x a 0 m m. 0 m 0 0 a w aw d I .m I d u w d I o w W m mu. 0”. d 0 w W do. u .w W m 0 m 0 a m 0 m o 0 m 0 9 9 3 a m 0 0 m a m a 02000.0:— :0.E:::0.r._ 3.0.0.0... 0.5.0.0.: 0: 0.0.00.0:— 0.5% "02000020 :2800020 0.0.0.0..— 05.00.03 .0 0.0.0200. 0B. .0 0008.000... 02.0.0. 0:. 0.00050 0000—0... 0.0.32.9. 0.00.0. 0. 00.000. a0.0.0.0... 0000.00. :2:3 0500.0 . 0.0.0.0.”. 0000...: 0 0.0.0.0.. .0. 000.03.00.00 :0. 0:. 00000.00. :02? .000.0000- 00.005 . 0.08.0.0 0000.03 0 guaca— HmO—z 0:. 00000. a... N 000 . .0: 00.000020 02.000020 .2050... 0.00.00.—3 ..0 0.0.0200. 03. 000:. .0 00009300.. 0:. 0.00.. 0000.0... ”00.000020 ..20.00020 .2050...— 0>0..00.—: 00 0.0.0200. 0B. .0 000200.000 0:.—.0 125 r 0.3.02.9. 000000.203 A , v 3 M S W W S .M 3 a m m. m. m m n m m m m m m m m m m m on. w m. .m w. .m aw w m w m 0 m 0 m m 0 w m 0 m 0 m 0 m m 00.0005... ..>..00.>/ 000.0000: 0...... 00. .: .0 0.0.0200. 0.5.; "00.000020 00.00...» 000.0003. 8.0.. 00 D: .0 0.0.00.00. 0B. .0 0008.000... 00.050. 0... 0.00050 0002...». 0000.000 w0.0..0. 92.0.00 0... 00.80.00. 00...? a0.0.0.0...va 0.0.0.0 0.0.50... 0 .8000, 0.3 .0 02.000.00.000 0... 000 w0.0..0. .0 000.0000. 0.00 0... .00>0. 0. .0002000...5 3.5.02.3. 3000.. o .5200... .002 20 88:. a... N 000 . .3 00.0000..0 00.00.? 000.0003. 8.0.. 003.. .0 0.0.0200. 0B. 000... .0 00002000.. 0... 0.00. 0000...... "00.000020 a000...? 000.0003. 5.0... 009.. .0 0.0.0200. 0B. .0 000200080 0...... 126 000.00% . 0.0000030. 0800... 209.0 0 0 0 0 0 0 0 0 e 0 0 0 0 0 0 0 0 3.20050 A v a A S a w S A a m m w .m m m 0 m m S 0U. m H m. 3 S w m m w m... m 0. m. m m w m m m m 0 W A. m 0 0 m 0 m 0 .0 m 9 m M” 9 m 9 00.0030... $000.00.. 0...... 0.5.0.2.... .0 0.0.0200. 05,0 "02000.0.0 0000200.. 0.00 0.0.0.2.... .0 0.0.0200. 0.3. .0 0050.000... 00.050. 0... 0.00.000 0000.0... 0000.000 0.8.0200 0.00.0. 0... 0. 0w00...0 0. 0.00.0. 0000.200... 00.0.. 00...? 0.0.32.3. 0000.00... 0000.6 . 0000.000 0.3.0050 0.00.0. 0... 0. 0.0000000... 30>. .000... . EHmOE. hmOS. 0... 00000. a... N 000 . .3 00.0000..0 00000.00». 0:0,. 0.0.0.2.... .0 0.0.0200. 0B. 000... .0 308.000.. 0... 0.00. 0000...... "020000.... 0000200.. 0:0,... 0.0.0.2.... .0 0.0.0200. 0.3. .0 002.0008... 0...... 127 0.00000020000000000000000006000.5088... §0<203 .005. .00....00089000006000000000000.02052883 0000.003 0.0.000... .0005. .90500000000000000060003052882 A v 3 M S W 3 W S M 3 w... m m m m m m m m S H S m : w. m. 0 W 0 m m w on. w w m w .m on d m w m .m m .w W m m. o m 0 m . m 0 0 m 9 m a a m 9 00.0020... ..0>..00...< 0:.—0.0... 00.02.. .0 0.0.0200. 00...; .00 00.000020 :0>..00...< M0050... 020.)... .0 0.0.0200. 00.... .0 00002000... 00.00.00 0... 0.00800 0002...». .0.0..00...MV 0.00.0.5? 00.00.. 0... 0.00.0.0 9 0002.000... 0.3.02.9. 0.00.0. 20.... 5.3 00...00\0.00m. . 0.0.2000 0,000.00. 9 0.0.0500 \0 000.02. 0000.00. 0 Z<...~.O...>.. HmOS. 0... 00000. x... ml. .3 02000.0.0 002.02.... w0.0..0.. 00.0.)... .0 0.0.0200. 00.... 000... .0 0003.008. 0... 0.00. 0002...". "00.000020 :0>..00...< 0:.-0.0.... 00.0.)... .0 0.0.0200. 00.... .0 002.000.00 0...... 128 .EoEQBoSu Rh: 98 533 8833 BE £35888 3:: 8:832 $025588 3:: E couflsaom 2.: .«o wfiwm Egan 9 6.583% \SFSEEGD 58:8: 0 .530; 3:08 Enigma 85:5 28 “5:80:28 Rumba 05 32:95 £023 .mxi \o 538$ 98.32: 0 .300: P0303 $08 3 83383 353388 05 She mm :03? .mES.S>V $2388me USN M233: 9 .8333“? 38235 :88 93 2680.5 8 mm :02? .2830 Ngohmnufi 53mm 0 Punch—E HMO—z 2: 2.38 :7. v2“ 3 12% .znofimomams .3 32255:. .58 3.2: .3 353.395 2: :5: 835.“. "_acw canon—manna: no 33258:. .58 no near—«9:90 2;. 129 2308i £5 bmasfifioo 58:82 @ Q 8 @ ® © ® ® 6 © @ © ® © 8 G @ mo mam—«=0 26am: obsosbm QBSR>< moEndtommO b82550 £8522 @ 0 ® © @ © © @ 6 ® ® © © © ® 0 @ _mcoumouoom mafia; 0528>< £5 @ 0 6 © @ @ mu @ 6 @ @ ® @ © 8 0 @ 825030 We bzmso 3895 60% wcfimE anagram 8830 3:58:50 58:82 @ 0 ® © @ ® ® @ G @ 6 © @ © 8 O @ HecEvfiH £89m £5 8330 we 93:0 3295 @ Q G © @ © @ ® 6 @ © © @ © 6 Q @ #2539“; £80m DEER>< moEEEOQQO 05:30 1823803“ mafia: @ Q G © ® @ ® ® 6 © ® @ ® © ® 6 @ 3:036“; 530% mm M S w 3 w S M 3 m ,M m m m m m a m w m an m m m... an. m m I m w 9 d 9 w m I. m a w m o w w s m u w m m m m m m o M w m a m m w m a m x m m a 1:33 ..:3_:>.:_.=V._:.32:21:25: .55— r 34an aunofihomnms no 22238:. .58 a: 3:83.95. 3533 2: 0.39:8 333.". 130 05:50 2330 5856 2: 883m @ Q 6 © @ ® ® ® 9 ® ® ® @ © ® 6 @ :5de 2t 0338.» 3 A S W a W S A 3 m w w m am m. w m w m a w m o m w aw m 0 l .m m .0 II. M u m m o o m .m m m m a m a m a m a m a a w 9 9 :3:S::: iii—:9 _.:_:_.:.=..,_. 5:32.32. 7:33:5— ::,_. 32255:. 29.3.50 132:3: H 533—: no E8355 9.5 no 3§ta35 $553 2: 0.39:3 anaer— .u. .ohfisocwm mo 26g 05 Sons aoufioaow Bo: 83:8 8 mm :oEB .3539 33x39 2: 83sz o .8830 18332“? E056 08 otomoa 8 mm :03? .3525 $3.30 2: wiuwmgm o HEHMOEH hmOE 2: 2..on a? N Una fl .3 noise—Eu sets—=0 _ancEvEH 559m: .3 9.33%.: 95 anon. no 3523:: 2: :52 0235.”. Being—E REE-=0 13355:. £33? .3 Easing 95 a: nomtanfioo 2F. 131 c383 1 meagre/x 385803— bqum @ @ Raotwouoom bqnsm A v w a A S W W S A a m m m m. m w m m m m S aw m H m cm S m 9 n I 19 w 19 I n 9 I m w H .m H m m I m s .m m m w s m o m m m m w m m. w m m a m a m M... w m a m m a m a m m :Em—S—EC ..o_._:__=>< 13:25.:EQC .uoz act—:2: t. 3:25:5— :.$,_. unease—Ev ass—«=65. 85:53an dow— wniaza a: 9.83%.: 95 .«c uuaatgmfim 9559. 05 0.89:8 0.235.». .8QO 3833.68 floofiso 8m mvoo: .fimgoh bmwmm 8 Eouuuaq Nunouumbmm bung o .8330th chamo— mo 85:35 05 Soon muons .fimgoh Amman.“ 9 83>qu Baotumgumz human 0 Plan—ROE): HmO—Z 2: 2:35 a? N can u 3 53:25.. ao_aa=a>< 855—2396 dam wan—«2.. no 283%.: 95 3.2: «a canton—Em 2: «.52 «mac—.7». 33308:. soEa=a>< 35:53an and unis—2.. no 9.33%.: 95 u: caning—.80 0:.—.0 132 82¢. .83. flog”. .0 £3 @ Q 6 © ® © @ ® 6 © © © 6 © 6 © @ .«o oncogmmfifi EBEmm 389.: 39.9.: 3 A W W 3 W m. A a w m m m m m m m m w m an m w m m m w I w w a d 9 w w I m a w w m m w s m o l u d e d I 0 H. w W o w o W w W m w m m a m m w w m m a m m 3 9 9 9 ..=_z._.....__: :8: .3 5:3..0 95.5.:72.1.5.1.21:— .Ifi. 332.98% son: .3 3530 95.5.5: .«o 9.33%.: 95 ac 3.3.335 8539. o... 9.....an once—ma. .muoo: 338285 to... @mumm v.8 .28: 3.8... .3058.“ not...» 9 w. .833 .Emsxuk \e MKS 35.23% $83.2: 0 45.58.35 an... mo 3:86 0:. v.8 m8... .8... mo egosbmmfifi on. 30.9.: 8 mm 2033 “32.x 35% \o m..5§h§.$.~ 988...: o EHMOEH HMO—Z 2: 2.3... a... N 1.... _ 3 5.2.2:... :85 .3 mum—«=0 963....3 no 9.83%... 33 one... no 353.59.: 95 :5... 3.33.”. 3:358... :35 no 5.350 96.5.5: .3 9.33%.... 95 .«o acmtanEcU 2.8. 133 38x. :35 2:183. .8923 @ 9 8 © @ ® ® ® 9 ® ® ® @ © 8 Q @ mum—8; 5:022... 385.: b_:=EEoU 08205 32... 535 «Ed. was :23. 503.3 @ 0 ® @ ® ® ® ® 9 ® @ © ® @ ® 0 @ 35?? 5:38:50 860825 08205 oEQEmGEoQ 05.5.5 «9?. Es; @ 0 6 © ® @ ® @ 6 ® ® © © © ® 0 @ 355323858 bEzEEoU 338.: u_:._a..onoQ 30.5.5 3 M S W 3 w S A a m m m m m m m m m w w an m n u an s w H m w m m m w w H m a"... m m m w w a... m o w u m m m w m o m u W .m m m a W «a. .M W a w a a m a w a a o 9 9 5.7.5.5: :a..:3=.51 .E::::::.v :_:_:_:_/x.._: 7:33:5— 3...: ”.8 5.2.9.5.. tongueuw b_..=......cU £35.32: .3 Ecuaomu... 8...: ma uu=ut¢55 9582 o... 9:588 ease—m.” .8888... mug—moon 3380.: v.8 man... :3... EH 7...... mo Hanna—gov on. 828.2 £053 63V 53.5 3.8 3.5% nuwéma 2889.85 3.3.2: o .mowESEEOO Rb... mo E08835“. on. 9.?me .3 $58.5 9.5.5560 3.3.85 9 6:28“. conning mo 8030... on. 3.8 223.38 mo mammm E255 o. .520 E 69%. 3.5% \o notmmomficb uEREMoEmQ 9.0.3.5.. o EHMOE HMO—z o... 2:6... :7. m... u .3 5.2.9.5.. :23935 b_:=......o0 £8532: ea 9.83%.: 3...: 36.: ac 3522...: o... :5... 883.". "nommnofig aoufioabm 5.5.8.50 55:32: no £835.: 8...: ac :ontanEoU on... 134 8892.. 88:82... 35883:... 8823. @ 8 © 6 © 6 @ e G 6 © 6 © 6 ® @ 185.85... £8.32 8892.. 38888388... @©@©®@@®6©@©®©@§§8€£§£ 8088835 gaoéaflfizne ®@®©@®e‘@@@e©©©@§§e§m§£ A v 3 A S S A 3 m m m W m M m m m m m m m m m m. m m m a"... w .m. m m. m a m w m m m m m m 1 w Wu .0 3 O 3 m m $.53 :>.u:_:um_: .3 155.55:— 3.5... ”2.2% {ac—com: .8 2.32.081 8...: a: 828.85.... 8.83.... o... 9:388 9235.”. .HGOEGEGD 2.. mo 8.5.38. 2.: Sons 2283 a...» 22.26 8.3.. 3:88 £033 £8839... 3.588....8m 888m 9 58.888 .85.. 8... 2.32.8.— ggoam "autumn... .633 ..§E=9...>=m 33.3...th 882...»: 0 8083—32. .«o 203.8 2553.. 25.60.. 2... $8.68.. 82.88.80 222... 82.80280 o... 8.603.... 523 .EmEzoxmgm 829$ o 2.5—0;— HmO—z 9.: 2.38 a... m... u .3 8% sac—com: 8 83238:. 3...: one... «a 383......8m a... 2...... 0235.". 2.3» c.3339. .8 2.32.98... 09.... ac 82......890 0...... 135 8089.95 2.9.800 33393920803. @ Q 8 © @ © © @ 6 ® ® © @ © 6 Q @ .8..08..0..>:m .300”. 80:50.03 .03....9302093 @ 0 ® © 6 © © @ 6 @ © © @ © 6 Q @ 80880.38”. 0308... 83830 380880.35. .303. © 0 6 © @ ® @ ® 6 ® ® © @ © ® 9 @ 80880325 0302.... m. M s w a w s M a m ,M m m m m m m m w m. a... m .1. u on s m H m w m m m m m I w w m w a m m m M... m m a w a m m a .57....3: :......:::.:,:_.... .0...:....:..: 1.3.3:.5 00...; 832.08... .....08..0...>..H 80.0.5: .0 8030...... 00.... .0 00:0...08... 03.30.. 0... 0.8.800 02.0... .u. ....08..0.0>0.. .0 0.00m8. 0333.. 0... 00.60. 0. m. 30...? ..=0E&0.0>0Q\0 8.08.8. 038$... 00:30.. 0 ....08..0..>..0 3.8.5380 0... 0. 0038.... 0... .300. 0. m. :3...» £0M0£0Q 3.588552% 580.0% o 80.3803... 3808:0325 m. 30...? ..=0E=0...>=m~ 0.20.0.0. o baby—OE).— .HwOS. 0... 2.008 x... m1 . .3 30m .2830...... .0 2.32.08... 00...... 00.0... .0 0088.098. 0... 3...... 02.0... .«. 832.08... $588038”.— .00.0....: .0 8080...... 00.... .0 800......800 0.. h. 136 p aawmvg 958800 gmotaoa @ © 6 © 9 © 6 a e e 6 © 6 © 6 © @ gaofiitaem A v a A S a S A a m s m m n w. m. s m 1 m w m m m m m 1 m an M. m m w 3 m m w m m m w m m m m m m a m a w m a u a a a FEE—SEE :EoEEIrEE _:._=:_5_._u< ___S=_:Z..._c {35:5— EXP 333256 :Eofinohgm _a..5_=otw< 53:32.. me 9.33%.: 95 no uuzatamsm 9553 2: 939:3 0233 .«o .momuggoo :23 25 $5282 58% 3:03wa 05 3.585 8 £808 £023 .bmzafiseb 3.3% 923.3% 9 .3on Eu: can obfisotwm mo oaaomvnfl o5 3.88.5 9 mm £033 .mmuumhtua Nuuzaautmw 923.3% 0 EHMOE— Rho: 2: p.58:— :7. m1 g 3 12% :hwc—SH: «a 332.25.. 09:: 82: ac 353395 05 :52 333.». "Egan-Eu ...:oE:c.—~>:H _§5_=utw< £355)? a: 9.33:2: 95 .«c floatanEcU 2;. 137 m >93 £5 5 56365.59 .59» .8.“ =9» v.52; mango mwmtsoh .«o cocaosvm O ® @ ® ® 6 G G G 6 © @ © 6 0 8.8m mo .838sz _Saofiaohzum gaugegcm aouwodwl awanxg @ 4 aoueuodwl Suons [(19 A saueuodwl Suons aouuuodml amapow aoucuoduq [12an aouuuodwl amapow aouwoduq fluons aoueuodml fiuons £19 A aoueuodwl amanxg V @ 5.7.555 ::=_....3c..._ _.3=Q:_::.__.Em_ 33:133.: 3.3.3:...— 33,—. 832585 auccooucum _Suofiuemifi 8.83am: no E3335 a3“ «a 3:8.355 3.5:»; 2: 9:588 ammo—mo". .mouréom 23:5th "uh—£2 E 338th >05 :23 9895 gamma: ounce.— 9 “.338 \c 2239th ~S§E§L3=M o .uaofiao~o>ov 231553“ chiv— mo $895 gumwoa 05 338 3 .2235 Such \o newcoahm ~S§E§b§m o Rafi—ROAS— HmO—z 2: 335 :7. ml fl .3 =3» sac—com: we macaw-SEE 09:: 985 .8 353.895 05 :52 case—mo». uflcmmfloamfl aflOCUOHGhA— —aufloafl¢hm>flm— Ouwufimvnms MG Eauflum‘fim GE .«0 flOmmhan—EOU 0&5. 138 APPENDIX VI: List of Potential Indicators No. Indicator Goal I-l Afiéflfififfiéfifillfi Economic I-2 Aféfi’fgfifiififi Economic I-3 $fi§fit§j§lb Economic I-4 Zfififi§gr饧¥Lifii§ftflflfi Economic 1-5 “Ai‘éfifiififi'fié'? Economic I-6 Llfifii'a‘a‘fifigé’fiéfii—gfifii Economic I-7 Lififiétéfifmé’lfiflfl Economic I-8 mafifii’tfifi'lfifiXfiWfifiéififi ’ ififiififiUflfifififi’ffilfi Economic I-9 Wfig’éi’éfifififiiflmfiflfififiifig Economic I- 1 O JAZEEEWI DB’J’fi‘é Economic H l 1‘ ~23§i§§¥§fi§i§rb Economic [-12 92mm Economic [-1 3 ii Economic I-14 iii] 1 *fltfiiifififfififiififififlifig’yflffi ° Economic I-15 EEEBWSIKIZIZ$ Economic I-16 Effimfigffifififlfifihi Economic I- 1 7 H? Fifi$¥ fiifé Economic I-18 Wfififififéfiéufit’fi Economic I-19 Wfiffibgjfiifiyfifififi Economic [-20 fifié’éfi Economic 1-21 fifigfliflfi’flafilfi ’ :Ffiéig—SEBmP—iéifiifi Economic 1-22 fiiffiié‘ ¥%B‘Jigflfltt$ Economic I-23 UZB’JifififigfiiT‘fé Economic [-24 @EEEDUA W'PO 347% 33 Economic 1-25 fififiifliKfifiéfiZflifi‘ Economic 1-26 itflfikjjfi’fl'ifift Economic 1-27 $Ez$§flfififi§¥¥ififli§$fififiafii§ Economic I-28 Effiifiim 20-25 Eilfijiié ’ $§Rf£é 1—2 $ififl§ig Economic gi§fi(fifl§fiflifififi%¥fifififi§§fifi ’ $E§3§Afigklkflggfi I-29 3+ ’ HEEEfifiEfiiEAEUE ’ Eififiéflgffikfi’flfifhfigéfifi Economic ) I-3O QEEJ’J 1 Eflfifijfifiifi ‘ Effifig ‘ fifiéfig’? Economic I-31 $2527] 3 é‘ffifflfffittfi’flfifiafitt ° Economic ‘1-32 magma Economic I-33 Wlfififi'igbfl Economic I-34 ”513373 3 Eflfiflffififitt$ Economic I-35 E%9&%Efiflffiififii%¥ Economic 139 No. Indicator Goal I-36 EEUfiB’JErfii Economic I-37 ?? if? 391$¥itiii§ Economic I-38 fi'fifiiiflifiifl Economic I-39 fifiifl‘fififi’9§¥lfi Economic I-4O fiifig’fiit Economic I-41 W3 BEE Economic I-42 fiifif§¥ Economic I-43 Efiifi’flfiggfii Economic 1-44 WEE—E’WKEKE Economic I-45 @fi‘ifififlflfil Economic [-46 EEEUi‘HWfiB’JfiEfl Economic 1-47 ’iifit/JZé/Eigé'iéfiikfiib Economic I-48 $395351? Economic I-49 Hfifigfiéfiig Economic I-SO {fifiii’fifi’flififfi Economic 1-51 fifififi’ié Economic I-52 fiiéififiéfiijiéfiifi Economic I-53 fifiifififi§¥$§fl Economic 1-54 iiifié¥fi§ Economic I-55 Kfii’éiffiigcgyéwifl Economic I-56 fififfifiEE-fififi Economic I-57 Ed‘fifififlflfillfigfi’flfi‘fi Economic *I-58 I'Eéfilflfizfif’afigifififiigflfléffi Economic 1-59 fiz’éfififlflifiéifi Economic I-6O fififififikéfligfifigfiififl ‘ E’r‘fi Economic I-61 EEE—i’éfii‘ffifififfi ‘ 'fiflfiifififi Economic I-62 EEEUEP‘Eifi Economic I-63 fééfiigfll]§%§fia Economic [-64 Efiflfl Economic I-65 fiitfgéfiflifi Economic 1-66 “13% Efifié’fifi‘éifi Economic 1-67 ‘IPSEJJJJ I iifififififififilifififi Economic I-68 DEE/3???? Economic I-69 fiifiéflfl’ffiQfié‘fifll ’ fiifi‘ifé‘iiéfi Economic I-7O BESZEEEEEJZf‘iallEfiBfifi/Eflfi? Economic I-71 fiEZHfiZKfiE‘iEDD Economic [-72 mififiéfi [3"] gfiitéfiifiéit Economic 1-73 {Elfiflifiéi‘é‘ afififififl‘é‘f Economic 1-74 fifiéfiiflfiijffi Economic I-75 fiéfiéfiéfiWJfi/Mfiffi Economic I-76 lfiéafififififlfifiijfi¥flfififi Economic 140 No. Indicator Goal I-77 Efiflififififififi Economic I-78 EB??§€%WW&%EWW%¥fififi‘fl‘lfibflz’f Economic I-79 Eifififi‘ggkfififi’fléfi Economic 1-80 53:551ng Economic I-81 ficfi’nfigfififi Economic I-82 @5553? Economic I-83 fiftér‘figfi’flgfifi Economic 1—84 figfilfi‘i ’ Eaifiifiéiémiififi‘ Economic I-85 ifiifilmgfifiZfif—n’? Economic 1-86 Efiug'ififh Economic .1-87 _ [511E551 Economic I-88 Efinfi'iféféfiifififi Economic I-89 Ec‘finéfiiffiigj Economic I-90 éfl‘é Economic 1—91 E55. Economic I-92 Effittfig‘ttfififiéflgfififiéfigz Economic I-93 Efl‘fifififiiififig ’ fiEWfi‘iB’Jfi? Economic 1-94 fiffifi‘fhfl’flttfi Economic 1-95 Eg¥99ifil Economic [-96 §¥W5<5§2 Economic I-97 fi555%@fixfifi%15§fl(migbflfifi%igégié ’ Wygzéfixfifififigflfl) Economic I-98 fifl5%’l§89§fi”l§§+% Economic 1-99 [535%5’553 Economic I-lOO 555%Ef’5 Economic I-101 585%3‘Ef’5 Economic [-102 EQW'E Economic [-103 Efiffiflfififififi§5afigfifififi5fidfiéfi§flfifi Economic I-104 Efiififfi%éfigfifii CEIEUE:% Economic I-lOS EWHEAWQ) Economic [-106 Eflfififififié‘ ’ Liigflflgii'ifit Economic I-107 EIJ1%%984J%¥%§ Economic 1-108 EUEWMEE Economic I-109 filJfaEPEfi Economic I-llO EUificfifififig Economic I-111 E'Jifigéi‘éfgffi Economic I-112 igfifiE-fi’ééf’it’flfi’flfiffi Economic I-113 [11555355551 Economic I-114 W¥ADEE§QHU Economic I-115 i55¥k§51 Economic I-116 fififi$5fi5§§ffi Economic I-117 fifiifig Economic 141 No. Indicator Goal I-118 5%15’3%—‘5§%¥5§~5%275fi Economic 1-1 19 fiffifi$gififiifikfifigflgfiffiifiifi Economic [-120 fiffifififiii’éfiéffififigfigfifiifilfi Economic I-121 fi1§§fi§5§fi€éi$fififlfi§b Economic [-122 {Efigiflfifi5fiifififig Economic I-123 fiffifigEfiEfiZ’fififiifigfifii Economic 1-124 551555§5589f£5§7£5§ Economic I-125 fifiééfifi’flufifi Economic I-126 fififiifgffi Economic I-127 $595555? Economic I-128 fi®1¢rfif§$ffififi§§fl§fi Economic I-129 Eff—5&5 ‘ @516 ‘ Wfifif’fiwfl Economic I-13O @fiéfiafizifi Economic I-131 E31584] mg: Economic I-132 @EB’JEE Economic I-133 @E§%%éfiffi Economic I-134 [Elfiflfilgfliflfififlgé Economic I-135 flififi‘JflJfififié’y Economic , i , ‘ E! " \c ‘ A {$5. , _ E? ‘ \ I-136. 55:51:5ng i$5§+fi§§ ’ EEEm5§§WW§¥E5§EJ§¥fi§iHfl Economic I-137 552%??? Economic I-138 ,fi‘éfiféflfiflfifilfi Economic [-139 [5535555551 Economic I-14O [WEREWEEZSJ’EE55E Economic I-141 fifiifiifiifififiififi 15% Economic I-142 fiififiifi/Dfiéréé Economic I-143 EEM‘M5'E ~ éEié/‘ifi Economic I-144 fifiéZ‘flflDflr‘éfifi‘ Economic I-145 %Efifiifitiflfiififi£§flé Economic I-l46 fififlfii’fifi 555155139552? Economic I-147 5%Efiéifi5’u‘ig735755 Economic I-148 %E€E§H§Clx’%?§i‘§flfl Economic I-149 %Efigfififififii§5§iflfifififififififi Economic I-150 EEB’JEBE‘EEE Economic I-151 5%551395115513353’5 Economic I-152 %Efiflfifit Economic I-153 5%E%f*fifi%¥fi’flifi% Economic I-154 fififiifififfiififlfifi’ifi‘éfl Economic I-155 5%E’55532155Jrgkfl Economic [-156 Eiéfifififiifififiéfiifié Economic I-157 ngifiiEgfiifiifififiiififiififlfiffiffififi Economic 142 No. Indicator Goal 1- 158 Erfiiflifififilé Economic I-159 5%E55555/F5’fl Economic [-160 5%E51 511157357515 Economic I-161 ‘~%E¥~1171<11§15%E55$‘55B4J1§RTE Economic [-162 5%5551555515551~~%§13111J‘5%Ef§ ”55558715555 Economic I-l63 1’ " E5555151151EEA51£ Economic I-164 5% EEELEW‘EEEEWWEEBWEE Economic I-165 5%E55fi841d555 Economic I-166 5%153955EAD Economic I-167 5%EE555E1’L13E55 Economic I-168 %iuu§&5rf1iuu11§figfiflfi1tgfi%@5% Economic [-169 15:5115558’155’6711‘55 Economic I-170 541%59’1555 5511511513 Economic I-171 5% Emifi’fi Economic I-l72 5% 3211*} 1%{15 Economic 1-173 ‘%i5’ 375555551532 Economic I-174 515551535513 Economic I-l75 1%E5EQFE545539555155 Economic I-176 EEfRiEE/A Economic I-177 fig‘fifir‘fléfififflfi ’ 11595355511511 Economic I-178 JEEEJEEE Economic I-179 5E35555fi$15111§51511fit21553915555E151 Economic 1-180 355% Economic [-181 SIESE Economic [-182 5.1555555111305115"? Economic I-183 51:1 551155W115b5£3995253f 11515: 57955195151555) Economic I-184 ¥1ii1313715151cu1m55155 fiifEE \121 A35 Economic I-185 5113155me521 51115111; / 15.-5:1" 15110 Economic I-186 éécLTiE’f‘E“? Economic [-187 “’“EE’EEEWJ‘EE Economic I-188 "11525151115- Economic I-189 igbfllf’flfig Economic I-19O 151101111X Economic I-191 iEHUEEEZWK Economic [-192 5551JU5%E111UK Economic I-193 iEDUEEB‘JI’EifiWE Economic I-194 iEDUF-‘EWEEEMEBS’XEZEQ ’ 151311115)? EEEEWEE Economic I-195 i‘EDUEHSfiEZi‘EE Economic [-196 563DU§EELRJEJJ§E§E§ Economic I-197 5555a Economic 1-198 gkfi‘églfififiififififi‘flfifl Economic 143 No. Indicator Goal [-199 55555891555WWE$1£55 ‘ 5555555551555395515) Economic I-200 $3515555~E¥55fl Economic I-20l '5iE1J5%E%555‘2 Economic I-202 EIEJTEEflZ-fl Economic I-203 51551589515525 Economic I-204 5558957155555 Economic I-205 555555 Economic 1-206 555551 Economic I-207 55555555115555??? Economic 1-208 15575: Economic 1-209 @5535 ~ 551551555551555151535115711 Economic [-210 ADI—£1115 Enjoyment 1-21 1 15513925511155 Enjoyment 1-212 vLV‘5E51‘513/L‘ Enjoyment I-213 311131391515 Enjoyment 1-214 11131514315)? Enjoyment I-215 flflifififiiifiifififiifié‘ Enjoyment 1-216 55:53:15 Enjoyment I-217 ‘IEEQ’EEEEE Enjoyment 1-218 $585151: Enjoyment I-219 EWEKZWlfififl? Enjoyment 1-220 1551555a3%igflfl Enjoyment I-221 %A£§ Enjoyment I-222 Eifi'fiiififfiifi Enjoyment I-223 3555157595J$55 Enjoyment I-224 Eigfififlfli‘fi Enjoyment I-225 55555569155555555 Enjoyment 1-226 fiiIESB’Jif-fésé Enjoyment I-227 1155152555 Enjoyment 1-228 515$EA5H55ZEBWEHU Enjoyment 1-229 355%(515555—5) Enjoyment I-230 5%??551‘9‘1—55515115 Enjoyment 1-231 1553551711: Enjoyment I-232 {Ei’ééfifiififltflgfljé‘ Enjoyment I-233 E55551 E19555 Enjoyment I-234 5555555155355555555 Enjoyment I-235 EEEEEEEfiL—I’EEREHUEE Enjoyment 1-236 551i: Enjoyment I-237 $E15§¥$1§995515§55555 Enjoyment I-238 $55AE5’15LEEQEQ15 Enjoyment 1'39 5555,55552555555539555,15 Enjoyment 144 No. Indicator Goal I-240 fififié‘figfig‘EfiWfiZfiE Enjoyment . I-241 Efifl’ffggfigfiififi‘éF‘a‘lZfié‘lifififiZEEfi’li Enjoyment 1-242 Effifififfifi‘éfi’flfifiglfi Enjoyment I-243 fit‘fifi’flgfi’fl Enjoyment 1-244 [[éEMWaH'iB’JfiflEE Enjoyment I-245 [@Elflafi Enjoyment I-246 Efifififlfififiifififi’flfififii Enjoyment I-247 fififilfig Enjoyment I-248 flfifilfi Enjoyment I-249 Vfitfifiifilfififi’figi Enjoyment I-250 Wéflkfi‘fiylfifififi’flfifiq Enjoyment I-251 filfififiiilfifigfifi Enjoyment 1-252 filfié‘lfilfifi’flfi‘fib Enjoyment [-253 filfifififi’flfiilfi’ffiifi Enjoyment I-254 lfilfilflkfifififfiffi Enjoyment I-255 filfil‘B—ZIEHKFHEJ Enjoyment I-256 fifiil‘aflfilfihfifiififiZffi Enjoyment I-257 filfififiZfifi Enjoyment [-258 filfififikggfigfifii Enjoyment I-259 figfiifidt‘ifi Enjoyment I-26O 7%?figfiéx ‘ {ETE ‘ fig)? ‘ Kai—3} %’%El§i§ Enjoyment [-261 @fifit'lfimfifiifil’affiififii ~ Sifti‘éficjifi) Enjoyment [-262 $lfl‘fiiE/fiigfié Enjoyment I-263 Elfiflfi‘glflifiifi’ Enjoyment I-264 Efifiififi ‘ filt¥fl§zfi Enjoyment I-265 Eil‘lfigégfi Enjoyment I-266 E§¥%§fifi@¥fl%§$§§%lfififi%¥ Enjoyment I-267 figfiflififijfifidlflféélfijflfi ‘ E‘E \ 3%X33‘é) Enjoyment I-268 figfifigfifi Dé‘Jj’fiI/Zifiijfilfi Enjoyment [-269 j‘hfiEIA¥JEE§EB’J%§X Enjoyment I-27O \Efigié’flitfikfi Enjoyment I-271 igfilflfifi§¥fi§fi2fifigfi Enjoyment [-272 iii Enjoyment I-273 [figlfijéfififiigfigjzfiglgrflEfiizfi’flii'éfixg Enjoyment I-274 EfiflmkgfifilflfiBWElaa‘i Enjoyment I-275 AIEQEEWEEAIRBSEFE Ecology I-276 $¥Efi§lfi§fififfi Ecology 1-277 4t$HElil€4898§fififiiU Ecology I-278 BZH: Ecology I-279 {tifi’flfifififi Ecology I-280 vkilfifi‘fiigé Ecology 145 No. Indicator Goal 1-281 EEEEEE‘EEE‘E Ecology I-282 EELE Ecology I-283 ff‘3?I$£ Ecology I-284 EEEHEWEEE Ecology I-285 :lEEfi‘iEEl’Jfiéux Ecology I-286 E‘EEEffiiE EEE EHWEE‘E Ecology I-287 f—EEE‘f " EEZEE‘JEW Ecology I-288 éffilififillffifi- “Ed's? Ecology I-289 lEEfiflmfi’JEWJ Ecology I-290 {El EBEHEB’JEQ Ecology 1-291 lEEEEEfEfE J’l’l’lf‘C m1 Ecology l-292 Efifiiflfififi Ecology I-293 ‘QIEEEEIEEE Ecology I-294 EFSJEEEEEEE‘EEEEB‘HEZF Ecology I-295 {Eff-SCH: Ecology I-296 {EYEEEHEKEE Ecology I-297 TE?¥%‘%B’J:E1‘3E Ecology I-298 {EEEEEEQEEE Ecology I-299 [FEE Ecology 1-300 lEEgilfiéig’Jfieitgzsrfil Ecology [-30] {EEEEEEEEE’JEQE Ecology 1-302 EEEEWEEEB‘JEEMEIE Ecology I-303 ETAEE’JAISQJE Ecology I-304 E7‘ 41EFfiFfil§€ HE Ecology [-305 E m7‘fiijffifii‘fi Ecology I-3067o; “Tfiflé’fllfil Ecology 1-307 Eff fifi‘ififioffi Ecology I-308 Efiiflzifififii EEE‘JEHEEEE‘E‘ER‘EEJ‘E Ecology I-309 [ETEEBUKB‘ME ’3}: Ecology I-3lO 1E3???) EEB’JEE ”llt‘ifilfi ‘EJfFl‘ HEW? Ecology [-31] [illffEffEJB‘JfEa Ecology 1312 WEEEWWFWWEE Ecology I-3l3 iszE‘l‘itfiEi/EEE’JEEXE E35023“; Ecology I-314 5‘2 {EE 941 E’JE5’H7F5W% BEER Ecology I-315 AJSLI’R’BE’LB’J Ecology 1-316 lli‘Tch"éjéo§Lcfi{gfi’l 5‘" ”l3 Ecology 1-317 693% fic/Aitmfifgb‘élo Ecology I-318 EEEB’JE‘EE Ecology I-3l9 Eiflfilfiijmi Ecology I-320 itflEEFEXEEElfi‘E‘iEéfEfiEHE’” Ecology 1-321 ENHEWEE(@EEFEI1§E€EEW) Ecology 146 No. Indicator Goal I-322 EEEEIEEEEJE i513 Ecology I-323 EEEEIB’JBEE‘H’EE‘J Ecology I-324 E ‘EEEEE§U¥§E(EEE§§E§UJCB’JEQE) Ecology I-325 EEEU‘EEETEEEEZEEERETE Ecology I-326 J‘EEEFMH’SMEEK‘FEZE Ecology I-327 EiflflifijEE Ecology I-328 ElfiEfi-fiffiifi Ecology 1-329 EEEEEEWEE Ecology 1-330 EEEE Ecology I-331 EEEEB’JQEE Ecology [-332 EEiEl-Egi Ecology 1-333 fEiEe’irEuE Ecology I-334 LEEEEE Ecology 1-335 EEEEEE ’ WEEE Ecology 147 BIBLIOGRAPHY 148 BIBLIOGRAPHY Aczel, J ., & Roberts, F. S. (1989). On the possible merging functions. Mathematical Social Science, 1 7, 205-243. Albayrak, E., & Erensal, Y. C. (2004). Using analytic hierarchy process (AHP) to improve human performance: An application of multiple criteria decision making problem. Journal of Intelligent Manufacturing, 15(4), 491-504. Andreoli, M., & Tellarini, V. (2000). Farm sustainability evaluation: methodology and practice. Agriculture, Ecosystems & Environment, 77(1-2), 43-52. Arbuckle, J. L. (1996). Amos 4.0 [Computer software] (Version 4.0). Chicago: Smallwaters. Arbuckle, J. L., & Wothke, W. (1999). Amos 4. 0 user's guide. Chicago: SmallWaters Corperation. Basarir, A. (2002). Multidimensional goals of farmers in the beef cattle and dairy. Louisiana State University. Bennett, M., James, P., & Klinkers, L. (1999). Sustainable measures: evaluation and reporting of environmental and social performance. Sheffield: Greenleaf. Bernadette, M. R., Krishnamurty, M., & Karen, P. (1998). The development of a systematic, aggregate measure of corporate social performance. Journal of Management, 21(1), 119-133. Bohman, M., Cooper, J ., Mullarkey, D., Norrnile, M. A., Skully, D., Vogel, S., & Young, E. (1999). The use and abuse of Multifunctionality. Economic Research Service/USDA. Retrieved Jan, 2003, from the World Wide Web: http://www.ers.usda.gov/brieginglwto/pdf/multifuncl 1 l9.pdf 149 Bosshard, A. (2000). A methodology and terminology of sustainability assessment and its perspectives for rural planning. Agriculture, Ecosystems & Environment, 77(1-2), 29-41. Braunschweig, T., & International Service for National Agricultural Research. (2000). Priority setting in agricultural biotechnology research : supporting public decisions in developing countries with the analytic hierarchy process. The Hague, Netherlands: International Service for National Agricultural Research. Brown, R. E., & Pyers, J. B. (1988). Putting teeth into the efficiency and effectiveness of public services. Public Administration Review, 48(May/June), 735-742. Byrne, B. M. (2001). Structural equation modeling with AMOS .' basic concepts, applications, and programming. Mahwah, N.J.: Lawrence Erlbaum Associates. Campbell, D. T., & Russo, M. J. (2001). Social measurement. Thousand Oaks, Calif: Sage Publications. Carson, D. (2001). Qualitative marketing research. London ; Thousand Oaks, Calif: Sage. Centers for Disease Control and Prevention Office on Smoking and Health. (1999). Best Practices for Comprehensive Tobacco Control Programs. CDC. Retrieved June, 2004, from the World Wide Web: http://www.cdc.gov/tobacco/bestprac.htm Chen, C. (2002). The strategy of leisure agriculture development. Journal of NCH U Agriculture, 31, 13-21. Chen, H. T. (1994). Theory-driven evaluations: Need, difficulties, and options. Evaluation Practice, 15, 79-82. 150 Chen, Y. H. (2003). Apply of fuzzy hierarchical additive weighting method on project evaluation? The case of “town beautification movements ” in Tainan. ‘ Unpublished Doctor of Philosophy, National Chung-Hsin University, Taichung, Taiwan. Cheng, E. W. L., & Li, H. (2001). Analytic Hierarchy Process-An approach to determine measures for business performance. Measuring Business Excellence, 5(3), 30-36. Cheng, S. H. (1996). A study of leisure farming development area systems. Journal of Taiwan Land and Finance, 33(2), 195-213. Chiang, J. C. (1997). The management of leisure farm. Journal of Agricultural World, 169, 31-33. Chwolka, A., & Raith, M. G. (2001). Group preference aggregation with the AHP- implications for multiple-issue agendas. European Journal of Operational Research, 132, 176-186. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334. Curry, B., & Moutinho, L. (1992). Environmental issues in tourism management: computer modeling for judgmental decisions. International Journal of Service Industry Management, 3(1), 57-69. DeGraaf, D. G., Jordan, D. J ., & DeGraaf, K. H. (1999). Programming for parks, recreation, and leisure services .' a servant leadership approach. State College, PA: Venture Pub. Deng, J ., King, B., & Bauer, T. (2002). Evaluating natural attractions for tourism. Annals of Tourism Research, 29(2), 422-438. 151 Dinc, M., Haynes, K. E., & Tarimcilar, M. (2003). Integrating models for regional development decisions: A policy perspective. The Annals of Regional Science, 37(1), 31-53. Ding, C. S., & Hershberger, S. L. (2002). Assessing content validity and content equivalence using structural equation modeling. Structural Equation Modeling, 9(2), 283-297. Dunn, W. N. (1994). Public policy analysis : an introduction (2nd ed.). Englewood Cliffs, N.J.: Prentice Hall. Easley, R. F ., Valacich, J. S., & Venkataramanan, M. A. (2000). Capturing group preferences in a multicriteria decision. European Journal of Operational Research, 125, 73-83. European Commission. (2001 ). A fiamework for indicators for the economic and social dimensions of sustainable agriculture and rural development. European Commision. Retrieved Dec., 2002, from the World Wide Web: http://europa.eu.int/comm/agriculture/publi/reports/sustain/index_en.pdf. Finland, M. (1997). Environmental benefits from agriculture: issues and policies: the Helsinki Seminar. Paris: Organisation for Economic Co-operation and Development. Fleischer, A., & Felsenstein, D. (2000). Support for rural tourism: does it make a difference? Annals of Tourism Research, 27(4), 1007-1024. Forrnan, E., & Peniwati, K. (1998). Aggregating individual judgments and priorities with the Analytic Hierarchy Process. European Journal of Operational Research, 108, 165-169. F orman, E. H., & Gass, S. I. (2001). The analytic hierarchy process- An exposition. Operations Research, 49(4), 469-486. 152 Forman, E. H., & Selly, M. A. (2002). Decision by Objectives. Washington, DC: World Scientific Pub Co. Gustafson, K. (2002). University of Minnesota, Tourism Center Web,. Retrieved July, 2002, from the World Wide Web: http://www.tourism.umn.edu/zagtour.html Hatry, H. P., & Wholey, J. S. (1999). Performance measurement: getting results. Washington, DC: Urban Institute Press. Henderson, K. A., & Bialeschki, M. D. (2002). Evaluating leisure services: making enlightened decisions (2d ed.). State College, PA: Venture Publishing. Hilchey, D. (1993). Agritourism in New York State: opportunities and challenges in farm-based recreation and hospitality (2nd Nov. 1993. ed.). [New York: Farming Alternatives Program Dept. of Rural Sociology Cornell University. Hollenhorst, 8., Olson, D., & Fortney, R. (1992). Use of importance-performance analysis to evaluate state park cabins: The case of the West Virginia State Park System. Journal of Park and Recreation Administration, 10(1), 1-1 1. Hornik, R., Maklan, D., Diane, C., Barmada, C. H., Jacobsohn, L., Prado, A., Romantan, A., Orwin, R., Sridharan, S., Zanutto, E., Baskin, R., Chu, A., Morin, C., Taylor, K., & Steele, D. (2003). Evaluation of the national youth anti-drug media campaign: fifth semi-annual report of findings. Prepared for the National Institute on Drug Abuse. Retrieved June, 2004, from the World Wide Web: http:flwwwmediacampaignng/publications/westatS/index.html Huan, T. C., & O'Leary, J. T. (1999). Measuring tourism performance. Champaign, 111.: Sagamore. Hudson, 8., & Shephard, G. W. H. (1998). Measuring service quality at tourist destinations: an application of importance-performance analysis to an Alpine ski resort. Journal of Travel and Tourism Marketing, 7(3), 61-77. 153 Jackson, E. L., & Burton, T. L. (Eds). (1999). Leisure Studies: Prospects for the 213t Century. State College, Pennsylvania: Venture. Jeng, H. Y., Liu, C. C., & Chen, H. C. (1995). Issues and solutions in the development of agriculture tourism in Taiwan. Journal of Taiwan Land and Finance, 32(1), 99- 118. Kajanus, M., Kangas, J ., & Kurttila, M. (2004). The use of value focus thinking and the A’WOT hybrid method in tourism management. Tourism Management, 25, 499- 506. K30, C. (2000). Measuring the performance improvement of Taiwan Forests after Reorganization. Forest Science, 46(4), 577-584. Kline, R. B. (1998). Principles and practice of structural equation modeling. New York: A Division of Guilford Publication, Inc. Koenig, L. W. (1986). An introduction to public policy. Englewood Cliffs, N.J.: Prentice- Hall. ' Krafi, M. E., & Furlong, S. R. (2004). Public policy .' politics, analysis, and alternatives. Washington, DC: CQ Press. Kravchuk, R. S., & Schack, R. W. (1996). Designing effective performance measurement systems under the Government Performance and Results Act of 1993. Public Administration Review, 56(4), 348-358. Lee, H., Kwak, W., & Han, I. (1995). Developing a business performance evaluation system: an analytic hierarchical model. The Engineering Economist, 40, 343-357. Leung, P., Muraoka, J ., Nakamoto, S. T., & Pooley, S. (1998). Evaluating fisheries management options in Hawaii using analytic hierarchy process (AHP). Fisheries Research, 36(2-3), 171-183. 154 Li, J. Y. (1996). Leisure agriculture. Agricultural Policy and Information, March, 53-56. Li, M. H. (1996). The guidelines of leisure agriculture development. Agriculture World, 153, 90-93. Lin, Y. C., & Yang, C. (2002). Fairness and justice of the procedures for the promotion system recommendations in government organizations: A case study of Taiwanese police officers. International Journal of Public Administration, 25(9,10), 1235-1249. Liu, F. S. (1994). The review of leisure agriculture area development. Agricultural Policy and Information, May, 15-19. Liu, F. S. (1997). The review of leisure farm development. Journal of Agricultural Finance, 40, 1-33. Lobo, R. (2002). University of California, Small Farm Center Web,. Retrieved July, 2002, from the World Wide Web: http://www.sfc.ucdavis.edu/agritourism/definition.html Lu, J. R., & Hsiao, W. C. (2003). Does universal health insurance make health care unaffordable? Lessons from Taiwan. Health Aflairs, 22(3), 77-84. Lundegren, H. M., & Farrell, P. (1985). Evaluation for leisure service managers: a dynamic approach. Philadelphia: Saunders College Pub. MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowski, K., & Turner, K. (2001). Introduction to program evaluation for comprehensive tobacco control programs. Centers for Disease Control and Prevention. Retrieved June, 2004, from the World Wide Web: http://www.cdc.gov/tobacco/evaluation manual/contents.htm Maier, L., Shobayashi, M., & Organisation for Economic Co-operation and Development. (2001). Multifunctionality .' towards an analytical framework. Paris, France: Organisation for Economic Co-operation and Development. 155 Mariampolski, H. (2001). Qualitative market research .' a, comprehensive guide. Thousand Oaks: Sage Publications. Mendoza, A. G., & Sprouse, W. (1989). Forest Planning and Decision Making Under Fuzzy Environment: An Overview and Illustration. Forest Science, 35(2), 481- 502. Normile, M. A., & Bohman, M. (2002). Non-Trade Concerns: International Debate & US. Policy. Agricultural Outlook, 14-17. Nugent, W. R., Sieppert, J. D., & Hudson, W. W. (2001). Practice evaluation for the 21st century. Belmont, Calif: Brooks/Cole-Thomson Learning. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill. Ong, S. K., Koh, T. H., & Nee, A. Y. C. (2001). Assessing the environmental impact of materials processing techniques using an analytical hierarchy process method. Journal of Materials Processing Technology, 113(1-3), 424-431. Organisation for Economic Co-operation and Development. (1999). Environmental indicators for agriculture. Paris: Organization for Economic Co-operation and Development. Organisation for Economic Co-operation and Development. Directorate for Food Agriculture and Fisheries. Policies and Environment Division. (2000). Environmental indicators for agriculture : methods and results : executive summary .' [agriculture and food] . Paris: Oecd. Ossadnik, W., & Lange, O. (1999). AHP-based evaluation of AHP-Software. Eur0pean Journal of Operational Research, 118, 578-588. Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1985). A conceptual model of service quality and its implications for future research. Journal of Marketing, 49, 41-50. 156 Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). SERVQUAL: A multiple-item scale for measuring consumer perceptions of service quality. Journal of Retailing, 64(1), 12-40. Partovi, F. Y., Withers, B. E., & Brafford II, J. E. (2002). How Tompkins rubber company used analytic hierarchy process to enhance ISO 9000-related decision making. Production and Inventory Management Journal, 43(1/2), 13-22. Patton, M. Q. (1997). Utilization-focused evaluation .' the new century text (3rd ed.). Thousand Oaks, Calif: Sage Publications. Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, Calif: Sage Publications. Pesonen, M., Kurttila, M., Kangas, J ., Kajanus, M., & Heinonen, P. (2001). Assessing the priorities using A'WOT among resource management strategies at the Finnish Forest and Park Service. Forest Science, 47(4), 534-541. Poister, T. H., & Streib, G. (1999). Performance measurement in municipal government: assessing the state of the practice. Public Administration Review, 59(4), 325-335. Posavac, E. J ., & Carey, R. G. (2003). Program evaluation: methods and case studies (6th ed.). Upper Saddle River, N.J.: Prentice Hall. Prem, B., & Michel, G. (1999). The "Multifunctionality" of agriculture: Recognition of agriculture as a public good or position against trade liberalization? Paper presented at the FAQ/Netherlands Conference on the Multifunctional Character of Agriculture and Land, Netherlands. Purdon, S., Lessof, C., Woodfield, K., & Bryson, C. (2001). Research Methods for Policy Evaluation. London: National Center for Social Research. Radcliffe, L. L., & Schiedeljans, M. J. (2003). Trust evaluation: An AHP and multi- objective programming approach. Management Decision, 41(5/6), 587-595. 157 Ramanathan, R., & Ganesh, L. S. (1994). Group performance aggregation methods employed in AHP: An evaluation and an intrinsic process for driving members' weights. Journal of Operational Research, 79, 249-265. Reisinger, H., Cravens, K., S., & Tell, N. (2003). Prioritizing performance measures within the balanced scorecard framework. Management International Review, 43, 429-437. Ridgley, M. A., & Rijsberman, F., R. (1994). Multicriterion analysis and the evaluation of restoration policies for a Rhine Estuary. Socio-Economic Planning Sciences, 28(1), 19-32. Rossman, J. R., & Schlatter, B. E. (2003). Recreation programming : designing leisure experiences (4th ed.). Champaign, 111.: Sagamore Pub. Russ-Eft, D. F., & Preskill, H. S. (2001). Evaluation in organizations: a systematic approach to enhancing learning, performance, and change. Cambridge, Mass: Perseus. Saaty, T. L. (1980). The analytic hierarchy process: planning, priority setting, resource allocation. New York ; London: McGraw-Hill International Book Co. Saaty, T. L. (1990). Multicriteria decision making: The analytic hierarchy process (2nd Edition ed.). Pittsburgh: RWS Publications. Saaty, T. L. (1994). How to Make a Decision - the Analytic Hierarchy Process. Interfaces, 24(6), 19-43. Saaty, T. L. (1995). Decision'making for leaders (Third Edition ed.). Pittsburgh: RWS Publications. Saaty, T. L., & Vargas, L. G. (1994). Decision Making in Economic, Political, Social and Technological Environments: The Analytic Hierarchy Process. Pittsburgh: RWS Publications. 158 Schmoldt, D. L., & Peterson, D. L. (2000). Analytical group decision making in natural resources: Methodology and application. Forest Science, 46(1), 62-75. Shen, C., Huang, C., & Chu, P. (2003). A performance evaluation model for govern mental conflict management organizations - a study of labor management departments. International Journal of Management & Decision Making, 4(4), 312-331. Siegenthaler, K. L. (1994). Importance-performance analysis: Application to senior programs evaluation. Journal of Park and Recreation Administration, 12(3), 57- 70. Silverman, D. (2000). Doing qualitative research .' a practical handbook. London ; Thousand Oaks, Calif: Sage Publications. Stufflebeam, D. L., & American Evaluation Association. (2001). Evaluation models. San Francisco: Jossey-Bass. Suh, D. K. (2001). Social and economic evaluation of the multifunctional roles of paddy farming. Retrieved January, 2003, from the World Wide Web: http://www.agnet.org/library/abstract/ebS1 1 .html Swiercz, P. M., & Ezzedeen, S. R. (2001). From sorcery to science: AHP, a powerful new tool for executive selection. HR. Human Resource Planning, 24(3). Torgerson, W. S. (1958). Theory and methods of scaling. New York,: Wiley. Tzeng, G. H., Teng, M. H., Chen, J. J., & Opricovic, S. (2002). Multicriteria selection for a restaurant location in Taipei. International Journal of Hospitality Management, 21(2), 171-187. 159 United States Department of Agriculture, & Service, F. S. a. I. (2002). Maxi/lex pilot program in the food safety and inspection service. USDA. Retrieved June, 2004, from the World Wide Web: http://www.fsis.usda.gov/Frame/FrameRedirect.asp?main=htg)://www.fsis.usda.g ov/OPPDE/peis/FinRepshtm United States Department of Education. (2004). Fiscal year 2004 program performance report. USDE. Retrieved June, 2004, from the World Wide Web: hm}://www.ed.gov/about/reports/annual/2004report/program.html Veal, A. J. (2002). Leisure and tourism policy and planning (2nd ed. ed.). New York: CABI Publishing. Wang, H., Xie, M., & Goh, T. N. (1998). A comparative study of the prioritization matrix method and the analytic hierarchy process technique in quality function deployment. Total Quality Management, 9(6), 421-430. Wang, J. (2004). The study on the planning and effectiveness of re-employment training programs for the middle-aged unemployed at vocational training institutes. Unpublished Doctor of Philosophy, National Chung-Cheng University, Chia-Yi, Taiwan. Wholey, J. S., & Harty, H. P. (1992). The case for performance monitoring. Public Administration Review, 52(November/December), 604-610. Wholey, J. S., & Newcomer, K. E. (1989). Improving government performance: evaluation strategies for strengthening public agencies and programs (lst ed.). San Francisco: Jossey-Bass. William, J. T. 0., Eaves, R. C., & Cox, C. (2002). Confirmatory factor analysis of an instrument designed to measure affective and cognitive arousal. Educational and Psychological Measurement, 62(2), 264-283. 160 Wolcott, H. F. (2001). Writing up qualitative research (2nd ed.). Thousand Oaks, Calif: Sage Publications. Worthen, B. R., Sanders, J. R., & F itzpatric, J. L. (1997). Program Evaluation (2nd ed.). New York: Longman. Xu, 2., & Wei, C. (1999). A consistency improving method in the analytic hierarchy process. European Journal of Operational Research, 116, 443-449. Ye, Y., Jin, K., Zhang, H. C., Ling, F. F ., & Barnes, D. (2000). A decision-making model for materials management of end-of-life electronic products. Journal of Manufacturing Systems, 19(2), 94-107. Yu, Y.H. (1991). Leisure agriculture development in Taiwan-present and future. Journal of Sports, 20(3), 6-14. Yurdakul, M. (2002). Measuring a manufacturing system's performance using Saaty's system with feedback approach. Integrated Manufacturing Systems, 13(1), 25-34. 161 11111110111191ij11111:»