,. .u .fl. . lllllJUlllUllllHIJUllWIJHIHHHHIlllHHlll/lllflfl 31293 10681 7947 - n ‘ -' ~ .‘5 _ . I I. .‘ _ . r. . s-. -_- d‘ ' . __ _ . - ' . — M H“- F .\ .1_....-..- .-4. 1 «EL s Lz_.if-,.__.rr I“, This is to certify that the dissertation entitled Identification of Critical Elements for Effective Nonformal Education Assessment presented by Mary Joy Pigozzi has been accepted towards fulfillment of the requirements for Phd degree in Minimal Administration @2141. wk Major professor Datew MS U is an Affirmative Action/Equal Opportunity Institution 0-12771 MSU LIBRARIES ”- RETURNING MATERIALS: Place in book drop to remove this checkout from your record. FINES wi11 be charged if book is returned after the date stamped beiow. lib“! ‘3 (Cult, 0‘.-. G) as its) t3 0 . :F “A; .r' ‘ ‘ ,o- ,. - 1 , i" . '7": Ah!“ 9:322; 23% 9;“ IDENTIFICATION OF CRITICAL ELEMENTS FOR EFFECTIVE NONFORMAL EDUCATION ASSESSMENT By Mary Joy Pigozzi A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY College of Education 1986 Copyright by MARY JOY PIGOZZI 1986 ABSTRACT IDENTIFICATION OF CRITICAL ELEMENTS FOR EFFECTIVE NONFORMAL EDUCATION ASSESSMENT By Mary Joy Pigozzi This research identifies and describes eight essential elements for an ef f ec- tive nonformal education (NFE) assessment. An NFE assessment is a detailed description and analysis of the status, objectives, plans, needs, constraints, and priority target areas in nonformal education with the assessment serving as the basis for rank-ordered action recommendations within a national education system. The major objective of the assessment is to enhance data-based planning and resource al- location in the NFE subsector. Results of this research will improve the selection, organization, and analysis of information on nonformal education for use by na- tional level planners. A This study is significant because of its foundational nature. There is need for better information on the scope and nature of NFE activities within a national education system and for improved allocation of scarce resources among these activities. And, there is no current literature to guide the planning and management of NFE assessments. The grounded theory approach, which uses comparative analysis to generate theory from data, was selected for the methodology because of its relevance to applications-oriented research in domains with limited existing theory. Open-ended questions were developed and administered through interviews with seven profes- sionals having recent national NFE assessment experience; they each had par- ticipated in one or more USAID-sponsored assessments in a developing country be- tween 1982 and 1985 (Botswana, Cameroon, Haiti, Indonesia, Lesotho, Liberia, Somalia, and the Yemen Arab Republic). Eight elements critical for conducting an NFE assessment are identified: purpose, preparation, policy environment, process, roles, methodology, NFE program descriptions, and linkages. Each conceptually discrete element is defined by an em- pirically derived set of between three and fourteen properties. Many of these ele- ments are not identified in the literature on NFE assessments, which has generally focused on NFE program descriptions, only one of the elements identified in this research. Most of these elements are intangible, so it takes special effort to ensure their inclusion. Recognition and inclusion of these elements will significantly im- prove the planning and management of NFE assessments. Additional research is needed to refine the conceptual nature of these elements. Such research could also identify additional elements. ACKNOWLEDGEMENTS Many people have provided support and encouragement while I completed my doctorate. Thanks go first of all to Bruce for his confidence, advice, and humor, and for the times when his patience was sorely tested! And, I am deeply grateful to my parents, who always encouraged me to look for a way to do the things that were really important to me. Joe Levine, who served as dissertation advisor, is a colleague and friend who provided helpful guidance and much needed support at exactly the right times. He was there for the review of my application to graduate school and still there after the dissertation defense! I thank Chuck McKee, who was chair of my doctoral committee and whose questions and suggestions assisted in the development of a focus for this study. The dissertation is the final step in a much longer series of learning opportunities. Bill Derman and George Ferree are faculty members who helped me learn a great deal and who were also kind enough to agree to be members of my doctoral committee. I thank them for their minds, teaching, time, and friendship. I am most appreciative of George Axinn’s advice and friendship. I am also grateful to MSU as an institution. It provided me with extraordinary opportunities to com- bine graduate school with exciting professional work and to allow each to nourish the other. Special thanks go to Barbara Hopkins, whose excellent skills, patience, and humor saw both me and this document through many drafts! I am sure that any readers will join me in thanking Vic Cieutat for editing the final draft. I am most ii grateful to all my colleagues at the Institute for International Research, who provided professional support in so many ways and who recognized that an unusual schedule can be just as productive as a usual one. Finally, I thank family and friends who showed interest and support from places as far away as Botswana, Great Britain, and Haiti and as close as Michigan. Two special women have always been there if I ever needed to talk—Annie McDonald and Janie Clark, thank you! iii TABLE OF CONTENTS ABSTRACT INTRODUCTION AND PROBLEM IDENTIFICATION Overview and Context Purpose Assumptions Definition of Terms Significance Areas of Inquiry Methodology and Data Sources Generalizability Limitations of the Study Delimitations of the Study LITERATURE REVIEW Assessment of the Education Sector Planning Nonformal Education Nonformal Education Assessment Summary METHODOLOGY AND DATA COLLECTION Grounded Theory The Conduct of Grounded Theory Appropriateness of Grounded Theory for this Study Data Sources The Interview Process Summary PRESENTATION AND ANALYSIS OF DATA Presentation of Data Respondent One Respondent Two Respondent Three Respondent Four Respondent Five Respondent Six Respondent Seven Analysis of Data Element 1, Purpose Element 2, Preparation Element 3, Policy Environment Element 4, Process Element 5, Roles iv Element 6, Methodology Element 7, NFE Program Descriptions Element 8, Linkages Summary FINDINGS, DISCUSSION, AND IMPLICATIONS Key Elements NFE Issues that Influence Assessment Appropriateness of this Research Data Collection Reflections on the Research Design Implications Future Research Recommendations Summary Reflections on this Study BIBLIOGRAPHY APPENDIX A. INDIVIDUALS INTERVIEWED APPENDIX B. INTRODUCTION AND INTERVIEW QUESTIONS APPENDIX C. TRANSCRIPT OF ONE INTERVIEW 106 110 115 117 118 118 125 126 128 130 131 133 134 136 136 139 A-l Table I. Table 2. Table 3. Table 4. Table 5. Table 6. Table 7. Table 8. Table 9. Table 10. Table II. Table 12. LIST OF TABLES Consequences of Minimizing and Maximizing Differences in Comparison Groups for Generating Theory Elements Identified by Respondents Critical Elements for NFE Assessment with their Properties Properties of Purpose Element Properties of Preparation Element Properties of Policy Environment Element Properties of Process Element Properties of Roles Element Properties of Methodology Element Properties of Descriptions Element Properties of Linkages Element Relationship Among NFE Assessment Elements and their Properties vi 40 83 86 87 91 95 99 103 106 110 115 121 CHAPTER 1 INTRODUCTION AND PROBLEM IDENTIFICATION v v’ w Accurate assessments of the current status of the education sector are essen- tial for national planning and subsequent resource allocations to be efficient and effective. Formal education has received significant attention in this regard. It is also only comparatively recently, however, that attempts have been made to sys- tematically assess the extent and nature of the wide range of activities that com- prise the NFE subsector in an education system. Only within the last two decades has nonformal education been recognized as a significant part of every national education system and a part that reaches a large number of otherwise unserved learners. For example, a working paper of the International Institute for Educa- tional Planning (IIEP ) states: Increasingly though, non-formal education has come to be regarded as important for its own sake, with the growing awareness that the school is no longer capable of satisfying a whole series of increas- ingly diversified educational needs. This diversification of needs is the direct outcome of the rapid pace of economic, technological and social change in our contemporary societies (requiring con- tinual adaptation on the part of the social factors) and of a gradual broadening of development objectives . . . . Seen thus non-formal education is no longer a simple corrective but plays a complemen- tary role. It could be that it is better suited than the school to achieve certain educational objectives. It is this new approach which has found conceptual expression in the theory of lifelong education, which signifies the encompassing relation between all the different forms, expressions and moments of the educative set. (IIEP, 1983:24) Burma This research is directed toward the development of tools that will help education planners, policymakers, and practitioners comprehend the NFE subsector. It is concerned with assessment of the subsector, that is, obtaining, organizing, and analyzing information on NFE activities for use by planners at the national level. The specific purpose of this research is to identify essential elements for an ef f ec- tive NFE assessment;1 for efforts at assessing the NFE subsector may be much less effective without knowledge of these elements. Once the critical elements have been identified it will be possible to examine existing methodologies for NFE assessment, determine their strengths and weaknesses with regard to the essential elements identified, and suggest ways by which they might be enhanced—thus, im- proving a major planning tool. A search reveals no research and no literature that focuses on identification of the elements of an NFE assessment and how these ele- ments should be addressed. Assumntiens This research is based on four general assumptions. First, nonformal education, in all its forms, is viewed as part of the education sector in any country. As part of the range of education opportunities available, NFE activities are regarded as having the potential for making important contributions to national development. Like all other education activities, nonformal education develops a nation’s most valuable resource, its people. Second, this research assumes that education planning is both necessary and desirable. Planning is seen as a process through which national priorities can be addressed. It is a process which facilitates the rational use of a variety of resources, 1In this research the term ”NFE assessment" is used interchangeably with the term ”assessment of the NFE subsector.” 3 including people, time, and money. This planning process also provides oppor- tunities for many individuals and organizations to participate in education planning. This assumption implies that education activities will improve if the planning process is used. Third, the focus of this work is based on the assumption that methodologi- cal improvements are important because they can have significant results. Research related to methodological improvements is valuable not only for the new knowledge it generates but also because better methodology has a positive impact on results ob- tained from its implementation. Finally, there are significant differences in the kinds of issues that are im- portant at the national (macro) and local (micro) levels in terms of nonformal education. It is assumed that many of the elements critical for local-level NFE ac- tivities may not be so at the national level and vice versa. In no way does this re- search suggest that the elements identified as critical f or an effective national NFE assessment are necessarily the same elements that characterize an effective NFE activity. D E . . [I An understanding of several concepts is critical to interpretation of this research. Terms defining these concepts, that may not be in common use, are used. These terms are defined below. Wm. This study uses the broad definition stated by Coombs, ”We define nonformal education as any organized educational activity out- side the established formal system—whether operating separately or as an important feature of some broader activity—that is intended to serve identifiable learning clienteles and learning objectives” (1973:“). This definition includes what Evans considers the critical elements in nonformal education which are “any non-school 4 learning where both the source and the learner have a conscious intent to promote learning” (1981:28). Sector. According to Robinson, "a sector is a complex of identified social activities treated as an interrelated system for purposes of analysis, planning and policy” (I975:5). Robinson notes that all activities of a society are related in some way and that a given sector is a matter of the boundaries chosen to define it. In this study the education sector consists of those activities that have learning as an explicit purpose. Activities where unintended learning occurs are important but not considered part of the education sector. This is because the study emphasizes ob- taining information for specific planning and problem-solving purposes related to intended learning outcomes. Even so, the definition of the education sector is broad and includes such activities as preprimary, schooling, higher education, vocational education, literacy programs, radio and correspondence programs, on-the-job training, adult education, and health and agricultural extension. Basic to the con- cept of sector as used in this study is the the assumption that the sector is a system made up of interrelated components. War, The education sector can be divided into its component parts, as defined above. Each of these components is a subsector. Thus, those education ac- tivities that are included within the definition of nonformal education together comprise the NFE subsector. Primary education is an example of another subsector. Because of the nature of nonformal education as defined earlier, the NFE subsector encompasses a wide range of activities including agricultural extension, literacy programs, health campaigns, cooperatives, artisanal associations, apprenticeships, and income-generating projects. These activities may be large or small scale, and have governmental or nongovernmental sponsorship. W. A sector assessment is defined here as a detailed description and analysis of the objectives, status, plans, needs, constraints, and 5 priority target areas with rank-ordered recommendations for action in a national education system. The purpose of the assessment is to facilitate rational planning and resource allocation throughout the education sector. It is important to recognize that assessment as here described is conceptually different from evaluation, which is a process whereby careful appraisal is used to determine significance or value. A subsector assessment is the description and analysis of a component, or subsector, of the education sector. For example, subsector assessments might address nonformal education, vocational education, higher education, or primary education. It is the composite of all subsector assessments that constitute a sector assessment. W. This study defines a systems approach to sector assess- ment as one which considers the education sector as a set of interactive and inter- dependent subsectors which together form a unified whole. These components have a shared purpose—to facilitate the development of human resources through learn- ing activities. Changes or adjustments in one subsector may effect changes in other parts of the sector. A systems approach extends beyond a recognition and descrip- tion of the component parts and focuses on the linkages within and between subsec- tors and an understanding of how and where changes in one subsector may be manifest elsewhere in the system. 11111111191!- An inventory is here defined as a list of all the activities within the education sector or subsector with data on selected characteristics. In primary education, for example, an inventory could focus on items such as numbers of schools and classrooms; in the NFE subsector, the emphasis could be on activities in areas such as agriculture, literacy, and health. The important distinction between an assessment and an inventory is that the latter does not include a focus on linkages or interactions within or between subsectors. Wm. A knowledge base on nonformal education is the shared understanding of this domain that results from human 6 thought, action, and experience. A knowledge base is made up of information units combined and integrated to produce a body of knowledge. W. This term is used to describe countries which were classified by the World Bank in 1984 as either low- or middle-income economies. Low-income economies have a 1982 gross national product (GNP) less than US. $410 per person. The average GNP for low-income economies, excluding China and India, was $280, whereas the average for middle-income economies was $1,520. This term is widely understood among development professionals. Selection of the term does not negate a recognition that many of these countries are poor because they have been subjected to the process of underdevelopment. mm. It is important to distinguish between professionals who are per- manent participants in an education system and those who are not. Insiders are practicing professionals, including educators, planners, and economists, who are part of an existing education system that is being assessed. They usually have a long-term commitment to that system and are nationals of that country. qusidsrs. Education professionals who are providing short- or medium- term services (up to a few years) in a given country, who are most often not na- tionals of the country, and who do not have a long-term or career commitment to the particular system under assessment are termed outsiders. It is the locus of their career paths and not knowledge or ability that defines individuals as outsiders. In some cases, outsiders may be more knowledgeable than insiders about the education system in question. S' ’1? There are several reasons why this research is significant, and these fall into two categories. The first category looks at NFE assessments and their 7 relationship to national education planning; the second examines what is needed to develop an adequate methodology for conducting such assessments. As educators learn more about the education processes and the environ- ments within which they occur, there is an increasing awareness that education can serve a variety of purposes. For example, the World Bank stresses that emphasis in developing countries is no longer just on increasing enrollments in formal education. Trends of ”assertion of self-reliance and national identity, broader concepts of development, and growing concern about the capacity of the system to meet the demands placed upon it” have emerged (The World Bank, l980:8). As education planners consider a wider range of education objectives, they need information on the entire sector. They need to know what activities exist, how they meet current learning needs, where there are major gaps in terms of education opportunities, and where there are major constraints. This information is important for education planners because they must meet learning needs as efficiently and effectively as possible while maximizing available education opportunities. Planners need to be provided ”with the tools to comprehend the new educational reality, and to optimize its potential without necessarily subjecting it, at its different stages, to centralized control" (IIEP, 1983:26). The recognition by planners, policymakers, and educators at the national level that the education sector is much broader than only formal education has resulted in demands for specific information on the scope and nature of nonformal education. Nonformal education is increasingly being recognized as an important component of national education systems in developing countries, yet the extent and nature of this component in a given country is often unknown. As stated by the IIEP— . . the study of educational development and planning can no longer be confined to the fields that fall within the Ministry of Education’s (or government ministries in general) sphere of competence. In all countries, there is a more or less developed net- work consisting of a multitude of diverse educational activities, 8 which play different roles and maintain de facto relations of substitution, competition, complementarity and even opposition among themselves (IIEP, 1983:25). There is no commonly used methodology for analysis of the NFE subsector and no research identifying the elements necessary for an effective assessment. Yet, at a time when some critics of nonformal education are suggesting that the field is not worth investigating, many at the highest levels of government are recognizing non- formal education’s role as a key part of a country’s education system. They are demanding more information on the subsector and how it interrelates with other parts of the education sector. Careful planning relies heavily on basic data, and it is difficult to plan nationwide education systems without a full understanding of the range of education activities and reliable information on their nature and extent. An understanding of the components of the NFE subsector and their inter- actions is important when education investments are considered. Much of the financial and human resources support for nonformal education comes from agencies based outside the country where the activities are being conducted. Regardless of the source of support, it is necessary to know where potential learners are, what their learning needs are, and what mechanisms or organizations might facilitate learning to channel appropriate and available resources toward particular kinds of learners or learning activities. Furthermore, as stated by the Academy for Educational Development— Creative conceptual work needs to be done to define how nonfor- mal education can best be supported by external funding agencies. Traditional f oci of education investment, such as school construc- tion and teacher training, are largely inappropriate. Indeed, for some NF educators any large-scale investment in infrastructure is contradictory to nonformal thinking. (Academy for Educational Development, n.d.:3) Many education planners are concerned with efficiency, or making the best use of available resources. Those concerned with resource allocation to prevent 9 duplication within the NFE subsector are addressing efficiency. Yet, it is impos- sible to even know if the efficiency of the NFE subsector needs to be improved un- less its functions and their interactions, both internally and with other elements in the education sector are understood, and any duplication and constraints have been identified. This is not to suggest that an encyclopaedia of facts is needed, but it is important that relevant data be available. Furthermore, those concerned with ef- f iciency issues find that evaluation of NFE activities, and hence priority setting, has been particularly difficult because there are so few available data, particularly data that are comparable, and data that relate costs and outcomes. Availability of appropriate data for evaluative purposes is also a need for those less concerned about the relationships between costs and outcomes, but who nevertheless wish to use data to make informed choices regarding such decisions as selection of ap- propriate learning methods. It appears that no synthetic literature on national level NFE assessment is readily available. There is a body of literature related to assessment of formal education but this literature does not address nonformal education. Most of the literature on nonformal education focuses on the uniqueness of individual NFE activities, and there is very little that summarizes or synthesizes the body of knowledge on nonformal education. Although one of the strengths of non. formal education is the uniqueness of separate projects and programs, there are many themes that are common among NFE programs. An understanding of these common themes can enhance the planning and implementation of individual activities. Generalizations have sometimes been made from projects or activities that have not necessarily been representative of nonformal education. This lack of a valid and generalized knowledge base has restricted theory building related to nonformal education. 10 An accurate knowledge base on nonformal education can serve yet another purpose. Information on the commonalities within NFE activities and generaliza- tions about nonformal education will foster a greater understanding of what is unique in any particular NFE activity. W This research identifies the elements that are essential to an effective NFE assessment that is used for education planning. This identification process will make it possible to improve NFE assessments. Two general areas need to be addressed to provide guidance for the research. First, it is necessary to examine the literature. It is important to consider the literature related to planning nonformal education at the national level to determine the extent to which assessment is addressed as a planning tool. If possible, it would be useful to identify any literature that refers to NFE assessment at the national level or to the critical elements in NFE assessment. Second, it is important to know what has been learned from practice, even if this experience is not documented in the literature. A number of NFE assess- ments have been conducted by professional educators who are knowledgeable about nonformal education. Their experiences might suggest the relative importance of individual elements in an NFE assessment. A key problem, however, is how to access, analyze, and synthesize the information and insights that have been gained through this fieldwork. W The grounded theory approach was selected to conduct this research be- cause it is particularly relevant for research in areas where there is little or no ex- isting theory and for applications-oriented research. Grounded theory uses com- 11 parative analysis to generate theory from data. The methodology was pioneered by sociologists, Glazer and Strauss, and is discussed in detail within Chapter 3, below. A set of open-ended questions was developed and administered by the re- searcher through interviews with seven professionals who have had experience with national NFE assessments. They have each participated in at least one USAID- sponsored NFE assessment in a developing country between 1982 and 1985. The countries were Botswana, Cameroon, Haiti, Indonesia, Lesotho, Liberia, Somalia, and the Yemen Arab Republic. Data were collected in contemporaneous written form and on audio cassette. They were then analyzed through comparative analysis to identify those elements and their properties that are essential for an effective na- tional NFE assessment. Interviewee selection was limited to those who had participated in assess- ments sponsored by the Bureau for Science and Technology Bureau of USAID for four reasons. First, this bureau sponsored a number of such assessments in a short period of time (1982-1985). Second, these assessments were conducted in situations which permitted comparability and generalizability, as discussed in more detail below. Third, each of these assessments had a specific focus on nonformal education. And fourth, restriction to these assessments provided access to available information. This availability was in terms of access to the individuals who con- ducted the assessments and to the documents that described their outcomes. E 1° 1.]. This study is designed so that it will have an impact on future NFE assessments. The choice of experience from NFE assessments from these countries is appropriate for several reasons. The assessments represent both the systems ap- proach and inventory approach described earlier in this chapter. The inclusion of Somalia and the Yemen Arab Republic ensures that coverage of NFE activities that 12 are unique to a particular type of environment, as they provide examples of Koranic education systems. In many ways Botswana and Somalia represent opposite ends of the spectrum in terms of national administrative structures to support non- formal education. This is important because implementation of national plans is partly dependent on a country’s administrative capacity. The countries selected exhibit a range of similarities and differences that support the generalizability of this study. None of these countries is wealthy and all face serious obstacles as they attempt to improve the quality of life of the majority of their populations. In each country, the use of more than one language often serves as a barrier to communication and learning. These characteristics are common to many other developing countries. There are significant differences among these countries. They have varied histories and cultures with a wide range of political and economic structures. Governments range from socialist, to democratic, to military, with each system con- tributing its individual strengths and constraints. The countries under considera- tion are no longer all classified as low income by the World Bank (1984). Only Haiti and Somalia are so designated. The others, Liberia, Yemen Arab Republic, Indonesia, Lesotho, and Cameroon, are classified as lower middle income countries, the average GNP of which was $840 in 1982. There are large differences between them, however. Liberia had a GNP of $490 and Cameroon $890 for the same year. Botswana is not classified because it has a population of less than one million. Thus, the findings from this study will be applicable in a wide range of national settings. Five of the eight assessments that served as the basis for this research were conducted in Africa. There are several reasons for this selection. It is generally ac- cepted that many of the countries in the African continent are among the least developed in the world. (See, for example, Accelerated Development in Sub-Saharan 13 Africa: An Agenda for Action by The World Bank.) Consequently, there is a great need for appropriate information to inform policymakers, planners, and prac- titioners with regard to the education problems they face. Various indices of development (such as health and mortality statistics, nutritional status, and levels of agricultural production) support this contention. Moreover, experience in sub- saharan Africa has established that national development is not merely an economic phenomenon but that the development of human resources is an essential element in the development process. (See, for example, Aid as Obstacle: Twenty Questions about our Foreign Aid and the Hungry by Francis Moore Lappe, et aI.). Finally, the re- searcher is familiar with a wide range of aspects of develOpment on the continent. She has lived in Africa for most of her life, studied the continent, and worked there professionally. Her work includes assignments in nonformal education and comple- tion of an NFE assessment in Botswana. Any valid methodological approach should be applicable to a variety of re- lated contexts. (This does not mean, however, that its results will be identical, or even similar, from situation to situation.) And, research results from NFE assess- ments could be useful for assessing other content areas of education with charac- teristics similar to those of the NFE subsector, for example, vocational/technical education and adult education. 1' . . E 1 S I This research is constrained by several limitations related to the ex- ploratory nature of this work and the methodology used. As has been noted, no comprehensive research identifying the key elements for NFE assessment is readily available. Even though a number of NFE assessments have been conducted, there is no research documenting why particular elements were included or a particular methodology selected. (This is demonstrated in Chapter 2.) Consequently, there is 14 no research on which to base this work. While it may be argued that it is advan- tageous to begin with a clean slate, there are many difficulties associated with working in a new area where there is little in the professional literature to guide the research. It is recognized that the existing NFE subsector assessments might be con- sidered a conceptual basis for this research. Although knowledge about them provided guidance to the experts who conducted them, it was decided that because these assessments were based on knowledgeable intuition rather than research, it was not wise to use them as the intellectual basis for this study. The study is also limited because it focuses on recent experiences in NFE assessment funded by a single sponsoring agency. (Most were cosponsored by governments, but their contributions were primarily personnel.) Imposing this con- straint had its advantages as well, as it provided relatively easy access to those in- dividuals who had conducted the assessments. The decision to work with experts in NFE assessment meant that there was a rather small pool of individuals from which to sample. This is because the field of nonformal education is small to begin with, and most individuals involved in this subsector are engaged in NFE activities at the local rather than national level. It is also necessary to recognize that this research involves individuals as the sampling unit, and as a result is subject to human bias and individual perceptions. In addition to the bias and perceptions introduced by the researcher, which are present in every research study, there is also the potential for biases from each respondent. 15 D 1° . . E 1 S 1 There are two important delimitations to this study. The first relates to sample selection and the second relates to the need to interpret the findings with caution. Individuals living in the United States who have conducted NFE assess- ments in developing countries provided information for this study. Although this sample is appropriate for the reasons provided in Chapter 3, it must be recognized that it is very selective. Accurate generalizability will be dependent upon further testing of hypotheses relating to critical elements through the use of a less restric- tive sample. For example, it would be useful to determine if developing country na- tionals would identify the same elements. It is clear from Chapters 2, 3, and 4 that this study is foundational in nature. It provides an initial glimpse into an important subject area. As such, it al- lows those concerned with national NFE planning to begin to develop a better un- derstanding of the field. Because the study is preliminary, its results should not be considered as a definitive organizational framework but rather as an initial ex- ploration of reality, the purpose of which is to provide a foundation for construct- ing an appropriate framework. Additional exploration is needed to develop theory that can be challenged and questioned. CHAPTER 2 LITERATURE REVIEW The literature on NFE assessment is relatively recent, and much of it is not easily accessed. This is partly because of the history of assessment in the education sector. Education assessments designed to describe the sector, analyze it, and present 'a strategy for action with a set of identifiable activities have been docu- mented only over the last 20 years. Many of these disregarded or only partially addressed nonformal education. The evolution of interest in nonformal education is important, and for this reason the literature review begins with a summary of assessment activities in the education sector. This is followed by a discussion of na- tional planning for nonformal education which leads into a more detailed review of nationwide assessment of nonformal education. In this review the elements addressed in the NFE assessment are not discussed. The reasons for this are given in Chapter 3, and these elements are discussed below in Chapters 4 and 5. MW Education has been viewed historically as socially good, with an underlying assumption that education is a basic human right. A correlate to this often has been that more is better, and quantitative expansion of the formal education system has been a prime objective. To improve the quality of life of populations in poor na- tions worldwide, western nations attempted to replicate the education systems they had developed and understood, and which they believed have worked for them. From this viewpoint, education could also be generally translated as schooling, that is, formal education. 16 17 In the 1960s, individuals and organizations concerned with developing and expanding education systems in Africa, Asia, and Latin America first began to recognize the need for a better understanding of the education sector. As a result, a number of education sector surveys, or assessments, were conducted. A wide range of perspectives on education sector assessment have been used. Harbison, in The Development of Nationwide Learning Systems (1973), argues that there are three basic approaches that have been employed: social demand, needs for national development, and employment generation. He notes that the second approach can be broadly specified or it can be more narrowly defined by concerns with manpower needs or economic returns on investments in education. A number of education sector assessments have been conducted over the last 25 years. It is not relevant to review all of these here, as this research focuses specifically on nonformal education. It is useful, however, to cite examples of the range of sector assessments that have been conducted in order to better understand the evolution of NFE assessment. In 1964, a Government-appointed Commission began a survey of education in India. They produced the Report of the Education Commission (1964-1966): Educa- tion and National Development, which reflects a combination of the social demand and the needs for national development approaches. Reports indicate that this highly regarded effort does consider nonformal education, but only from the point of view of its relationships with formal education (see Harbison, 1973:39). The research for the Ashby Commission report (Federal Ministry of Education, 1960) was carried out in 1959. It reviewed the entire system of formal education in Nigeria and was one of the earliest attempts to use a manpower approach. Nonformal education was not included in this assessment. Five years later USAID sponsored a sector review oriented toward national development needs and employment generation in Nigeria (Education and World Affairs, 1967). This l8 assessment was unusual in that it included nonformal as well as formal education, although in a limited way. Tanzania incorporated an education sector survey into each of its first two f ive-year development plans (United Republic of Tanzania, 1969). These surveys were conducted with very little external assistance and, in light of Tanzania’s strong desire to have its education system meet its national development needs, they are good examples of the manpower needs approach. In 1971, the Overseas Liaison Committee conducted an assessment of formal and nonformal education in Tanzania for the World Bank. The development needs approach employed was very broad and included training programs sponsored by employers, as well as formal and nonfor- mal education. In 1971, the Florida State University conducted a USAID-sponsored assess- ment of education in Korea. This assessment emphasized the economic returns ap- proach as a basis for determining education investments. It is also noted for its recommendation that middle schools change to a system of individualized education. It presents little data or analysis on nonformal education. The social demand approach was the major thrust of a number of sector assessments sponsored by the Bureau for Latin America of USAID. Robinson has written extensive analyses of the methodology, which stressed education efficiency as well as improved quality and access. These assessments were conducted in the late 19603 and early 19703. Although these methodological papers did not address nonformal education, Harbison recognized the need for this: . . . it would be desirable for every country . . . to build a strategy for integration into a more logically consistent and better functioning system of the motley assortment of unre- lated activities. Such a master plan of nonformal education would be ideal. But few countries are able to undertake so comprehensive and time-consuming a task" (Harbison, 1971: 3-4). 19 From the documentation available on the education sector assessments spon- sored by the Bureau f or Latin America, it is clear that attention was directed to the nonformal subsector. In Paraguay Education Sector Assessment 1977 (Academy for Educational Development, 1977) one of the fifteen chapters is solely devoted to nonformal education, which is also discussed in a chapter on policy analysis. It addresses, types, coverage, organization, methods, costs and financing, outcomes, benefits, and impact of nonformal education. The analysis in this assessment is based on a more detailed document that was prepared as part of the assessment. Rural Non-Formal Education Inventory and Assessment of Non-Formal Education in Paraguay (n.d.) provides detailed descriptions of 30 national NFE programs in Paraguay. An education sector assessment for Guatemala was completed in 1978. This was also conducted under sponsorship from USAID’s Bureau for Latin America. Nonformal education is addressed in three chapters. These cover the structure of the education sector, education development strategy, and current issues. In each chapter attention to nonformal education is significantly less detailed than for f or- mal education. Unfortunately, the backup documentation to Guatemala Education Sector Assessment 1978 (Academy for Educational Development, 1978) is not readily available, so it is not possible to identify how the NFE subsector was assessed. Educacion No Formal (Agency for International Development, Bureau for Latin American, 1977) provides data collected for the 1978 assessment of nonformal education in El Salvador. This document addresses conceptual aspects of nonformal education and then provides a description and analysis of available data followed by a set of recommendations. The final document on the assessment, Executive Summary of the El Salvador Education Analysis (Diebold de Cruz, 1978), addresses nonformal education in one chapter and part of a second one. 20 In the early 19703, the International Labour Office (ILO) conducted a series of three assessments based on the employment generation approach (International Labour Organization, 1970, 1971, and 1972). These studies covered more than the education sector. Their conception of the sector is broad, including nonformal education and vocational training, although these are not discussed in much detail. These studies are interesting because they place education in the larger social, politi- cal, and economic context and consider the relationships between education and employment. A comprehensive and extensive sector review was conducted in Ethiopia in 1972. The Ministry of Education and Fine Arts implemented the study with f inan- cial support from The World Bank. It included nonformal education and identified strategies to effectively link formal and nonformal education. This assessment is particularly noteworthy because of the way in which it was conducted—the majority of participants were Ethiopian. As a result of the National Education Dialogue held in Lesotho in 1978, the Government of Lesotho commissioned an Education Sector Survey. This was con- ducted by a team of educators active in Lesotho and their report, The Education Sec- tor Survey: Report of the Task Force, was presented in 1982. This addressed both formal and nonformal education and presented the government with an analysis of education problems, objectives, and strategies for the development of the education system to the year 2000. This survey is particularly interesting in that it uses data from the National Study of Non-Formal Education in Lesotho: First Report to the Lesotho Distance Teaching Centre (Adams, Bastion, and Makhetha, 1982), which was being conducted concurrently. (This study of NFE activities in Lesotho is described later in this chapter.) In 1982, USAID’s Bureau for Science and Technology began activities that formed the foundation for the Improving the Efficiency of Educational Systems II 21 (IEES) project. The first major field activity was the conduct of an assessment of the education sector in Liberia. The methodology devised for that assessment with a few procedural and organizational changes has since been used in Botswana, Haiti, Indonesia, Niger, Somalia, and the Yemen Arab Republic. The methodology was based on work by Harbison (1973), Robinson (1975), and the World Bank (1980) and is described in a draft document by Cieutat (1983). These assessments use a development needs approach and are noteworthy in that nonformal education, vocational/technical education, and management are considered subsectors of the education sector—they are viewed as interactive components of an education system. The approaches to NFE assessment vary somewhat from country to country, and these variations are discussed later in this chapter when the literature on NFE assessment is reviewed. E1 . I! E 1 E 1 . As noted in the first chapter, education planning is seen as both necessary and desirable. Much has been written about national planning for formal education. Nonformal education, however, has not received very much attention from national education planners. Individuals concerned with nonformal education have stressed the need for planning but not much has been done beyond statements of need. Moreover, the majority of NFE planners have tended to focus their attention on the planning of separate NFE activities or programs rather than on rationalizing and coordinating the range and diversity of existing activities. A first step toward planning nonformal education at the national level was the recognition of the important role of nonformal education in any country’s learn- ing system. This role was taken seriously by internationally oriented educators such as Brembeck of Michigan State University, Coombs of the International Council for Education and Development, and Harbison of Princeton University, who all stressed 22 the need to consider more than formal education when addressing issues surround- ing human resources development in support of nation building. Scholars emphasized that the existence of extensive NFE activities demanded further analysis and provided alternatives for planning national development. Case and Niehof f argue that governments "may wish to find better ways of integrating formal and non-formal education in a national policy and program of education” (l976:8). They promote sector reviews as an important first step in education planning for this purpose. For many funding agencies, the concerns of these educators regarding the role of nonformal education became particularly relevant because there was - ”evidence accumulating on the ineffectiveness and inefficiency of programs to ex- pand public school systems in developing countries," and there was a need for ”alternative efforts to meet the ‘world education crisis" (Krueger and Moulton, : l981:l). A focus on existing alternatives to formal education provided a logical next , step for those concerned with limited education opportunities in developing countries. For example, The World Bank initiated a research project on distance teaching for school equivalency (Perraton, 1982). Also, a number of other govern- ments and agencies have explored alternative delivery systems for formal school content such as interactive radio (Spector, n.d.), peer learning, and programmed learning (IDRC, 1981). Some of the early work appropriately focused on a general taxonomy of education which included nonformal education. This provided educators with a basic understanding of alternatives to formal education. The categories formal, nonformal, and informal quickly became standard (Coombs, 1973:10). A much more sophisticated taxonomy was developed, and a two volume explanation of the Inter- national Standard Classification of Education (ISCED) was published (Unesco, 1976). This hierarchical taxonomy includes both formal and nonformal education, 23 and was designed as a set of guidelines for assembling, compiling, and presenting education data by level and subject matter content. The aim of ISCED is to improve the quality and international comparability of education data, particularly with regard to the use of education statistics for manpower planning. To accompany these volumes and facilitate data collection in adult education, a Manual for the Col- lection of Adult Education Statistics (Unesco, 1978) has also been prepared. This in- cludes guidlines for collecting information on NFE activities directed toward adults. Concurrently, it was recognized that the seeming simplicity of selected terms masked definitional complexities. If Coombs’ three categories of education are viewed as lying on a continuum, for example, it is simple to order them. Deter- mining where formal and informal education stop and nonformal education begins on such a continuum is much more difficult. Similar problems exist with the ISCED taxonomy. Though a strict definition of nonformal education may not be regarded as particularly critical, a general understanding of the concept of nonformal education is critical if planning is to be meaningful. That is, it is essential to know what is being planned. A number of writers have addressed the definition of nonformal education. Kleis et al. (1974) considered the problem in some detail, identifying a range of issues to be considered in determining the parameters of a concept as dif- f icult as nonformal education. For many, Coombs’ definition (l974:8) is the easiest to use. It is the one selected for this study because it allows for the inclusion of a wide range of education activities. This does not mean that Coombs’ definition is universally accepted, however. A number of authors have argued that Coombs’ ap- proach stresses function far too heavily and that there are process considerations that must be taken into account. For example, Eiehgfg stressedthat_nonformal_a . $12931” is a @{hqéflflliilgadflpdsrmm WMLJ’LJS mcINEE. process that separates formalfrom nonformal ”education (1979:58ff). It is not 24 essential to deliberate the point regarding the definition of nonformal education here. Suffice to say that the definition selected will certainly affect what is planned. As more countries began to seriously address their education problems, and gained a better understanding of the range of existing education activities, they took very practical steps regarding the existence of NFE activities. These steps resulted in the institutionalization of nonformal education. In several instances, departments (or their equivalents) of nonformal education were created in minis- tries of education. For example, in India a Department of Adult Education was formed, and Botswana created a Department of Non-Formal Education. In the Philippines nonformal education was given even higher status, and there is a Minis- try of Non-Formal Education, Sports, and Culture. A3 countries became more deeply concerned about their formal education systems, they became increasingly aware of the potential and complexities presented by nonformal education. Plan- ners recognized that nonformal education could no longer be ignored. The challenge, however, was how to understand and organize a diverse phenomenon that was changing almost daily in any given country. First attempts at national planning for nonformal education were made early in the 19703. Brembeck of Michigan State University, in collaboration with the Southeast Asia Development Advisory Group (SEADAG), directed three semi- nars on nonformal education in 1971. Planning nonformal education was one of the discussion topics at these seminars. Two volumes containing compilations of papers presented at these seminars are useful, New Strategies for Educational Development (Brembeck, 1973) and Planning Out-of-School Education for Development (Ryan, 1972). Both contain papers on planning national NFE activities though no single paper is especially detailed or conceptually rigid. 25 In the Ryan document, Callaway, Anderson, and Coombs all stress the need for an inventory of education programs to facilitate national planning. Anderson (Ryan, 1972:31-42) argues that one of the major difficulties is that education programs and policies are heavily influenced by activities in other sectors. Coombs (in Ryan, 1972:43-55, as well as in his paper in the Brembeck book) stresses further that there are special problems presented by out-of-school education that complicate its planning. Coombs identifies some of the logistical and political issues that need to be addressed and argues that it is not feasible to achieve comprehensive planning immediately. He pleads for the identification of basic planning principles and f or the selective application of planning to key areas immediately. The International Institute for Educational Planning (IIEP) has published a series of books dedicated to the fundamentals of education planning. One of these, The Planning of Nonformalfducation (Evans, 1981), stresses the relative youth of na- tional planning for nonformal education and the need for flexibility so that NFE activities can maintain their nonformal characteristics while still operating effec- tively within the entire system of education. Evans does not present a handbook, but rather summarizes current thinking on the topic. He outlines important issues related to national planning for nonformal education, including questions concern- ing locus of control and the role of non-governmental organizations. Evans argues for a “differentiated set of planning procedures with different methods for dif- ferent categories of programmes" (1981:64). Evans believes that national planning for nonformal education at the national level should emphasize policy alternatives and quality. He does not think that there should be detailed quantitative plans but instead stresses the need for an overall education sector analysis and for the development of planning capabilities at the regional level (1981:69). In 1983, the IIEP reemphasized its interest in nonformal education. It has decided to focus its 1984-1989 research program on how education planning might 26 be adapted to the increasingly diversified field of education (1983:25). The research plan describes the need to consider the forms and dynamics of education diversification, effects of diversification, and the object and degree of education coordination. Consideration of education diversification as described by IIEP will require some understanding of education sector assessment. W In many ways, the interest in NFE assessment parallels the recognition of the role of NFE activities in national learning systems. This is because education planners soon realized there was very limited information on nonformal education. They also realized that a survey of the activities of a ministry of education would not yield comprehensive results because the majority of NFE activities in most countries are conducted outside the ministry of education. Planners recognized that to plan nonformal education they needed to understand not only the relationships between nonformal and formal education but also how ministry of education activities interacted with those of other ministries (such as agriculture and health) and of non-governmental organizations. This next part of the literature review covers assessments that focus on nonformal education and the recent NFE assessments that serve as the basis for this research. Some early attempts at NFE assessment went beyond the national level to a much larger scale. Two of these are included in this review. The first is a survey of nonformal education in Africa and the second a survey of NFE activities in Southeast Asia. In the book Non-Formal Education in African Development, the authors con- ducted a survey to 'identif y productive non-formal education programs in selected African countries" (Sheffield and Diejomaoh, l972:xi). Financial support for this activity was provided by USAID. The study was carried out to gather evidence on 27 projects related to employment that appeared to be successful, innovative, and transferable. Thus, it emphasizes activities that prepare individuals for the work force, especially in the formal sector. The survey relies on a case study approach—about 80 altogether from selected African countries. Some of the cases receive detailed treatment whereas others are described only briefly. The authors stress the value of central coordina- tion of NFE activities although they recognize fully that the majority are imple- mented by nongovernmental organizations. They also "f eel that certain base-line in- formation is essential for planning national strategies for human resources development“ (Sheffield and Diejomaoh, 1972:200). In this regard they suggest a series of variables or characteristics upon which they believe it would be important to collect data. In 1973, the Southeast Asian Ministers of Education Organization (SEAMEO) sponsored a study on nonformal education in Southeast Asia (SEAMES, 1974). The study covered eight countries and was based on a description of economic, social, and education conditions in the region. It addressed four general areas: literacy, rural development, vocational/technical skill development, and mass media in nonformal education. The study concluded with a series of recommendations. Data for the study were gathered through questionnaires sent to National Liaison Committees in each country. The authors admit that this procedure prob- ably resulted in many important programs being omitted. The study is particularly notable in that it is cross-country in nature, stresses that better education alone can- not solve economic deficiencies, and makes recommendations by subject area and type of implementing organization rather than by country. There have been a number of national assessments of nonformal education. The scope, format, and methodology have varied considerably, although recently (in 28 the 19803) there have been attempts to develop a standardized approach to, and methodology for, the national assessment of NFE activities. This final part of the literature review addresses NFE assessment methodology and then considers recent national level assessments of NFE activities. Very little published literature exists regarding methodology for national level assessment of nonformal education. A 1978 doctoral dissertation reported on the development of a survey method to identify a community’s lifelong education activities (Coles, 1978). Coles developed a survey method to inventory adult educa- tion activities and their sponsoring organizations in a given community. The proce- dure was designed to include NFE activities. It emphasized interviewing chief ad- ministrators from community organizations, and using these individuals to obtain information on activities of their own and other organizations. Coles tested her procedure in Jackson County, Michigan. A more detailed description of approaches to NFE assessment is contained in The Learning Opportunities Inventory: Preparing Comprehensive Inventories in the Field of Education (Evans, 1983). Evans promotes the need to develop comprehensive inventories of learning activities, including those in the nonformal subsector. He argues that there is a great deal of information on developing an inventory of f or- mal education activities but that more attention should be given to surveying non- formal education. Evans identifies issues which must be addressed prior to beginning a NFE assessment (such as purpose and availability of resources), provides criteria for determining the assessment’s scope (such as content and geographical coverage), and describes the various types of assessment that can be conducted (such as a simple list or a full national survey that allows for an understanding of relationships between program characteristics). He suggests that the methodology employed be either the one developed for the Lesotho national NFE survey or the one developed by 29 Creative Associates. Both of these methodologies are addressed below in this section of the literature review. One of the earliest major assessments of NFE activities at the national level was conducted in Ethiopia in 1972. As noted earlier in this chapter, this assessment was an integral part of the Education Sector Review which was conducted by a large team, most of whom were resident in Ethiopia. Two booklets, Non-Formal Education in Ethiopia: Literacy Programs (Niehof f and Wilder, 1974) and Non-Formal Education in Ethiopia: The Modern Sector (Niehof f and Wilder, 1974), describe the findings of the review. These were designed to contribute to a larger volume on nonformal education in Ethiopia. Literacy programs were regarded as worthy of separate study because they were so numerous and because literacy had an extremely high value for Ethiopians. NFE activities related to the modern sector were those that concentrate on skills training, skills maintenance, and upgrading of employment in the sector. This part of the sector review was conducted with a view toward modern sector manpower requirements. The documents do not claim to be exhaustive in describing NFE activities, but do claim to be representative. This assessment is noteworthy in that the authors move from the specific to the general in drawing their conclusions. This is particularly notable in the volume on literacy programs which addresses concerns such as female enrollments and language of instruction. This volume also raises a series of important questions to be resolved if investments in such NFE activities are to be effective. Non-Formal Education and Human Resource Development in Brazil (de Oliveira and Lewis, 1973) was prepared to improve the existing information base on nonformal education in Brazil. Its primary focus is the development of NFE ac- tivities in Brazil and on initiatives and trends. An extensive appendix identifies and describes some 'of the major publicly and privately sponsored NFE activities. 30 A number of assessments of nonformal education more recently have been conducted at the national level. Several of these have been funded by USAID and, for the reasons stated in the first chapter, have served as the basis for identifying respondents for this research. Over a two-year period, 1980-1982, an extensive study of NFE activities in Lesotho was conducted by the Lesotho Distance Teaching Centre (LDTC) for the Ministry of Education. Financial support and technical assistance for this study was provided by USAID. The purpose of the study ”was to provide essential inf or- mation and analyses regarding NFE particularly for the consideration of national policy-planners, and also for others involved in conducting NFE activities” (Adams and Bastian, 19832). The study had three objectives, namely to describe and analyze the current status, to identify major gaps, problems, and needs, and to con- tribute to the international understanding of nonformal education. The Lesotho study is a remarkable effort. It was the largest national survey of NFE activities conducted up to that date, and the research consisted of inter- views of NFE program directors through the use of a structured questionnaire developed and tested in Lesotho. The survey included 240 NFE programs dis- tributed over the 10 administrative districts. The analysis of these programs ex- amined organizational structures, participants, education methods, learning contexts, inservice training, community relations, research and evaluation, relationship to formal education, interinstitutional relations, program development, and program improvement. It is clear from this study that nonformal education is not a new phenomenon in Lesotho but one that has been active for many years and that in- cludes many traditional learning activities. A number of the conclusions of this study relate specifically to planning for nonformal education. The authors argue for policy actions that would decrease duplication of content areas in a given region, and for central initiation and 31 planning combined with decentralized implementation. The authors also call for better evaluation to guide planning decisions. Most of the recommendations, however, would require substantial financial commitments if they were to be implemented, resources that just may not be available. Between 1980 and 1983, Creative Associates, Inc., a consulting firm in the Washington, D.C. area, was contracted ”to develop a prototype method of assessing and analyzing nonformal education in developing countries" (Creative Associates, 1983:1). The model was developed and tested at two field sites: the Centre National d’Education in Cameroon and the Department of Non-Formal Education in Botswana. The model consists of five phases: planning, design, implementation, analysis, and dissemination. Upon completion of these phases, it is anticipated that there will be sufficient reliable and comparative information about NFE activities for use by education planners and staff of NFE organizations. Each of the phases is divided into several steps, and worksheets are developed to facilitate completion of each phase, where appropriate (Cain, n.d.). Unfortunately, the model does not really address analysis and interpretation of data in any meaningful way. As part of its contract, Creative Associates conducted two national NFE assessments, one each in Cameroon and Botswana. The data from two districts in Cameroon are presented in Final Report: Nonformal Education Assessment/Analysis Project by Creative Associates (1983). This report inventories 267 activities covering a range of subject areas, but data analysis is not very detailed. The results of the Botswana NFE assessment are published in An Assessment and Analysis of Non- Formal Education in Botswana (Botswana, Department of Non-Formal Education and Creative Associates, Inc., 1983). This document also provides what is essentially an inventory of selected NFE activities. It begins with a general introduction address- ing the sociocultural context. It then presents a single page of data analysis 32 followed by 37 pages of charts. The information of these charts could possibly be reorganized and used for planning purposes. As noted earlier in this chapter, USAID’s Bureau for Science and Technol- ogy began activities related to its Improving the Efficiency of Educational Systems (IEES) project with an education sector assessment in Liberia in 1982. Out-of-school activities are included in Liberia: Education and Human Resources Assessment 1983 (Liberia, Ministry of Finance and Economic Affairs, 1983). They are addressed in three separate chapters: ”Health Education," ”Agriculture Education," and "Adult Basic Education.” As with other chapters in the sector assessment, each of these chapters had five major divisions: status, needs, plans, analysis, and recommendations. The analysis and recommendations are primarily directed toward improving education efficiency. In the summer of 1983, a similar assessment of the entire education sector was conducted in Somalia. In this case, however, four chapters covered NFE activities. In addition to chapters on ”Health Education,” "Agriculture and Live- stock Education,” and ”Adult Basic Education," a chapter on 'Koranic Education'I was included. Each chapter has a similar structure to those in the assessment of Liberian education. The findings are documented in Somalia: Education and Human Resources Sector Assessment 1984 (Somalia, Ministry of National Planning with USAID, 1984). Later in the same year, another team was sent to Botswana to conduct a similar assessment. Treatment of nonformal education differed significantly, however, from the two previous assessments. Two areas of divergence are of particular interest. First, rather than presenting information in separate chapters, nonformal education was considered as an integrated subsector, with a number of component content areas such as health, agriculture, and income-generation activities. This approach resulted in an analysis that crossed content areas and was 33 based on data from the private sector, seven ministries, and the Office of the President. Second, within the status section of the chapter consideration was given to specific elements of each of the component content areas such as health and life skills and income generation. For example, data were published on characteristics of NFE activities, such as learners, trainers, content and method, and facilities and support services, so that it was possible to aggregate information but also to compare within the subsector and between the NFE subsector and other subsectors (e.g., primary or vocational/technical education). The results of this assessment are pub- lished in Botswana: Education and Human Resources Sector Assessment, 1984 (Botswana, Ministry of Finance and Development Planning and the Interministertial Steering Committee with USAID). In 1984, a team conducted an education assessment in the Yemen Arab Republic under the aegis of the IEES project. An assessment of nonformal educa- tion was included in this activity. The results of the NFE assessment were" presented in a way modeled after the Botswana NFE assessment—NFE activities were considered an an integrated subsystem. Attention focused on the national‘ literacy campaign, district training centers, local development agencies, and the V Yemeni Women’s Association. Outcomes of this assessment are published in Yemen ; Arab Republic: Education and Human Resources Sector Assessment 1985 (Draft) ' (Yemen Arab Republic, Educational Research and Development Center with IEES, 1985). Two more assessments have been conducted under the IEES project: one in Haiti and one in Indonesia. The assessment of the NFE activities in Haiti was limited to those conducted through government. It also used the same approach as the one in Botswana—NFE activities were considered as components of an NFE sys- tem and treated as such in a single chapter. A draft document, Haiti: Education and Human Resources Sector Assessment 1984 (Institute for International Research, 1985), l 34 is available. The IEES assessment in Indonesia focused on a single, major govern- ment NFE program. No documentation is available on this assessment at this time. 8mm It is only recently that NFE assessment for planning purposes has received attention from scholars and practitioners. And, it is important to note that most of the attention has come from practitioners. Their work has generally be based on the need for immediate information. It has seldom been supplemented by rigorous thinking on the long-term implications, the elements selected for data collection, or the assessment methodology employed. The majority of the national NFE assessments conducted to date are not exhaustive. Yet, there is seldom any rationale for why some activities were selected for inclusion and not others. It is clear that selection in many cases is biased and frequently not systematic. This can be partly justified by the motivation for col- lecting the data, but this justification cannot ignore the fact that decisions may then be made with only partial information. Of particular interest is the evolution of NFE assessment methodology as demonstrated by the literature. It has changed from simple inventories to detailed information along with a sufficiently detailed description of relationships within and between programs to support policy decisions. Thus, the practice of NFE assessment is becoming more and more useful as a tool for education planners. CHAPTER 3 METHODOLOGY AND DATA COLLECTION The purpose of this research is to identify the critical elements for assess- ing nonformal education at the national level. This chapter describes the method selected to identify these elements, indicates how it is implemented, provides a jus- tification for its appropriateness, and presents details of the data sources and process used to apply the methodology in this research. W This research is based on a grounded theory—that is, "the discovery of theory from data systematically obtained from social research” (Glaser and Strauss, l967:2). Grounded theory uses an inductive rather than a deductive approach in its initial stages. It uses the particular to develop the general—it evolves from the part to a whole. Emphasis is on the process of discovering theory. A deductive approach is more commonly used in social science research and results in a conclusion that comes from reasoning derived from a series of premises. These premises are accepted, taken for granted, and not questioned. The deductive approach em- phasizes verification of existing theory as opposed to the discovery of new theory. Theory provides a basis for handling data. It gives a conceptual framework for examining, describing, and explaining reality. It should also provide categories and hypotheses that can be used and tested in subsequent research so that the theory becomes more useful for explaining reality and, ultimately, predicting outcomes. This relationship between theory and reality is especially important. The various explanatory categories in a theory must ”be readily applicable to the data under 35 36 study” and able to explain the existing situation (Glaser and Strauss, 1967:3). But, theory must have a rational basis and be tied to data. Thus, Glaser and Strauss sug- gest that the best approach to generating a theory is ”an initial, systematic discovery of the theory from the data of social research” (1967:3). The newly generated theory can then be subjected to the more traditional approach of testing and refinement that is based on deductive procedures. Thus, the grounded theory approach produces concepts and hypotheses that emerge from the data and that "are systematically worked out in relation to the data during the course of the research" (Glaser and Strauss, 1967:6)—-the theory developed is experience based. The research is not solely limited to observational data, however. As with any research, the ideas, suggestions, and innovations may come from a variety of sources. Insights from other sources must, however, be syn- thesized with the available data. Concepts and hypotheses cannot be inferred randomly from the data. There must be some systematic way of analyzing the data. Grounded theory relies on com- parative analysis as a strategy to facilitate the discovery of both formal and substan- tive theory. Formal theory refers to a conceptual area. An example from adult educa- tion might be learning behavior. Substantive theory is developed for an empirical area of inquiry. NFE assessment is an example of a substantive area of inquiry. This distinction between formal and substantive theory is important. Generation of substantive theory requires ”comparative analysis between or among groups within the same substantive area” (Glaser and Strauss, 1967:33). A focus on formal theory demands comparative analysis among ”the different kinds of substantive cases which fall within the formal area, without relating them to any one substantive area” (Glaser and Strauss, 1967:33). 37 Comparative analysis is a well known and commonly used research tool. The grounded theory approach requires that comparative analysis be based on data collection, classification, and analysis carried out together, to the extent possible. The joint collection, classification, and analysis of data is a process termed theoreti- cal sampling. "This process of data collection is controlled by the emerging theory, whether substantive or f ormal" (Glaser and Strauss, 1967:45). Initial decisions for data collection are based on a general perspective on the subject or problem area. Only decisions regarding initial data collection can be made with great clarity. Emerging theory and its obvious gaps identify the next steps or suggest subsequent research questions. Outcomes of previous data collection influence and guide what happens next in data collection and analysis. Selection of what to pursue next is based on theoretical purpose and relevance. As the collected evidence mounts during the research process, a broad range of indications may become acceptable for categories and properties. It is through this process that conceptual categories and their relations among the categories and their properties are developed. Wm. Categories and properties emerge from the data as elements of a grounded theory. According to Glaser and Strauss, "A category stands by itself as a conceptual element of a theory. A property, in turn, is a con- ceptual aspect or element of a category” (1967:36). Although they emerge from the data, it must be noted that they are separate from the data—they are indicated by the data and the generated categories and their properties remain even if the data change. W- During the collection, classification, and analysis of data, and the identification of categories with their properties, generalized relations between groups being studied become apparent. These initial, generalized relations are sug- gested hypotheses. Continuous examination of the data comprises the start of the verification of the hypothesis. At the same time, immersion in the various aspects 38 of processing the data for theory development results in integration of appropriate and meaningful aspects: ”In the beginning, one’s hypotheses may seem unrelated, but as categories and properties emerge, develop in abstraction, and be- come related, their accumulating interrelations form an integrated central theoretical f ramework—the core of the emerging theory. The core becomes a theoretical guide to the further collection and analysis of data. (Glaser and Strauss, 1967:40) This approach demands that the researcher possess theoretical insight into the research area, an ability to make something of insights, and a temperament that is able to cope with data before it is classified and categorized. In fact, it is essen- tial that the researcher not be committed to a particular organizational structure or explanation, but rather be open to a range of conceptual possibilities. 111W As noted earlier, the concurrent collection, classification, and analysis of data is fundamental to the conduct of grounded theory. This process is called 1 Knowledge about the problem area to be studied provides a theoretical sampling. partial framework for the structure and processes that are to be applied. In non- formal education, for example, it is likely that concepts such as learner, facilitator, and content area will be utilized. What is not known is the relevance of each of these concepts to the problem. These initial concepts are referred to as local concepts by Glaser and Strauss. Local concepts provide a base for the research. It is quite possible, through the research process, that some of these concepts may not turn out to be particularly useful when applied to the problem under consideration. Existing knowledge, concerns, and issues that have defined the research problem also help define initial decisions regarding data collection. They provide lTheoretical sampling is very different from statistical sampling. Theoreti- cal sampling is conducted to identify categories and their properties. Statistical sampling is employed to obtain accurate evidence on distributions to be used for description or verification. 39 the questions that are posed at the beginning of data collection for comparative analysis. The emerging theory, that is findings and existing knowledge gaps, guides subsequent steps in data collection, organization, and analysis. An important proce- dure in the research process is the identification of groups for data collection (a group may be an individual) and next stages in the process. Initial selection of groups will depend on whether a conceptual or empirical area of inquiry is under consideration. In the case of research on an empirical area, comparison should be between or among groups of the same substantive area. In the study of a conceptual area, where development of a formal theory is anticipated, groups should come from substantive areas which embrace the concept under study. It is important for the researcher to be aware constantly of the types of groups to be compared to control their effect on the scope and conceptual level of the theory being generated. Selection of comparison groups is based on "their theoretical relevance for furthering the development of emerging categories. The researcher chooses any groups that will help generate, to the fullest extent, as many properties of the categories as possible and that will help relate categories to each other and to their properties” (Glaser and Strauss, 1967:49). The ultimate number and types of groups selected for data collection is determined during the research process and finalized when the research is complete. Comparability of groups is not relevant. Whereas comparability is very important when accuracy of evidence is at stake, the only es- sential requirement for implementing theoretical sampling is that the data contained within the groups undergoing study apply to a similar category or property. This approach allows a great deal of flexibility in selecting groups. The criterion of theoretical relevance, however, serves to control data collection. It en- sures that the data collected are useful and avoids wasting time collecting unneces- sary data. As noted above, group selection provides control over the conceptual level and population scope. It also provides "simultaneous maximization or 40 minimization of both the differences and similarities of data that bear on the categories being studied (Glaser and Strauss, 1967:55). The consequences of this are noted in Table 1. Table 1. Consequences of Minimizing and Maximizing Differences in Comparison Groups for Generating Theory Differences Data on Category in Groups Similar Diverse Minimised Maximum similarity in data leads to: Spotting fundamental differences under which (1) Verifying usefulness of category; category and hypotheses vary. (2) Generating basic properties; and (8) Establishing set of conditions for a degree of category. These conditions can be used for prediction. Maximized Spotting fundamental uniformitiee of greatest Maximum diversity in data quickly forces: scope. ( 1) Dense developing of property of cate- goriee; (2) Integrating of categories and prop- pertiee; and (8) Delimiting scope of theory. Source: Glaser and Strauss, The Discovery of Grounded Theory, 1967, p. 58. Three terms are important to an understanding of theoretical sampling: theoretical saturation, slices of data, and depth. Theoretical saturation is reached when no additional data are found that further define categories and their properties. It is reached by the joint collection and analysis of data and is the point at which the researcher stops sampling the different groups. The data collected on a category may represent different slices of data, comprising a range of types and views of the category obtained through a variety of collection techniques. These different kinds of data highlight different aspects of the category for the researcher. The amount of data collected on a category is termed the depth of theoretical sampling. Time and its passage are important factors in theoretical sampling. Concur- rent collection, classification, and analysis of data result in very different temporal 4l aspects than traditional research. In traditional research data collection, classification, and analysis usually occur sequentially. When they occur concurrently, it is more difficult to assess immediately the status of the research process. The researcher is continually engaged in systematic classification and analytical note taking in the search for emerging categories, reformulation of categories, and rejection of those that fail to consistently accommodate the data. In addition, the methodology requires that the researcher step back from the research for uninterrupted reflection and analysis. Thus, there is an alternating tempo to the research process. Data collection has the initial priority, but classification and analysis gradually require increasing time. The final part of the research is primarily analysis with brief attention to data collection and classification to tie up loose ends. i n T f hi This part of the chapter addresses the relevance of grounded theory methodology to the current study. It also addresses some of the major criticisms that are directed at the approach. As it is a distinctive research methodology similar to ethnography and the case study, it is considered unnecessary to justify it as a research procedure. As the present areas of inquiry were being developed and as the initial literature review proceeded, it became apparent that this research would focus on a new domain of study, and that there were no precedents upon which to base deci- sions regarding the choice of a research methodology. Further exploration of the literature confirmed that elements used in prior NFE assessments had been chosen primarily through intuition rather than from research. It soon became clear that grounded theory was an appropriate research tool. No theory exists regarding the critical elements for NFE assessment, and the area 42 under consideration is an applied field. Moreover, it is important that the findings from this research have an empirical foundation. And, the existing literature does provide a basis for initial questions and identifies a research sample. According to Merriam and Simpson, ”grounded theory is particularly suited to investigating problems for which little theory has been developed” (1984:99). As has been noted in the first chapter, no formal theory exists in the area of national level assessment of nonformal education, so it is important to use a methodology that can accommodate the existing knowledge base. In fact, Glaser and Strauss would argue that use of the approach in these circumstances is especially important because it shows the value of the methodology to generate theory in "nontraditional areas where there is little or no technical literature” (1967:38). The activities of nonformal education in developing countries are primarily related to basic needs such as health and well-being—it is practical, and related to real world needs and concerns. The purpose of this research is to improve the plan- ning and use of those limited resources that are available for NFE activities. It is essential, then, that the research methodology selected be one that is designed for use in an applied field. As noted earlier in this chapter, this research generates sub- stantive theory—theory addressing an empirical area of inquiry based on reality. The grounded theory approach is particularly well suited for this. In fact, Darken- wald would argue that the major purpose in employing grounded theory in an ap- plied field "is to improve professional practice through gaining a better understand- ing of it“ (Long et aI., 1980:69). Theory generation is conducted with an existing knowledge base as its foundation. The literature on nonformal education, most of which describes learn- ing programs and activities, assists with the integration of the categories and their properties into a meaningful whole. Knowledge of the existing literature and NFE 43 activities worldwide contributes to the process of collecting, classifying, and analyz- ing data. The value of the interaction between reality and theory in the grounded theory approach should not be underestimated. More traditional approaches to re- search emphasize the verification of hypotheses or a theory. Real world data are collected and analyzed to estimate how well the hypotheses or theory explain reality or how well they can be used for prediction. Grounded theory requires that the concepts and hypotheses comprising the theory emerge from real world data. Real world data are also used to verify, further develop, and clarify concepts and hypotheses. Thus, the grounded theory approach relies very heavily on the rcsearcher’s moving back and forth between reality and theory—perhaps even more than is required in more traditional approaches which usually focus on testing the theory rather than stating or restating it. Because nonformal education addresses basic needs, it is critical that re- search findings that direct or modify these activities be relevant and valid, and not result in program changes that could endanger the lives of those whom they are designed to assist. Theoretical findings must be accurate, relevant, immediately applicable, and enduring. Theory generated by the grounded theory approach is usually valid. As Glaser and Strauss note, ”theory based on data can usually not be completely refuted by more data or replaced by another theory" (l967:4). The exten- sive lifetime of a grounded theory is especially critical to the field of nonformal education. A major strength of grounded theory is that it does not assume that existing explanations or assumptions are necessarily correct. This is because it does not begin with a series of assumptions that are accepted without question. The deduc- tive approach that relies on premises often leads researchers to focus on minor problem areas within large theories rather than address the relevance of the theory 44 itself. Grounded theory allows movement beyond these traditional limitations. In addition to providing a means for developing theory where none exists, it provides an appropriate methodology for theory generation where existing theories do not provide sufficient explanation or accommodate the existing data. In this approach data are not collected to satisfy some existing, preplanned set of procedures that are followed even though evidence indicates that redesign would be appropriate. This methodology escapes the problems inherent in following preconceived notions during the research process. Although there is no substantive theory or even a set of stated propositions regarding NFE assessment, there is a body of literature that addresses the nature of nonformal education. Within this literature there are numerous, usually unstated, assumptions about the field. Because most of these assumptions have been derived independently of the others and few, if any, appear to have been developed systematically, it is important that they neither restrict nor hamper the process of developing a better understanding of national level NFE assessment. Furthermore, the selected approach allows different kinds of data to be used in generating theory. This flexibility is especially important for nonformal education, where comparable data are frequently not available. Application of the grounded theory approach to this research will produce what Glaser and Strauss describe as categories and properties. The categories will be those elements that are critical for effective NFE assessment at the national level. Again, it is stressed that these may not be the only critical elements for ef f cc- tive NFE assessment. The properties will be the conceptual aspects that describe these elements. In the analysis section of Chapter 4 and in Chapter 5, the terms element and category are used interchangeably. 45 W An important first step was to decide what kind of data were to be collected. Given the lack of existing literature, it was decided that recognized and accessible sources of information on NFE assessment were those individuals who had actually conducted such assessments in developing countries for education planning purposes. These individuals comprise the richest data sources with theoretical relevance to the research questions under consideration. Next, criteria were identified for the selection of NFE assessment experts for comparative analysis. To qualify, an individual had to have been responsible for or participated in an NFE assessment sponsored by USAID in a deve10ping country between 1980 and 1985. Using these criteria, nine individuals were identified as potential groups for the analysis. Seven of these were interviewed. Their names and affiliations are in Appendix A. Each respondent’s professional career is summarized below; the in- dividuals are not identified. Although the term informant is commonly used in eth- nographic research, the term respondent has been selected for this study. 3mm earned her Master’s degree in Political Science from the American University of Beirut. She is fluent in Arabic and French. Since 1974, Respondent A has been engaged in professional activities related to development with a special emphasis on women in the development process. It was the involve- ment with women’s education that led her to nonformal education. She has worked in the Arab Emirates, Egypt, Fiji, Indonesia, Iraq, Jamaica, Jordan, Kuwait, Lebanon, Liberia, Mauritania, Morocco, Senegal, Sierra Leone, Somalia, Sudan, Tunisia, the Yemen Arab Republic, and Zaire. A has been involved in designing and implementing NFE field projects. She has also managed large staffs who are engaged in overseas projects. She has conducted NFE assessments in Somalia and the Yemen Arab Republic. 46 W received his Doctorate of Education from the University of Massachusetts in 1983 in the fields of Adult and International Education and Curriculum Development. He is fluent in French and Spanish. B’s professional work in education in developing countries dates back 20 years to the time when he served as a Peace Corps volunteer in Senegal, where he was involved in agricultural and literacy work. Since that time he has worked in Afghanistan, Cameroon, Ecuador, Honduras, Iran, and Tunisia, as well as in the United States. While in the United States, most of his professional activities have been international in nature. Most of respondent B’s work has been in nonformal education. Emphasis has been in the areas of adult literacy and the development and provision of train- ing programs. He has also been responsible for producing publications in support of NFE activities. A major contribution to NFE assessment by respondent B was the development of a model for NFE assessment and its implementation in Cameroon. W was trained as an anthropologist. She received her Doctorate of PhiloSOphy from Stanford University in 1968. Respondent C is fluent in Spanish, speaks French, and has some knowledge of Portuguese and Zapotec. In addition to working in several universities in the United States, Respondent C has worked in Colombia, France, Lesotho, Mexico, and Puerto Rico. Respondent C has had a wide range of responsibilities in education. These include research, project design and implementation, policy design, and program development. She has had major responsibility for two national level NFE assessments—one in Colombia and the other in Lesotho. W received his Doctorate of Education from the University of Massachusetts in 1979. While there, he was based in the Center for International Education. Respondent D has a working knowledge of Nepali and Bahasa Indonesia and has conversational ability in Hindi and Maithali. His first international work was as a Peace Corps Volunteer in Nepal—in fish farming and agricultural 47 extension. He served a long-term assignment in Indonesia, where he was responsible for a major national NFE project that was sponsored by the Government of Indonesia and the World Bank. He has worked for shorter periods of time in India, Liberia, and the Philippines, . Nonformal education has been the focus of respondent D’s professional work. Within the general area of nonformal education he has focused on literacy, income generation, rural development, food production, and the contribution of women to development. While in the United States his primary responsibilities have been to provide technical and administrative assistance to field projects. This in- dividual was responsible for an assessment of NFE activities in Liberia. W is another individual whose first international experience in education was obtained through the Peace Corps. While attached to the University of Rwanda, she also developed and managed a bread-baking cooperative with a rural women’s group. Respondent E possesses two Masters Degrees, in Linguistics and Nonformal Adult Education, from Michigan State University, where she is a doctoral candidate in adult education. She is fluent in French, and speaks good Swahili and fair Spanish and Kinyarwanda. She has also worked in Haiti and Zaire. In addition to her practical work in nonformal education overseas and in the United States, Respondent E worked for four years at the Non-Formal Education Information Center, Michigan State University, where she focused on many of the conceptual aspects of nonformal education. She conducted an assessment of NFE activities in Haiti. W received his Doctorate of Education in International Educa- tion from the University of Massachusetts in 1981. He has a good working knowledge of French. His initial work in international education was as a member of UNESCO’s staff in France. Since that time he has worked in a number of over- seas settings: Cameroon, The Gambia, Haiti, Indonesia, Lesotho, Morocco, Nigeria, 48 and Swaziland. His work in the United States has included technical and ad- ministrative support for field projects in nonformal education. Much of respondent F’s work in nonformal education has been in the area of literacy. He has engaged in field work and research related to the acquisition and utility of literacy in developing countries. This individual has extensive ex- perience in the area of assessment. In 1980 he participated in the subsector assess- ment of vocational and technical education in Morocco and undertook a feasibility study for a national assessment of nonformal education in Cameroon. More recently, he participated in an education sector assessment in Haiti, and he con- ducted an assessment of nonformal education in Indonesia. W possesses a Masters degree in International Education from the University of Massachusetts and is engaged in doctoral studies related to the evaluation of adult education. Like several other respondents, her first experience with education internationally was as a Peace Corps Volunteer; she served in Honduras, where she was involved in literacy training and adult education. Other countries in which she has worked are Botswana, Cameroon, Lesotho, and Zimbabwe. Since 1979, Respondent G has been professionally active in the area of non- formal education, particularly with regard to literacy and educational programs in support of the participation of women in development. She worked on an assess- ment of NFE activities in Cameroon and was responsible for one in Botswana. W In light of the decision to use individuals to provide data on NFE assessment, it was clear that the use of interviews would be a relatively simple way to access these sources. Furthermore, and consistent with the methodology, the in- terview process would allow the researcher to probe any areas as guided by the data collected and analyzed up to that point in time for any particular interview. The 49 most appropriate data gathering technique for the grounded theory approach was the open-ended interview. This part of the chapter addresses the interview process. It describes the preliminary instrumentation, preparation for the interviews, and their general conduct. The next chapter contains specific information related to in- dividual interviews. W- Prior to scheduling the interviews, a series of questions was developed and tested. Questions were designed to allow respon- dents to give their professional opinions about NFE assessment without having to defend their own field work. They were designed to facilitate clarification, to en- courage the organization of key elements, identify major problem areas encountered while conducting NFE assessment, clarify the role of the assessor, and urge ref lec- tion on the part of the respondent. These questions served to begin the interviews and to guide the first ones. Existing NFE assessments, knowledge about nonformal education, and the author’s own experience conducting an NFE assessment in Botswana provided the local concepts for the questions—the initial concepts, based on existing information, that are thought to be relevant to the study. The initial questions and the statement which introduced them are in Appendix B. A transcription of one of the interviews is given in Appendix C. The first question was deliberately very open to enable respondents to iden- tif y key elements for NFE assessment with no suggested framework or point of view. The second question served to clarify and allow the researcher to check that she had interpreted the responses accurately. It also encouraged respondents to note why the elements they selected were important. The third and fourth questions had essentially the same purpose—asking the respondents whether they had other items or issues they wished to identify as critical to NFE assessment. The fifth, sixth, and seventh questions encouraged the respondents to group elements together. It was anticipated that responses to these questions would assist 50 in determining the differences between elements and their properties as defined in the grounded theory approach. The eighth and ninth questions distinguished which of the key elements identified the individuals had used for assessment, and which ones, upon reflection, they felt they should have used. Question 10, in concert with question 8, identif ed how one measures the presence or absence of an element and for which elements it was particularly difficult to obtain data. The eleventh question elicited information on those elements that had been used in an NFE assessment but which, in the respondent’s professional judgment, were not essential. Because the study uses respondents who had worked in countries other than their own, it was thought that it would be useful to have a sense of each respondent’s role in the assessment in relation to host country professionals and institutions. Question 12 elicited this information. The final question, number 13, gave the respondents an opportunity to describe if and how they would like to see NFE assessments conducted differently. WW. 0f the seven individuals who participated in the research, two were interviewed in person and the remaining five were inter- viewed using the telephone. In all cases, the experts had been contacted at least two days prior to the interview and had agreed to discuss their opinions on the key ele- ments for a national NFE assessment that is conducted for planning purposes. Each interview lasted a minimum of one hour, and all respondents indicated a willingness to talk with the researcher again if that would be helpful. Interviews were scheduled at least one full day apart to allow the re- searcher time f or reflection, classification, and analysis of the data collected to that point in time, and to prepare for the next interview. 51 W3. This section of the chapter gives an overview of the interview process. Its major purpose is to give a sense of the utility of and general response to the original questions. The open-ended approach to the interviews proved very fruitful. The first question would often elicit as much as 20 minutes of comments from a respondent. At this time the individual would generally identify key elements and indicate why they were important, thus often making it unnecessary to ask the second, third, and fourth questions. The fifth, sixth, and seventh questions, which encouraged the respondent to group elements into categories, were particularly useful in that they allowed some of the respondents to reply in the negative. Other respondents would present the key elements in groups when responding to the first question. Responses to the questions did not provide clear and easy access to information on the properties of categories. It soon became clear that the eighth, ninth, and tenth questions were not likely to yield much new data. They attempted to determine whether assessors were aware of elements that were important but that they had not included in their work. As with questions six and seven, these questions addressed issues that were so impor- tant that they usually were mentioned during the response to the first question. Question ll, which asked about elements that had been used but might not be par- ticularly useful, was usually addressed as part of the Opening statement. Question 13, which asked how a respondent would do things differently if given an oppor- tunity to conduct another assessment, was also addressed in the initial statement most frequently and so did not have to be asked very often. The twelfth question provided many interesting responses. It asked about the role of the assessor in relation to host country professionals and institutions. Al- though this question was seldom anticipated by the respondents (unlike many of the 52 other questions), individuals talked openly about their desire for better counterpart relationships and their frustrations with the situations with respect to host country collaboration. Posing a few questions and listening to extensive answers without having to engage in much prompting resulted in the identification of several additional areas for questions as the interviews progressed. Questions on these areas were not asked of every respondent as they resulted from statements made by those who had al- ready been interviewed. This is discussed in the next chapter, as this data collection occurred concurrently with data analysis. The emotional responses to the question about counterpart relationships prompted further questions about the roles of host country nationals in NFE assessments. In particular, the researcher was interested in what would be an ideal situation. While exploring these issues, it became clear that the respondents put host country education planners in a very different category than other host country professionals. This resulted in a specific question that asked about the appropriate role of host country planners in an NFE assessment. The original set of questions did not address preparation of the expert counterparts for the assignment. Yet this was a major concern of the respondents. The introduction of an open-ended question that asked for the respondents’ ref lec- tion on their preparation for the assessment resulted in fairly detailed expositions on problems faced and what kinds of preparations had been provided for. The comments also included some creative suggestions for improving preparatory activities. During the first two interviews it became apparent that there were a num- ber of concerns expressed related to the policy environment in which decisions about nonformal education are made. This resulted in the addition of a specific question to the remaining interviewees that asked whether attention to the policy 53 environment was important in an NFE assessment. Finally, it must be noted that similar questions were asked differently as the interview process evolved. For example, question five on the questionnaire is "Now you have given me __ elements. Organize them for me." The last in- dividual interviewed was one of those who gave the key elements in categories. In her case the question was phrased, "You indicated that you would present the key elements in terms of the analytic framework that you were required to use in the assessment you conducted. In retrospect, how useful do you see that framework? As noted, seven individuals were interviewed for this study. The decision to stop after the seventh interview was made because the sixth individual identified no new elements. Had theoretical saturation been reached? The seventh was inter- viewed to test this assumption. The interview with the seventh respondent did not elicit either any additional categories or properties so the researcher was reasonably confident that all the categories had emerged during the interviews. Summarx This chapter has described the grounded theory approach and shown how it is implemented. It has documented why this approach is an appropriate methodol- ogy for this research. In the final sections of the chapter the data sources were presented and the interview process described. The following chapters present and analyze the data and contain the conclusions of this study. CHAPTER 4 PRESENTATION AND ANALYSIS OF DATA This chapter presents and analyzes the data used in this research. The or- ganization and content of this chapter is unusual in that it not only presents the data but also demonstrates the research process. This is consistent with the grounded theory approach which demands that the ongoing research process be modified by data collection and classification. As a result, the presentation of the data is mainly comprised of a summary of each of the seven interviews in the order they were conducted. The information within each interview is presented in the same order it was given by the respondent. Two exceptions exist. Where inf orma- tion was repeated within an interview, it is dealt with in one place to facilitate the organization of the data. And, if the researcher had to ask how the individual would do an assessment differently, that is recorded at the end. After each inter- view summary a short section records the researcher’s responses and outcomes of those responses. This section points to the initial identification of apparent critical elements, or categories in Glaser and Strauss’s terminology, emerging from the re- search process. This is followed by an analysis of the data, presenting the elements for NFE assessment, with their properties. PRESENTATION OF DATA The seven interviews are summarized below. The respondents are iden- tified by numbers and not by the letters used to identify them in the previous 54 55 chapter for reasons of confidentiality. The first five interviews were conducted by telephone and the last two in person. W This interview was conducted on 10 December 1985 at 9:45 am. and lasted 1 hour and 25 minutes. Wm. This person’s responses were based on experiences with two NFE assessments. Respondent One opened the discussion by saying that critical elements would vary according to the size of the assessment, its goal, and the planning and national context in which the assessment was being conducted. She felt that it was essential at a minimum to have a listing by geographical area of all NFE activities, an estimate of the number of participants in each, and qualitative information on the programs. This qualitative information should cover such things as goals and objectives; organization and management; program content, methods, media, and materials; training (pre-service and in-service) and supervision of personnel; prior, current, and existing research; monitoring, feedback, and f ollow-up systems; and characteristics of participants, such as ethnic group, level of prior education, age, gender, and socioeconomic class. The planning of and preparation for the assessment were viewed as critical parts of the process. Planning included design of the research process and the ques- tionnaire that was used to collect data on NFE activities, preparation of the person- nel who would administer the questions, and supervision of the interviewers. In this regard, the respondent noted the importance of people in the process—not only have they participated in the assessment, but there have been two other important out- comes of the assessment process. These individuals have translated their assessment experience into policy—there has been impact as a result of the assessments. Moreover, the assessments have been learning processes for those involved, and there 56 is evidence that these individuals have transferred their learning to other situations. She stressed that it takes many people to make a study functional and that the im- portance of their participation should not be overlooked or undervalued. Respondent One stressed that it would be ideal to get as much detail as pos- sible on every aspect of every element for every NFE activity but admitted that human and financial resources limited this approach in reality. She suggested that dynamics within a single project are important but that they do not seem as impor- tant at the national level. She suggested that there is a place for case studies or the use of some kind of sampling procedure to get a better sense of the depth and tex- ture of NFE activities. She also suggested the need for related studies on topics such as the characteristics of paraprofessionals and volunteers, exchange networks, and the flow of information to meet planning needs. This respondent stressed that she had collected data on even more elements than those we discussed in the interview in the assessments in which she had been involved and that she saw a need for more information related to communications and on the linkages between nonformal and formal education. Of the elements described, she felt that information on project goals and objectives was often the weakest, even though this was an extremely important element. Also, the nature and flexibility of many NFE activities makes it difficult to realistically assess their duration and intensity in ways that are comparable with other NF E activities. The first respondent felt it was important to do a preliminary study before conducting an assessment in order to have a sense of the range of activities. She felt that this would contribute to the development of a useful framework for con- ducting the assessment. Respondent One felt that the type of NFE system in exist- ence would partly influence the kind of assessment that could be conducted. For example, she suggested that in a large country with a highly developed centralized 57 system of government, it might be better to do separate, regional assessments so as to be able to take major, cultural, and regional differences into consideration. She also stressed that there is a significant difference between an inventory and an assessment. Inventories were seen as an important first step but lacking analysis and therefore limited. A preexisting commitment to the NFE assessment from within the country was viewed as essential for a successful assessment—financial support for the ac- tivity from an external source is simply insufficient. In-country support that in- cludes local institutions and participants is critical. She believes that participation is one of the aspects that separates assessments from inventories. Respondent One stressed the difficult role that she had in bringing the assessments to completion. She felt that she had to expend a lot of energy to get the various agencies and factions to become part of the assessment process. She felt she played an important role bringing people into the project, engaging in public relations, building information bridges in-country, and politicking among different interests. In this regard she felt that one should not underestimate the need for skills to deal with individuals from another culture as well as with a range of people. She stressed the importance of the human skills of the person guiding the assessment. If asked to do another assessment, this individual would develop a much shorter questionnaire, emphasize addressing those points needed for policy purposes, and pay even greater attention to the people involved in the process. W A large proportion of the interview focused on the structure and content of NFE activities themselves—the parts or components that might be used to describe an NFE program. She emphasized the pe0ple involved in the assessment, discussing aspects such as training host country nationals to conduct data collection activities and the delicate but important role that an outsider must play in an NFE 58 assessment. A fundamental issue was the learning process that occurs during an NFE assessment. Comments about the role of the outsider suggested a need to have some understanding of the political environment in which both the assessment and decisions regarding nonformal education are made at the national level. The em- phasis on the need to plan the assessment process and to ensure that the process would focus on items and concerns of particular interest to the host country sug- gested that preparation and purpose are important components of the assessment. Her notion of the significance and possible use of a preliminary study suggests ques- tions regarding the methodology employed. Questions asking Respondent One why she felt certain elements were im- portant were considered unnecessary by her. She felt that in her response to what were the elements she had given ample justification of their importance. The ques- tions that attempted to get her to categorize elements did not yield much, as she felt that she had given them initially in their priority order. Nor did the questions designed to identify critical elements that she had not used elicit further information, as she felt that she had used all the critical elements plus additional ones. She did not show any enthusiasm for other approaches that had been used for NFE assessment. W The second interview was conducted on 12 December 1985 at 10 am. and lasted for 1 hour. Wm. This individual opened the interview by providing the critical elements in four areas—beginning with two that he did not use in the assessment he had conducted. He opened the discussion by emphasizing the need for a description and analysis of the present policy environment and a history of the development of policy within the country where the assessment is to be conducted. 59 He feels that it is essential for an assessment to include a description of the mechanisms for changing policy, for whatever the findings of the assessment, the process for making changes must be known. The second key area that he had not addressed but now feels is critical is the need for an understanding of the planning process and especially the mechanisms for planning NFE activities at the national level. The third category of elements that he introduced were the issue areas that are addressed in the methodology used in the Improving the Efficiency of Educa- tional Systems (IEES) project. These are internal efficiency (the optimal use of the resources within a given program or system), external efficiency (the relevance of a program or programs within a system), administration and supervision, costs and financing, and access and equity (access refers to the proportion of a target p0pula- tion participating in a program whereas equity is the degree to which participation in benefits is not restricted to circumstances beyond an individual’s control). Finally, he argued that it was important to include NFE activities sponsored by nongovernmental as well as governmental organizations. At the researcher’s request, Respondent Two elaborated on the five elements that fell into the third category. Under access and equity he indicated that it was useful to gather inf orma- tion on topics such as instructors’ and participants’ gender, caste, and ethnic group. He also believed it important to consider the appropriateness of materials and methods for their target populations and to reflect back on issues such as the value of particular content areas for these individuals. For example, the value of literacy for men and women may be different, and the available staffing and the type of training they have received reflects back on the higher education system. He stressed that it is important that external efficiency not concern itself only with economic analyses. He believes it is important to assess the social value of skills that individuals are learning and to investigate motivational issues, as he feels 60 that many NFE activities are out of touch with reality and are offering learning Opportunities that may not be particularly valuable for a given target group of learners. He suggests that external efficiency issues should consider the relationship of NFE efforts to overall country goals and that literacy programs should be evaluated in relation to the entire schooling system. Measures of achievement are required for a sense of the degree of internal efficiency of NFE activities. These are rarely available, and it is difficult to arrive at them in the short period of time generally allowed for an NFE assessment. An area where internal efficiency could be improved significantly would be for central government to make much better use of the NFE activities of nongovernmental or- ganizations and by providing these agencies with access to curricula, materials, and training programs in return. Respondent Two suggested that major improvements could be made in ad- ministration and supervision by better coordination among agencies and sharing of central resources. Nonformal education tends to use many temporary staff as facilitators, and attention should be devoted to this use of resources for short-term programs and employment. He views positively the central service agency approach, where one agency in a country provides NFE programs a range of services such as needs assessment and evaluation, because it improves internal efficiency. He recog- nizes the importance of different constituencies and how NFE activities can serve them but feels that a centralized organization does use resources better. Reliable data on costs and financing are hard to obtain. There are ample budget data, but this is mostly for costs paid by central government. There are al- most no data on local financing. NFE activities do provide a range of funding sources, however. Pluralizing the implementation of NFE activities provides a much wider base for finanacing education opportunities. Another way to expand 61 financing possibilities is to have good materials and programs, as these result in immediate participation and act as a stimulus to local groups. It is very difficult to do cost/benef it analyses, particularly as there are no data on the value of benefits derived from NFE activities. But, it is essential to ex- amine adult education and its relationship to factors such as better health and agricultural practices. For adult education we must assess secondary as well as im- mediate benefits. It is known that the educational level of parents is critical in en- couraging them to send their children to school (especially girls). It is important to examine parental participation in NFE programs in relation to their willingness to finance educational activities (especially primary school) for their children. These secondary benefits probably have more value than the immediate benefits, and yet little attention is given to them. This individual did not have a national of the country as a counterpart. He worked in three ministries (health, agriculture, and education) and felt that he had to force individuals from these agencies into counterpart relationships during the process in which he took his findings back to them for review and comment. In one case there was a great deal of resistance and defensiveness in response to his work. Respondent Two felt that he had a very difficult leadership role while conducting the assessment. With regard to the interactions with other individuals engaged in assessing other subsectors, this respondent felt that their discussions were very rich and that many innovative ideas that should have been shared with people in the ministry were lost. If he had an opportunity to repeat the process, Respondent Two would have structured the assessment differently to allow for better preparation on his part, more time for data collection prior to his arrival, formalized interaction between 62 the assessment team and ministry officials, more time for data analysis, early review of the findings in-country, and a better counterpart relationship. W The respondent began the interview by placing the critical ele- ments in categories. He began with a clear sense of the policy and planning en- vironment as an essential element for NFE assessment. He defined methodology as an important element by using an existing framework which forces data analysis along certain areas to identify a series of categories. Within this analytical framework he addressed NFE program descriptions, mentioning such items as structure, function, learners, and outcomes. He also noted that it was important to consider the entire spectrum of NFE activities and suggested the need to examine the linkages among them. He identified the importance of a counterpart and noted the difficult role that he had to fulfill and the value of other members of the team. Like the first respondent he raised the issue of preparation, but he also included concerns about logistics in this area. The way Respondent Two structured his response to the first question streamlined the interview because he anticipated several of the subsequent questions. As he started with categories, questions 5 and 6 were unnecessary. Similarly, questions 2, 4, 8, and 9 were not necessary. His response to question 10, about difficulties he encountered in obtaining data, raised some interesting issues that pointed to the relationships between formal and nonformal education. As with the previous interview, the response to question 11 about other approaches that had been used for NFE assessment was lukewarm. Other approaches that differed sig- nificantly in terms of their focus on other elements or use of a very different methodology were not seen as particularly worthy or interesting. Both this respondent and the previous one placed a great deal of importance on the roles of counterparts. Consistent with the grounded theory approach, it was 63 decided to add a question to the remaining interviews which would ask the respon- dent who they believed should conduct NFE assessments. W This interview was conducted on 16 December 1985 at 2:15 pm. and lasted for 1 hour and 5 minutes. W. This person’s responses were based on experiences with two NFE assessments. She stressed that the primary purpose of an assessment should be to involve people in the educational process and in planning education. She also emphasized that the purposes of NFE assessment will vary according to the level of development and degree of understanding of nonformal education in the country. In addition to describing and validating the concept of nonformal education, management structure and the number of participants are fundamental elements that must be addressed initially to be able to identify and classify existing projects. A second important area is to obtain aspiration or goal statements for the NFE activities. This respondent was still very conscious of the difficulties she had encountered, even though several years had passed since the assessments. When pressed, she identified two other areas that she felt were important for data collection. These were professional staff and age of participants. She identified problems with obtaining local participation in the assessment process, morale and time problems once participants had been identified, logistical difficulties, and lan- guage barriers. A focus on the second assessment, which had been a little easier than the first one, yielded several other items for consideration in an assessment. A basic aspect is to get timely information on such items as number of participants, content, 64 educational level of trainers (because this is related to employment opportunities), relationship to formal education (because credentials are so important), program description, names of program personnel (especially the Director), mailing address and directions, kinds of materials, and strength of the program (because many NFE activities exist mainly on paper or in pe0ple’s minds). She also indicated that it was important to listen to what the individuals in- country wanted to have included in the assessment and that this should guide data collection. For example, in one assessment she gathered information on number and entry level of participants, overall program structure, number of beds, and available meeting space. She noted that the most difficult topic to get information about is the magnitude and texture of the NFE activities, but that this is important in order to determine where investments should be made. She recognized that there are usually multiple clients for a single survey and that different people will require different kinds of information. What is essential is to know where detail is impor- tant and to provide that information if it is requested for good reason. For example, expatriates in a country are usually interested in the sources of sponsor- ship because of issues surrounding the distribution of financial resources. There are instances, however, when there are not enough funds or time for the assessment to fully examine all of the questions that might be asked. When asked about other assessments, Respondent Three raised a number of issues. She is concerned that there tends to be very loose use of terminology (for example, statistical terms such as population and sample are misused) and that this leads to a pretense of 'scientificness." She also argued against conducting the data analysis out of the country as she feels that too much is lost in the process if in- country people are not involved in this activity. A major concern was the amount of information contained in an assessment document. She feels that policymakers are too busy to read vast amounts of information and that large amounts of 65 information often contain many inaccuracies. It is more important to have a small amount of information that people agree is accurate so that planners can focus on valid data. This individual felt that she really struggled with the counterpart relation- ship and this was confounded by time and financial limitations on the project as well as by administrative problems. Nevertheless, she felt that she served to bring a number of counterparts along. In response to a question about how she would change things if she were in a position to do them differently, she suggested that it would be essential for in- dividuals in-country to want to have the assessment conducted. She also believed it was important to recognize that westerners seem to need definition much more than other cultures and that next time she would not have the analysis done by a group but instead would have it be the responsibility of one individual who would consult with others during the process. A final question about who should actually do NFE assessments elicited a strong response that they should be firmly grounded in an NFE center in-country. Outcomes. Two categories permeated Respondent Three’s responses: pur- pose and process. In her opinion the assessment must be conducted because the country wants it—suggesting the concept of in-country ownership of the assessment. She also stressed that the learning that occurred within the assessment must have high priority. She mentioned items relating to NFE program description, suggesting that it was most difficult to get at aspirations and at the magnitude and texture of activities. Reference to analysis activities suggests that methodology is important. Emphasis on the difficulties encountered with counterpart relationships indicates that the role of participants in the process deserves attention. During the interview it seemed that it would not be useful to ask the ques- tions that attempted to get the respondent to categorize or organize the elements 66 (numbers 6 and 7). Because of her emphasis on purpose and the uniqueness of each assessment situation, and her statement that she was being pushed too hard for structure in the first few questions, the researcher decided not to pursue this line of questioning. Question 8 was not necessary, as items she mentioned as critical were identified with regard to their relationship to the specific purpose of the assessment. She was critical of several other approaches, suggesting that they were not sensitive enough to process and purpose issues related to NFE assessment. Her response to the new question about who should conduct NFE assessments was interesting. Although she had expressed great frustration with regard to counterpart relationships, she stressed that assessments should be based in a local institution that is concerned with nonformal education. It is already clear that those who have conducted NFE assessments have taken it very seriously. Their immediate recall of small details and their responses suggest that they are still closely identified with the activity they conducted. It is not merely a professional assignment that was completed and then f orgotten—perhaps this is a result of the learning process. W The fourth interview was conducted on 18 December 1985 at 9:10 a.m. and lasted for 1 hour. [Mew—Summary. This individual cites three major elements in NFE assessment: source of initiation and ownership, level of control (which relates to ownership), and the conceptualization and guidance provided by the funding agency and the contractor (the organization conducting the assessment). It is important to know who initiated the activity and for what reasons. If the assessment is exter- nally initiated, it is essential to spend significant amounts of time creating a sense of ownership among the organizations and individuals who will be involved in its 67 conduct. Ownership can occur at a variety of levels, but the individuals must per- ceive the need for the information and should value the integrity of the data for it to occur at all. It is impossible to coerce pc0ple to make a commitment to an activity. This develops over time as benefits are perceived. Ownership of the im- plementation of an assessment is particularly difficult to achieve. In-country level of control is important because this is what builds local in- stitutional capability. The more a local institution controls, the greater the likelihood that the skills used in the assessment will be learned and applied again. Of course, the level of control is partially dependent on the general capabilities of individuals in the local institution. The other side of this is the extent to which the outside agency controls or influences the sector assessment. Respondent Four was involved in two assessments—one in which an attempt was made to leave all control to the local institution and a second where the contractor assumed most of the responsibility for data collection and analysis. The third critical element, according to this individual, is the need to look at the evolution in thought (or conceptualization) on the part of the funding agency in terms of assessing the NFE subsector. In particular, what has been learned that can make a difference to the conduct and outcome of the next assessment? When pressed by the researcher for other areas in which one might collect data, he indicated that it would be important to only collect information that would be used for decisionmaking. There is a need for more than listing the number of NFE activities. There is a need for more depth, more descriptive information on topics such as methods, materials, and outcomes. Unfortunately many NFE groups cannot give that kind of information in a way that can be readily synthesized by an outsider. Respondent Four believed that much more attention should be given to categories within which data is collected. He had believed that he had been 68 exhaustive in his work but found there was a need to return for more in-depth case study analyses. He believes that an assessment should allow for data gathering in more than one way but stressed that this requires very careful planning so those ac- tually conducting the research understand how it all fits together. This individual felt that the existing definition of nonformal education is an important question. An area where it is very difficult to get accurate data is that of instructional methods and materials that are used. The interpretations of responses to questions in this area vary greatly depending on the training of those administering questionnaires and gathering data. In general, he felt that assessments should make better use of open-ended questions but admitted that the ability of researchers is important in this regard. There are three areas that he believes have not received sufficient attention in NFE assessment. The first is that there should be more emphasis on ethnic perceptions of nonformal education. The second is the amount of borrowing that occurs between nonformal and formal education. The third is the history of NFE activities. He believes that it is important for planners to be aware of the how and why of an NFE program—who initiated what kind of activity and with what kind of par- ticipant involvement. When asked how he would conduct an NFE assessment if there were less funds available than he had for his previous work, he indicated that he would use a different methodology. He would identify a group of people to conduct in-depth studies of fewer activities that represent a fair geographic spread. The study would include questions such as what are the minimum resources necessary for sustaining the activity? what effects would increased resources have? what kinds of non- financial assistance would be necessary if the program were to expand? what grow- ing pains has the activity experienced? and what kinds of problems can be 69 anticipated as a result of increased investment? One area where there is a lack of data relates to village-level financing. The assessment document must be interesting reading. It cannot be too long. Planners need to have information that has been synthesized into action recommendations. Although the process for doing this can be implemented by an outside organization, it should be carried out internally. There were two areas where Respondent Four found it particularly dif- f icult to gather data. These were in the areas of program finance and instructional methods. In the former, people did not want to respond to questions about the amount of. funds coming into their programs. And there was a lack of comprehen- sion regarding the variety and classification of different methods of instruction. If he could do it differently, Respondent Four would make changes in two areas—management and ownership. Quality control was a management issue that emerged. This individual would insist that all organizations involved in the assess- ment agree on the objectives of the assessment and that reference points would not be changed without consensus. A principle of nonformal education is that activities be learner directed. Therefore, it is essential that a national organization own, control, and conduct the NFE assessment. For example, the research tools, such as questionnaires, should be developed by those who are going to use them. There is a real need for the agency funding the assessment to assure that the activity receives sufficient logistical and moral support. In this regard, he felt that USAID- supported assessments should be funded by the local Missions and not centrally from Washington, as the latter puts any implementing agency in a difficult position be- tween two parts of the same bureaucracy. Finally, Respondent Four stressed the important educational role that out- siders who are asked to conduct NFE assessments must play. 70 911mm. As a result of the discussion, six major areas of concern emerged: purpose, methodology, roles of participants, linkages, process, and the NFE programs themselves. The first five were volunteered by the respondent and the sixth discussed in response to prompting by the interviewer. Because Respondent Four provided the elements in categories, it was not necessary to ask question 5, and question 6 was asked very differently. Rather than asking him to categorize the elements in terms of their importance for others who would be doing NFE assessments, he was asked how he would address the critical elements if he had less funds available for an NFE assessment. This resulted in greater detail about how one might go about an NFE assessment methodologically differently. Questions 8, 9, and 12 were not asked because he had already addressed those issues. The new question about who should conduct NFE assessments had also been answered as part of his other responses. In discussing participation, he addressed the learning environment as a fundamental part of the assessment process. Although he detailed a wide range of participants who should take part in the assessment process, it was interesting that he did not mention planners. The potential role of planners from the country in the NFE assessment was explored in subsequent interviews. His comments about control of and methodology used in an assessment raised further questions regarding preparation. It was decided to add a question about preparation to subsequent interviews. One of the most interesting aspects about this interview was that the respondent was not at all defensive about his work—he seemed to be able to assess it critically without difficulty. 71 W This interview was conducted on 20 December 1985 at 4:25 pm. and lasted for 1 hour. W. This individual’s responses were based on the two NFE assessments she had conducted. She began the interview by stressing the utility of the IEES methodology because it provided a helpful framework for assessment. Within the methodology she suggested that it was important to consider seven elements, some of which can be further divided. First, it is important to examine what the curriculum of an NFE activity offers. One must also consider the participants. Aspects that might be considered include location (rural/urban), gender, and age. A next step is to consider the relationship between curriculum and the target learners. In Morocco, for example, curricula designed for adult women were being offered to young girls. It is also necessary to consider the teachers and the level and kinds of training that they have received. This is a useful indicator of how the curriculum is being implemented and allows the assessor to consider issues of the educational process at this stage. Probably the most important element in Respondent Five’s opinion is how well the program reflects daily life—does it relate to the needs of the population? She also argues that it is important to examine the relationships between the elements cited so far and to consider location and timing of NFE activities as part of the relationship, as these are often not emphasized sufficiently. The sixth element she identified was decentralization. That is, when and where the activities take place and whether they were centrally or locally designed. Finally, one needs to consider whether activities are public or private. When they are publicly sponsored, the range of relevant ministries must be considered and inclusion of private sponsorship means being very open to the entire range of NFE activities. 72 She stressed that other elements or aspects may need specific attention in a given country. For example, nomadism can be another dimension of the concern with target populations. Also, the breakdown of elements may differ considerably from country to country. Respondent Five emphasized again the value of the IEES methodology which focuses on five issue areas: external efficiency, internal efficiency, access and equity, management and administration, and costs and financing. Upon reflection, this person did not believe that any of the elements she mentioned are more or less important than any others. She was concerned that at- tention is frequently given primarily to curriculum without examining the other elements or the relationships among elements. Major difficulties she encountered re- lated to locating data on costs and financing of NFE activities and with data that were not sufficiently disaggregated. Age and gender data were particularly problematic with regard to the latter difficulty. Other areas where Respondent Five sees major gaps relate to data on reten- tion rates and impact for NFE programs. Both of these have significance for state- ments or conclusions regarding the outcomes of nonformal education. She believes that information on the number of registrants in a program is not enough—it is im- portant to know how many complete a course and how many complete a cycle. Without this it is impossible to evaluate the success of a program. Related to this is the lack of information on how the program improved the quality of life of the learners. Quality of life should be measured in both economic and social terms. The question regarding her role in the assessments in relation to other professionals and host country institutions was of particular interest to Respondent Five. She found working with a team of colleagues an interesting assignment. One important aspect of this was her role educating others on the team, responsible for assessing other subsectors, about nonformal education. Local counterparts brought 73 specificity about nonformal education in their country to the assessment. They also brought to her attention the diversity of sponsors of NFE activities and played the role of catalyst, widening the scope of her thinking during the assessment process. She found that local institutions often were not clear about nonformal education and as a result did not give it any status. She played an important role dialoguing with representatives from various institutions about nonformal education. Also, in many cases countries define nonformal education very narrowly. The NFE expert has the important task of raising consciousness about nonformal education—its depth, breadth, and potential. She felt that she was not adequately prepared for the task she was assigned and that she had to learn on the job. Regardless, she believes that it is important to train counterparts but that in her experience it is unlikely that they will be able to conduct a neutral assessment on their own. She feels that there are many pressures on national professionals and they will be forced to be more compromising in their analysis and assessment. She believes there could be a role for a national planner on the team but this would depend on the person’s substantive understanding of non- formal education. anmmes. Respondent Five used the framework provided by the IEES project to present the key elements. This approach places major emphasis for data collection on NFE program description. Results of the interview with her indicate that roles is another important category, including the roles of the outsider NFE assessor, counterparts, and team members. The interview also confirmed that preparation is essential to a successful NFE assessment. Comments about decentralization suggest a concern about the overall structure of the environment in which NFE activities operate. Her response to the question about the roles of NFE planners from the country in the NFE assessment was interesting. She believes that it depends on the 74 planners and their orientation to the field of nonformal education, further em- phasizing an underlying concern by respondents that planning and planners of non- formal education are acceptable to the extent that they do not undermine the nature and key characteristics of nonformal education. Only two questions were not asked: numbers 9 and 13, which addressed key elements she had not used in her work and how she would change the assessment. These were omitted because she had already provided relevant information else- where in the interview. W The interview with Respondent Six was conducted on 26 December 1985 at 11 a.m. and lasted for 1 hour. W. This person had conducted two NFE assessments and been a close observer of two others. He began by identifying four key concerns: simplicity, availability of data, accuracy of data, and preliminary work. He felt that it is critical that an assessment not be too elaborate nor should it contain information that is not particularly useful. If either of these weaknesses exists, then the assessment is more likely to alienate than facilitate. Sufficient data are important to conduct an assessment. Often there are less data than it originally appears, and it is very difficult to conduct an assessment in their absence. And, even if data are available, they may not be particularly useful. For example, they may be inadequate for the questions that need to be answered. Or, the quality of the data may not be reliable or the information may have been collected in ways that pose constraints on their use. Preliminary work is essential. Existing documentation and an inventory of NFE activities can be very valuable, and it is important to examine the relevance of NFE activities. He does not necessarily recommend an inventory if one does not 75 already exist, suggesting that it might be better to conduct studies on a smaller sample. An inventory is not policy oriented and only adds reams of data. He indicated that an assessment should consider who is participating and the nature of that participation, internal efficiency (including time on task, dynamics, and availability and quality of materials), and external efficiency (where short f ollow-up might be needed). When prompted to consider the variables that are considered in the IEES project assessment methodology, he indicated that he thought all of these should be considered, but that some might need adaptation. What he regards as critical for nonformal education is the need to have measures that can be compared. For example, time engaged on a learning activity. It is also important to be able to place things into perspective. That is, to establish benchmarks for what is possible given a particular investment. It is essential to have realistic expectations from nonformal education. For example, 100 hours of learning will not produce a reader or a writer, but it might have a significant effect in improving child care. It may mean that people have to be argumentative in order to force others to have realistic expectations or to make wise investments. A major error made by NFE programs is to raise expectations and suggest that they can be met by tailor-made programs. Often, however, not enough is known about community needs. And even if needs are well understood, this does not mean that the NFE program is necessarily equipped to meet them. Needs assessments may be conducted even though the program offered is essentially what already exists. Rather than providing bureaucrats with a meaningless needs assess- ment activity, it might be better for programs to simply offer whatever it is that they do best. It seems that small scale community efforts may be more effective than centrally managed programs. Maybe assessments should look at why and how 76 programs are successful with a view to investing in those that seem to be most effective. A major problem is that nonformal education has become bureaucratized. Central coordination of nonformal education has not proven to be particularly useful. Agencies designed to provide NFE services (such as USAID’s service agency concept) tend to provide employment for bureaucrats who do not add much to program effectiveness but do use a large amount of resources. These individuals may not have any experience with what is actually taking place in nonformal education throughout the country. Often assessments are for these people. This raises the question of the purpose of the assessment. In discussing an approach to address these elements, Respondent Six thought the IEES issues-oriented approach was quite workable, particularly if non- formal education was regarded as part of a larger educational system. He felt that the process within a program was not highly important at the national level and that time should not be devoted to investigating process in an NFE assessment. He sug- gested that one could take a more streamlined approach, for example, by employing a design that did not require cost or enrollment data. He noted that there is a trade- of f in being tied to a particular structure—if you examine good programs, determine why they work, and get a measure of impact, you would get different kinds of in- formation than from employing some other methodology. Respondent Six felt that which elements are important depends on where the assessment is being conducted. He suggested that Haiti required significant time and resources to collect primary data, whereas data were available in Indonesia, but they were probably greatly exaggerated with respect to topics such as enrollments. He stressed that it is important to identify ways to triangulate, or cross-check, data to be able to form conclusions that are more reliable. Triangulation is important because it is easy to be misled by enthusiasts and one’s own misinterpretation. 77 When asked about the roles of counterparts, he indicated that they made a major contribution in his most recent experience. He added, however, that he felt it might be feasible to conduct an assessment without counterparts. The existence of counterparts adds complexity to the process. It also adds to the utility of the assessment, and counterparts often provide valuable access. It provided a training opportunity for the counterpart and justified the presence of an outsider engaged in the assessment task. It also provided a safe outlet for counterparts to be critical of NFE activities. There was some potential for conflict, however, as nonformal education was located in a different part of the ministry than the host institution for the sector assessment activity. Those at the head of the NFE unit were not eager to release information that might show ineffectiveness or a need for a shift in focus, as this might have budget implications. In retrospect, he felt his recent experience with counterparts was ideal. He worked closely with the host country unit. He served to transfer skills, lend credibility by being from an external source, and look into areas where insiders could not while still using their knowledge and information. Host country nationals participated in the analysis and made recommendations. The end product must belong to the host country. He feels the outsider can stimulate and act as a catalyst. The assessment is useful for several reasons. It is visible. It gets the ear of people at the top of the bureaucracy, and it can lead to a more rational approach to education. The nature of the assessment does not necessarily lead to changes, however. In order for an NFE assessment to be more likely to influence policy changes, there is a need for data on measures of program effectiveness, costs in rela- tion to effectiveness, and relationships between investments in formal and nonfor- mal education. Respondent Six does not believe that it is appropriate for assessors to know where decisions are made. 78 He believed that he received a good briefing. A cultural briefing is espe- cially important. He would have liked more time to become better acquainted with existing documentation. He feels that much of the learning has to be done on the job. Logistics are important although often not noticed if the support is there. If he were planning an assessment, he would arrange the timing differently. He would use the expert to organize the data collection task, have the local institution do the data collection and analysis, and have the expert return to discuss conclusions and recommendations. In this way it would be more of a partnership study. W. Five elements were clearly identified during the interview with the Respondent Six. These were purpose of the assessment and the assessment document, preparation, program description, methodology, and roles. In the discus- sion about program description, his analysis suggested a relationship between NFE programs and the structure within which they operate, emphasizing concerns about the policy and planning environment. Issues related to the policy and planning en- vironment were discussed several times during the interview but Respondent Six had a somewhat conservative stance toward this element. For example, he saw the roles of in-country planners solely as consumers of the assessment and not as contributors. It was unnecessary to ask three questions, as he had already provided responses to them earlier in the interview. Two of these related to essential ele- ments that he had or had not used in his own work and the third concerned how he would conduct the assessment differently if he had the opportunity. He had already provided responses to these questions earlier in the interview. When asked about other approaches to NFE assessment, he was extremely critical of inventories. At this point in the series of interviews, no new elements were being men- tioned by the respondents. The same elements in different combinations were being cited as critical for an effective NFE assessment. It was judged necessary to 79 conduct one additional interview to ensure that the domain of information had been completely defined. W This interview was conducted on 4 January 1986 at 1:40 pm. and lasted 1 hour and 20 minutes. 113mm Respondent Seven began by saying that she wished to answer the first question using the IEES organizational structure that she had used when she conducted the NFE assessment because it provided a useful framework. This framework first requires the preparation of program descriptions, which are then followed by an analysis of needs, constraints, and issues. The analysis is then used to derive a set of conclusions and recommendations. Program descriptions require a number of elements to be examined. The organization and management of each program, for example, should be described. Related to this, one should identify the program’s philosophical approach. Program planning and curriculum development are particularly important. A close examina- tion of the curriculum indicates how learners are participating. How is the cur- riculum developed, by whom, what content, and why this content with respect to the audience? It is also important to obtain descriptions of the participants. Who are the facilitators and how are they trained? Are they trained in the content of the area in which they are teaching? Who is training the facilitators and how are they trained? Have the facilitators received training in NFE methods? One must be aware of how NFE programs are evaluated and what evalua- tion findings are available. Related to evaluation, one should examine the program benefits. That is, who benefits from the NFE program and in what ways. It must 80 be recognized, however, that evaluation studies do not always emphasize or even address benefits. Determination of what kinds of programs should be included in the study will probably be done by the country in which the assessment is occurring. Com- monly selected programs include health, agriculture, basic education, community development, and any related to income generation or helping people economically. The relationships between nonformal and vocational/technical education are also important. Under the IEES methodology an assessment must address needs and con- straints of the entire subsector. This is especially difficult for the nonformal education subsector because one has to aggregate data from such a wide range of programs. The discussion of issues is an important part of the assessment because it is this aspect that ties the assessment together. An important issue that is not specifi- cally included in the IEES methodology is participation. In the discussion of participation, attention should be given to the ways in which learners participate in all steps of the program, the relationship of levels of participation to benefits gained, and the issue of individual versus group participation. Access and equity is a critical issue for nonformal education. It is rela- tively easy to design NFE programs for those who are less economically disadvantaged, but it is more difficult to provide learning opportunities to those who need them most. Access must include geography, how pe0ple are reached, and associated costs. Equity must consider factors such as gender and age. Costs and financing is an important area because there are generally very limited funds available to support NFE activities. There is a need to examine the sources of funds and help to identify creative ways to finance nonformal education. 81 One can use the concepts internal and external efficiency to identify some of the important issues. But these can be misleading concepts because they are often used in a more economic sense in other subsectors. Activities and outcomes of non- formal education cannot be as easily quantified as other subsectors because of scar- city of reliable data. The problem is one of knowing what is the amount of gain or effectiveness from a given program, but we generally have very little information about levels of achievement. There is a need to clarify what these terms mean for nonformal education. For example, it might be very efficient in some cases to reach just a small group of learners. Respondent Seven stated that there should be a preliminary study of the government’s development strategy. It would then be possible to assess how dif- f erent NFE programs are consistent with this strategy, and the role that nonformal education is playing. For example, is it adversary to or supportive of overall development strategies? There should be an overall descriptionlof the subsector with a view to the organization and management of its component parts. Particular attention should be given to the relationships between governmental and nongovernmental activities. A next step should be to examine the linkages between formal and nonformal education. In response to a question about the policy environment, Respondent Seven indicated that she believed it is a difficult but important area on which to focus. Examining the management of each subcomponent is internal, but the more global sense might also be useful. It could also be counter to the basic principles of non- formal education. In part it depends on the source of funds for nonformal education—funds must be managed and accounted for. With local or smaller sources of financing, the policy environment might be less important. 82 Upon reflection on the methodology that she used, Respondent Seven said that she would have preferred to design her own structure for the assessment. The IEES framework was useful, however, given the time constraints she faced to com- plete the assignment. It did not force too much and it does make cross-country com- parisons easier. This individual made several observations with regard to the question about counterpart relationships. She felt other members of the team were a real help. Being part of a team gave her more legitimacy. It provided camaraderie and sup- port and a way to share daily experiences and gather additional information from her colleagues. As an outsider with no prior experience in the country in which she worked, she felt she was sometimes seen as a representative of USAID and as a pos- sible source of funds. Sometimes people thought their programs were being evaluated and as a result deferred to her. It would have been more effective to have a counterpart who could have given more depth, provided more information, and given the report recommendations greater legitimacy. She would have liked to have a small advisory committee to meet periodically and provide her contacts with professionals working in the country. She believes this would have created a stronger sense of ownership of the assessment. She still stated that it was good to have an outsider conduct the assessment. This allowed for objectivity and provided someone who had no preconceived ideas, represented no vested interests, and was not answerable to any organization in the country. If given the opportunity to conduct another assessment, she believes that she should have better preparation—specifically more advance notice of the assign- ment and more time to do background reading. She would be more aggressive about interviewing people and spend more time defining the scope of work so that all 83 interested groups and individuals would have ample opportunity to clarify the pur- pose of the assessment. W In this extensive discussion, Respondent Seven identified six fundamental elements for NFE assessment. These are NFE program descriptions, methodology, policy environment, linkages among NFE programs and with formal education, roles, and preparation for the assessment. Under NFE program descrip- tions she emphasized that the concept of participation was usually overlooked even though it was fundamental to the successful operation of a given NFE activity. All questions were posed except questions 6, 7, and 8, as she had answered these earlier in the interview. The interview with Respondent Seven identified no new elements. As with Respondent Six, elements mentioned earlier were repeated, albeit in different ways and with different descriptions and justifications. Table 2 lists the eight critical elements and indicates which respondent con- tributed to their identification. Table 2. Elements Identified by Respondents Respondent 1 Beam" 1234567 Purpose x x x x x Preparation x x x x x Policy environment x x x x Process x x x Roles x x x x x x x Methodology x x x x x x x NFE program descriptions x x x x x x x Linkages x x x x x It must be emphasized that this table is the final product of data collection, classification, and analysis. Although the respondents are identified with particular elements, it does not mean that those individuals specifically mentioned that 84 element. In some cases they alluded to elements or discussed issues that related to properties of these elements. And, in some instances respondents discussed properties as if they were elements. In the end, however, it was data from these statements and comments that resulted in the identification of the elements. ANALYSIS OF DATA This part of the chapter presents the findings of the research described in Chapter 3 and is based on the data presented earlier in this chapter. It begins with a brief introduction that summarizes the remainder of the chapter. The purpose of this summary is to provide the reader with an overview of the analysis and to facilitate the reading of the text that follows. Upon completion of the last interview, it was clear that the interview process had identified eight key elements. The eight elements that emerged are as follows: purpose, preparation, policy environment, process, roles, methodology, NFE program descriptions, and linkages. These are a direct result of the comments and statements made by the seven NFE experts who were respondents in this research. The purpose category addresses the reasons for conducting the assessment, which will determine much of its content. Preparation relates to the range of preparatory activities that make an assessment possible. The policy environment is the context within which the assessment and decisions that af feet nonformal educa- tion occur. How the assessment is conducted is the major concern of the process category. The roles category is a clear recognition that many individuals with dif- ferent points of view participate in an NFE assessment. The analytic structure of the assessment is the content of the methodology category. The program description category is self-explanatory and the linkages category concerns the relationships among NFE activities and between nonformal education and other activities in the education and other sectors. 85 Identification of these eight elements is not sufficient, however, for the purpose of this research. Application of the grounded theory approach requires that the properties, or conceptual aspects, of each of these elements be identified as part of the research process. Some of the properties that define the critical elements were clearly described by the respondents during the interviews. Identification of others required more systematic examination of the data. Every interview was reviewed with respect to each element, statement by statement, to select specific information related to all elements. These separate items of information were assembled for each element. Next, this information was carefully reviewed and closely related items were grouped together as properties. Each element is described in more detail below with the properties that define it conceptually. For summary purposes these are listed in Table 3, containing each element with a list of its conceptual properties. Consistent with what the respondents said, the elements are not independent. In effect, NFE assessment is complex because it deals with interrelated variables. Moreover, the respondents in- dicated it was important in their role as analysts that they be able to examine the relationships among properties and among elements. For this reason, commonalities or overlap of properties with elements other than the one that they describe are noted in the analysis. Comments or statements of respondents are included in the following analysis. They provide support for or amplification of the analysis. The inclusion of these statements in the overall analysis is consistent with the methodology described in the preceding chapter. Through the process of comparative analysis, data from the respondents provided information for the identification of the eight elements and their properties. 86 :28»; 3223"“ 529:6 2: «SEE. 35388 222:: .32.. .3335 .0.- .323330 ”25qu ondn OOhSOuO“ nouns—6.: COmuahunouQ 880290 cozaeaoufi :aSSaaoU. 3533on 23338 2.2529, 832:: 3303 308308. 3.3. cc. 32:22 a .32... .5 .522: 33.8 of «5.2:: do 3230 3:828 28. 22.30 g n. 2030235 .3330 252.5 mmz 832.3 coca-3 an.” an . 3833‘ . on». «Sosa—a 2 c2832 mmz “Nam “mamas Sauce“. 33:30 23 E; 8222 mmz 3830.3 335 3:2 «men .863 26 :8 32520 3.8 823:0 522:5 .583 .32: . “a. causewm 35w 5.222 .22 :22: 22:3 E23 3:8 .233 :3 an 38.3 . 236:8? 22:: 3.325 .omcmnwmu 2232:: sex 282.. an 3:2 5:59: a mo—aeoaoem :2: 53 «58333 mmz do.“ ”Susan." 3035 .m 033. 87 W Without exception, the responses pointed to purpose as a critical element of an NFE assessment, even if the respondents did not mention this term specifically. It is essential because it often defines what is done, how it is done, and by whom. The purpose element has six properties, or conceptual aspects. These pr0perties (learning process, policy change, attention to nonformal education, data provision, control of assessment, and sponsorship) are listed in Table 4 and described in more detail below. Table 4. Properties of Purpose Element Property Learning process Policy change Attention to nonformal education Data provision Control of assessment Sponsorship 9.“?pr W A principle of nonformal education is that activities provide learning opportunities for all involved. It is not surprising then, that NFE assessments are viewed as opportunities to involve pe0ple in the learning and plan- ning processes. Respondent Three stated, ”The purpose of the assessment was to in- volve more people in the education process and in planning.“ An assessment can provide an opportunity for all individuals involved to learn new skills which can later be transferred to different situations or to have the opportunity to develop skills acquired elsewhere in the conduct of the assessment. Respondent One said, ”Ultimately, it’s a learning process. It allows the transference of that learning into something else." When new skills are acquired by individuals representing a developing nation’s institution, the capacity of that institution and of the nation are 88 strengthened. New knowledge can be acquired as well as new skills. Participants in the assessment process can also learn about nonformal education. The assumption that those involved in the assessment are also involved in a learning process has sig- nificant implications for the behavior of these individuals. This is addressed under the element entitled "Roles.” Involvement in the learning and planning processes also creates a sense of ownership of the activity. This is important because it relates to another property of this category, control of the assessment. Respondent Four said, “Ownership should come out of the assessment. This is a principle of nonformal education. It should be learner directed." W. Attention must be directed to what is wanted in the assess- ment and why. Respondent Three said, "It is important to know what they want and why. . . . and where detail is important for decisionmaking." The majority of respondents maintain that it is essential to include in a national assessment NFE ac- tivities conducted both by governmental and nongovernmental organizations. The broad sweep approach is necessary for policy development. And even policy directed at governmental activities must take the nongovernmental sector into consideration. One policy area relates to the identification of successful NFE activities. These can serve as opportunity targets for future investment of resources or as a source of information on what is effective in a particular context. ”Maybe we should look at why and how programs are successful and look to investing in those programs," said Respondent Six. A major policy area affecting nonformal education in developing countries includes decisions regarding centralization and decentralization. In a number of countries there is a tendency to centralize nonformal education. Although this has some strengths, some argue that centralization is a mechanism to control and harness 89 a type of education which functions best when it has opportunities to be flexible and innovative. Or, centralization may require an unnecessary bureaucracy that consumes already scarce NFE resources. For Respondent Six, "Central Coordination hasn’t proven to be particularly useful. . . . Nonformal education is very bureaucratized.” For many, decentralization is seen as consistent with the emphasis on local control of NFE activities. In some cases, however, decentralization deci- sions have been equated with a government’s unwillingness to assume responsibility (particularly financial) for educational opportunities for the out-of—school population. W. Nonformal education activities are sel- dom very visible because they often consist of varied and disparate programs. Consequently, it is often treated as a poor sister in the education sector, and in other sectors, such as agriculture and health, the education divisions also have low status. An NFE assessment can give nonformal education visibility and can bring it to the attention of higher level decisionmakers. Respondent Five states, ”The local institu- tions are not clear about nonformal education. They don’t give it status. You raise consciousness. You’re training them on the job." The gathering and analysis of large amounts of data, along with recognition from planners and policymakers, plays an important validating role. This attention can also lead to a more rational approach to nonformal education. Won. For an assessment to be worthwhile, it must meet the needs of the people who plan to use it. It should provide current information in time for it to be useful. Policymakers are usually extremely busy, with nonformal education only a part of a much larger portfolio. They seldom have their attention focused solely on nonformal education. It is important that they receive a synthesis of the current status with policy options that are described in terms of action steps. The end product of the assessment process should strive for simplicity and accuracy. A 90 lesser amount of accurate data is much more useful to decisionmakers than reams of information that may not be reliable. Furthermore, it is important to know when and what levels of detail are important. According to Respondent Three, 'What is important is timely, up-to-date information, preferably computerized. . . . Who needs huge amounts of information? It’s better to have a small amount of information that people agree is accurate." W. This property also relates to the category that addresses the roles of those involved in an assessment. It does so because it focuses on who does what in an assessment and under whose direction. Sponsorship (who is financing the assessment) is not necessarily the most important factor. Outside funds are not sufficient to guarantee control or influence. For an assessment to culminate in a useful document, there must also be commitment to the process from individuals and organizations in the education sector. For Respondent One, "A preexisting commitment to the project [NFE assessment] is essential. Overseas money isn’t enough.“ Individuals in a country will only use the assessment findings if they feel they are at least part owners of the process and its final outcome. Ownership can occur at a variety of levels—from data-gathering assistants to ministers of education and planning. For ownership to occur, ”people must per- ceive the need for the information and should value the integrity of the data,“ ac- cording to Respondent Four. In situations where much of the responsibility for the assessment is on individuals who are guests in the country where the assessment is being conducted, this raises some difficult questions. These include who provides the conceptual guidance, who is responsible for quality control, and who is ul- timately responsible for delivering the product at the agreed upon time. Sponsorship. Though respondents agree that nationals should conduct NFE assessments in their countries, "it is unlikely that this will happen in the majority of cases in the near f uture,‘ concludes Respondent Five. Even when skills exist in a 91 given country, professionals have such heavy work loads that they cannot be spared for such intensive activities as an NFE assessment. "A donor-sponsored assessment provides much needed data at a low opportunity cost . . ." in terms of personnel and financial resources, according to Respondent Six. It also complicates issues such as ownership and purpose of the assessment. When Preparation for an NFE assessment is an essential element because it deter- mines the stage and tone. Ill-advised decisions or procedures at this stage can com- plicate the conduct of the assessment or even decrease its value. Thus, it is very im- portant that this element receive sufficient attention. This research identified six properties of the element: early preparation, in-country players, briefing, planning the process, counterpart preparation, and logistics. These are listed in Table 5. Preparation for an assessment is necessarily closely related to the methodology selected, and it must be recognized that the methodology will affect the parameters of the properties of this category or element. Table 5. Properties of Preparation Element Property Early preparation In-country players Briefing Planning the process Counterpart preparation Logistics 93".‘PP? W. In general, the respondents believe that it is important to have as much advance notice as possible to enable them to assemble relevant materials and to think through the assignment ahead of time. According to 92 Respondent Seven, “I just didn’t have the preparation time. . . . a more advanced warning, I guess. . . . I ended up reading there things that I had collected here.” It also allows those involved to bring to the assessment what has already been learned in the area and so provide additional conceptualization and guidance to the assess- ment process and help decide whether they have all the necessary skills (such as language) for the assignment. In the end, however, it is important to recognize that there is only so much early preparation that can be conducted—much of the learning related to an NFE assessment is done "on the job,” according to Respondent Five law. To conduct a useful assessment, it is critical to be familiar with the major institutions and individuals active in nonformal education and in decisionmaking as it relates to nonformal education. In some cases linkages between these participants exist and are well used. In others, those responsible for the assessment may have to build information bridges to make it possible to com- plete the assignment. For example, Respondent One states, "I was working with the Permanent Secretary, the Lesotho Distance Training Center, The National Univer- sity of Lesotho, and four or five main NFE programs. I had to do a lot of PR for the study. . . . I had to bring a lot of people into the project. I was building inf or- mation bridges. Politicking was a major role." The identification of key people and institutions will be partly dependent on the purposes of the assessment. Their early identification allows for the development of a sense of ownership (as discussed un- der the purpose element) and facilitates data collection and analysis. Preparation must also involve the early identification of, and appropriate roles for, counterparts. These may change during the process of the assessment, but initial attention to this important aspect is essential. Included in this property is the idea of "identifying the formal interaction between the assessment team and high level officials” who will be responsible for making decisions based on the findings 93 of the assessment, as proposed by Respondent Five. In this way the property is closely linked to the policy environment clement. Two clarifications are important at this point. This property is concerned with identification both of individuals and institutions that are likely to be in- volved with the assessment process. And, this property is conceptually distinct from the one related to the preparation of individuals who will serve as counterparts for implementation of the assessment. This latter property is discussed later in this part of the chapter. Briefing. In addition to the preparation that the individual expert engages in prior to assessment work in-country (especially reflections on reading and prior experience), briefings by individuals who are familiar with the history and current context of nonformal education are essential. It is particularly important to have detailed information on the history and development of nonformal and adult educa- tion in the country. Equally as important is a general cultural briefing—without this blunders can delay the assessment process, questions be asked incorrectly, and statements be misinterpreted grossly. Respondent Six stressed ”the importance of the cultural brief ing" with regard to his relationship with counterparts. The cultural briefing also sensitizes those involved in the assessment to factors such as language issues. Though awareness of such issues is important for entree, it is even more im- portant for the analysis of data later in the assessment. W. This property is closely related to the in-country players property. It includes clarification of the purpose of the assessment, ref ine- ment of the agreed upon scope of work, and development of an understanding of all the steps that will be involved in the assessment, from planning the research through to reporting and report writing. According to Respondent Seven it was necessary to "get it very clear right from the beginning what my purpose, task, scope of work was. And I think because I didn’t do that, that led to some misunderstanding.“ In 94 some situations it may involve a short, targeted preliminary study of NFE activities to identify what exists as a basis for developing a framework for the conduct of the assessment. ”It is important to do a prestudy to get a range of what exists. For a framework. You can begin to see substudies at this stage," says Respondent One. Concerns relating to the development of a framework for the assessment are dis- cussed under the methodology category. W Experience has demonstrated that even insiders must engage in some initial preparation for an NFE assessment. According to Respondents Three and Five, "there is a need to train counterparts." Although they may not require the same kind of briefing, they may need to rearrange their regular work loads, clarify the nature of their participation in the assessment process, and refine their skills in some areas. For example, Respondent Three found that coun- terpart participation was a problem in one instance because ”people didn’t want to go out in the country,” and in another, "there was a severe lack of personnel." Lack of clarification and the related preparation can cause serious dif- f iculties later in the assessment. Outsiders often have different expectations than insiders with regard to such factors as who gives directions, how decisions are made regarding division of responsibility, and what is appr0priate feedback. Respondent Three feels that she ”really struggled with the counterpart relationship,” and Respondent One noted the importance of the ”training of interviewers“ who will col- lect data. Failure to train participants in questionnaire development and interview techniques may result in the collection of unreliable data. Counterpart preparation is separate from the interactions that occur be- tween insiders (host country nationals) and outsiders (experts from other countries) during the assessment process. The roles of insiders and outsiders was regarded by respondents as important enough to constitute a distinct element, which is discussed later in this chapter. 95 Logistics, Meeting logistics requirements and anticipating logistical problems is an important part of preparing for an assessment. There should be suf- ficient work space available for professional staff and support staff. If several drafts are required or the report is to be delivered very soon after the data have been collected and analyzed, it is likely that word processing equipment will be necessary. For example, Respondent Two noted that if he were to participate in another assessment, he would "be sure we had word processing.” In most cases, it is beneficial if one individual begins work early on the assessment by making initial appointments so that it is possible to begin data collection very soon after the assessment begins. "It would have been good if someone had set up initial appointments” prior to the arrival of the experts, as this ”would bring people in fresh at the second stage of assessment. This wouldn’t have cost that much more, and it would have had substantial benefits,” said Respondent Two. El 3 E l' E v' An understanding of the policy environment, the context in which policy is made and the status of nonformal education in that situation, is a key element in NFE assessment. Three properties comprise the element: policy development, NFE system type, and decisionmaking structure. These are listed in Table 6 and described in more detail below. Table 6. Properties of Policy Environment Element Property 1. Policy development 2. NFE system type 3. Decisionmaking structure 96 MW It is important for the assessor "to have an under- standing of the history of development of educational policy in general and NFE policy in particular in a given country,” according to Respondent Two. In some cases there has been little or no educational policy to guide day-to-day operations. In others, there may be a long history of policy, but it may not have been developed by individuals who have a good understanding of the education sector. It is espe- cially important to know who has been responsible for decisions in the past, how those who currently hold leadership positions in the education sector feel about the decisionmaking structure, and whether they plan to change the policy development process. W. The impact of nationally developed policy toward non- formal education will vary according to the degree to which NFE activities are, or can be, centrally coordinated or controlled. This makes it imperative to have an overview of the structure of the NFE system within a country. Whatever the pur- poses for an NFE assessment, it is probable that some understanding of the various linkages within the subsector and between the subsector and other subsectors will be necessary. According to Respondent Five, it is important to "look at decentralization of implementation. Also, how programs are designed—whether centrally or not." A centralized system may take advantage of economies of scale in areas such as train- ing and materials development, and, thus, it is easy to influence these kinds of ac- tivity through policy. Decentralized systems may or may not offer such advantages with regard to policy. If a system is centrally designed and decentralized for implementation, it might be relatively easy to make systemic changes through policy. If the NFE activities are disparate and not integrated through any management sys- tem or coordination, the relationships between policy and implementation are much more complex. For example, Respondent Seven states, ”But if the money is obtained 97 more creatively, more locally, and from different, smaller sources, then the issue of policy environment might not be as important. And that is the way I would really like to see nonformal education work. . . .But, that’s more haphazard, too.” Two important issues are included in this property—the development of an NFE bureaucracy and control of NFE. Centralized systems are often viewed as being more efficient because there are direct lines of communication from one level to another or from one program to another. Experience has shown, however, that centralization often results in bureaucratization, which, in turn, is often totally con- trary to streamlined and efficient use of resources. "A major problem is that NFE is very bureaucratized. For example, the service agency concept tends to provide employment for bureaucrats who don’t add much but take lots of resources,“ says Respondent Six. Moreover, some would argue that one of the strengths of nonfor- mal education is that activities can be short lived—beginning when an educational need is recognized and ending when that need has been met. Centralization and bureaucratization usually support institutionalization, even when institutionaliza- tion is not called for. Unlike formal education, nonformal education has not traditionally been a socialization institution within a given country. There is concern that centralized planning, coordination, and policy development related to nonformal education might be used to control the content and methods used in NFE activities to the ex- tent that it could become yet another tool of the system rather than providing learn- ing opportunities for individuals and groups. For example, Respondent Six feels that centralization and national planning may not be in the best interests of NFE learners. He states, ”Small scale community efforts are probably better.“ Wm. Respondent Seven states, “I mean to try and see how these decisions are made in the different parts of government, where they are made, and how those decisions get coordinated or whatever, if they do. But yes, I 98 think that would be a much better way of looking at the policy environment." An understanding of the planning mechanisms and decisionmaking processes makes it possible to collect, present, and analyze data in ways that will be useful to education planners. It is also essential to be aware of who uses information and makes recommendations to policymakers. An understanding of the roles of these in- dividuals and their training and capabilities will help an assessor provide inf orma- tion in a useful form. According to Respondent Seven, ”If it [money] is coming from the World Bank or USAID through the government or even through some large NGOs, then I think the policy environment decisionmaking is important.” Knowing who uses data also provides those involved in the assessment with contact points to verify whether the assessment itself is on track. Cases where these individuals are not particularly supportive of, or even hostile to, nonformal educa- tion require different amounts and kinds of information than those where decisionmakers and leaders are knowledgeable about the area. Sometimes those responsible for the assessment find themselves in situations where individuals or in- stitutions responsible for nonformal education are not held in high regard and so the assessors have to promote the value of the NFE activities at the same time as they are trying to develop a useful analysis and set of recommendations. Knowledge about the decisionmaking structure has two additional advantages. First, it provides very important clues to the information networks re- lated to decisionmaking. And, it allows for the identification of mechanisms to support successful NFE efforts. Wm An NFE assessment should ultimately be a learning process, and this process is a fundamental element to be considered throughout the assessment. The process element has five properties. These are learning, definition and data, assessor 99 interactions, insider involvement, and information transfer. These properties are listed in Table 7 and discussed in more detail below. Learning. An NFE assessment provides a variety of learning opportunities and allows a wide range of people to become more involved in the educational process. For Respondent One, it is essential to recognize ”the importance of people. They have translated this experience [the assessment] into policy. . . . These people are now in important positions. These people have moved to policy positions—and Table 7. Properties of Process Element Property Learning Definition and data Assessor interactions Insider involvement Information transfer 3"?P’E‘N‘ more than just nonformal education.” Thus, a purpose of an assessment is to en- hance development of the entire human resources sector. It does this through of f er- ing training opportunities both in assessment and nonformal education. Skills gained through the NFE assessment process, such as developing and administering a questionnaire, can be transferred to other contexts, and practitioners can learn how to influence policy formulation and decisionmaking related to nonformal education as well as to other parts of the education sector. The potential for the application of skills gained through an assessment in the long run highlights the importance of people who participate in the process. In this regard, it is not only the gain of assessment skills that is important but also individuals’ participation in decisionmaking about the conduct of the assessment it- self so they have an opportunity to direct their own learning—this is a fundamental NFE principle that should be incorporated into the planning for an NFE assessment. 100 For example, Respondent Four argues that ”even the research instruments should be tools developed by the people actually using them.” A commitment to ensuring that the assessment consists of a process designed to enhance learning means that the assessment cannot be bulldozed through. First, any learning process takes time, and in this case it is essential to give participants time to define their learning needs as well as time to learn. Second, this commit- ment requires a certain amount of flexibility, or the learning process could be hampered. "We spent a lot of time facilitating the involvement of a wide range of people,” states Respondent Four about one assessment. W. The assessment process is dependent upon a clear un- derstanding of what is meant by nonformal education in a particular situation. "A major concern was to describe nonformal education. To validate the concept of nonformal education,” said Respondent Three. In some cases an overly precise definition of the field might not be regarded as very important. Experience has shown that in other cases, nonformal education can be defined very narrowly, such as literacy training for adults. In still other situations, ethnic definitions are important. The assessment process may also allow for the opening up of an ex- tremely narrow definition and provides opportunities to explain terms such as ”problem solving instructional methods." The process is an important factor in relation to gathering appropriate data for particular assessment purposes. "We must be sure we are aware of what they want to know” says Respondent Three. The process provides an opportunity for all involved in nonformal education, whether at policy or implementation levels, to state their needs in terms that can be addressed by the assessment. How this is done will be influenced by the context. In some countries, such as Indonesia, needs are already clearly defined and many data available in published documents. In others, such as Haiti, much of the data have to be collected through interviews. Regardless, 101 the process must allow time to deal with issues surrounding the accuracy and verification of data. Respondent Six raises the issue that there is often ”little actual data, and it is hard to do an assessment in an absence of data." And, it is important that the process encourage candid responses. Respondent Three was concerned that that the methods employed and data collected not be couched in a pretense of scien- tificness that is not warranted. Wiggle. Whether an NFE assessor was on assignment in- dividually or as part of a larger team, the importance of interactions with fellow professionals was identified as a basic part of the assessment process. Others provided the assessor with support, information, ideas, and excellent opportunities for rich dialogue. Support is greatly appreciated, especially when individuals are working away from a familiar environment. Respondent Seven felt very positive about having been part of a team. She felt that it was ”a real help, that it gave me more legitimacy to explain to people that I was part of a team and that we were doing this overall assessment of all the different subsectors. . . . And I felt that the camaraderie and support of team members and sharing of daily experiences was very helpful. And also, there was a lot of sharing of inf ormation.‘ Additional in- formation and ideas are invaluable when individuals are working under pressure and with severe time constraints, both of which are common during the conduct of an assessment. Most appreciated, however, were the rich conversations between professionals that were sparked by the common interest in the assessment but that went beyond the day-to-day data and analysis issues. These were conversations in which new ideas were explored and major concerns shared. What is regretted is that these conversations are not recorded anywhere—they do not become part of the documentation that reaches those in decisionmaking positions even though they evolved out of the assessment, are a critical part of the process, and often could be 102 extremely helpful to those making decisions. Respondent Two notes that ”many in- novative ideas get lost. . . . They should have been shared with ministry people.” As part of a larger team, the NFE assessor plays another significant role. Often this individual serves as an educator about nonformal education in general for the rest of the team. ”Somehow nonformal education is the domain of NFE people. I played a role as educator to the rest of my team,“ states Respondent Five. W. The participation of those actually involved in non- formal education in a country is an important part of this category. The necessity of their involvement in the process of the assessment requires that insider involve- ment be a property of this process category. Involvement in the process brings people into the assessment and enhances the concept of ownership, which is impor- tant if purpose and policy environment objectives are to be met. One of the major problems in the assessment process is the limited availability of insiders. They often are extremely busy and outsiders have been employed to conduct the assessment because insiders cannot be spared for the task. Respondent Three noted that ”lack of personnel made it difficult to conduct the assessment the way I would have liked.“ Their participation is critical, however, for input and feedback, analysis, quality control, and reality testing. The number of participants involved in the process is also important. Collaboration with a very limited number of individuals can lead to bias and misinterpretation. Says Respon- dent Seven, "There may be a problem with who you talk to in a sector assessment.” W. A critical part of the process brings information to those decisionmakers who are part of the policy environment. Respondent Two notes that if he were to be involved in another NFE assessment, he would “formalize a set of meetings with higher level people and be sure to share a rough draft or summary with them early on.” In some cases the process may require information or activities to raise consciousness about nonformal education itself. Respondent Five 103 believes that an NFE assessor “may have to dialogue with higher level people, as they may not be clear about nonformal education.” W As with any other activity an assessment requires people for its implementation. Because of the large scope of an NFE assessment, it often involves the work of a large number of participants with a variety of roles. The contribu— tion of each individual is important and can significantly enrich the final product. Furthermore, each participant in the process will benefit from this experience through the rest of his or her professional career. The extent to which this ex- perience has been positive and professionally useful can have significant outcomes in any number of different contexts. This research has emphasized the importance of individuals involved in NFE assessments for developing countries. The roles element is tightly defined with three properties: insider roles, outsider roles, and context. These are shown in Table 8, and the properties are described in more detail below. Table 8. Properties of Roles Element Property l. Insider roles 2. Outsider roles 3. Context msjdeLBeles. Insiders play a variety of roles and their active participation is vital to an NFE assessment. Even though the time available for this participation might be limited because they are so busy, they ultimately must feel a sense of ownership of the outcome, and should have benefited professionally from their involvement. 104 Although it may be argued that assessments ideally should be conducted solely by objective insiders, it is recognized that this is currently not very practical in many countries because of manpower constraints. Insiders may assist an assess- ment with data collection, and as providers of data and analysis, developers of data collection tools, advisors, inf luencers, and decisionmakers. Respondent Seven says, "So, I don’t know how you would find an objective insider to do it [NFE assessment]. But that would be the best possible situation. . . . But I would say a disinterested in- sider would be much better than someone who knows little about the country and hasn’t worked there before coming and trying to do it.“ Counterparts are regarded as being particularly valuable because they can bring specificity and depth to data that has been obtained. In fact, Respondent Two feels that the assessment in which he was involved was “severely limited because a counterpart was not identified for me.“ Counterparts often serve as a catalyst, widening the scope of the thinking of an outsider. And they are usually facilitators—bridging the language gap, initiating contacts, providing access to networks, and serving as an entree. Their familiarity with the existing situation means that analyses can be firmly grounded in reality and recommendations can be more valid and relevant. The involvement of insiders in an NFE assessment can also have some difficulties. In addition to adding utility to the assessment and its process, they can also add complexity. Their presence, depending on who they are and what con- stituencies they represent, can increase the influence of politics on the assessment. Some individuals may come with a fixed point of view, and others may have vested interests which force them to be compromising when it comes to analysis and recommendations. Respondent Six sums it up with, ”Counterparts added a great deal in Indonesia. Their presence also adds to the complexity of the process.“ 105 In some situations the counterpart relationship is delicate. If a single in- dividual is responsible for presenting or writing a report, it can be an onerous task if all involved do not make serious efforts to reach a consensus or agree to disagree. In other situations, outsiders are often frustrated by insider counterparts who refuse to participate as equals or to provide as well as take direction. For Respondent Three the counterpart relationship was extremely difficult, especially when she ”looked for participation in directing the research." mam. Because of the limited numbers of highly trained prof es- sional educators in developing countries, it is accepted that in the near future it is likely that NFE assessments will be conducted at least partially by outsiders. An outsider has a difficult leadership role. One aspect of it was termed ”politicking“ by one of the respondents. The outsider has to bring insiders into the process, con- tinually keep the purposes in mind, and frequently balance out-of-country interests (often those of financial sponsors) with in-country interests. In addition to the purely technical aspects of the job, the outsider often has to serve as an educator about nonformal education and assessment, a stimulator, a catalyst, and as a spot researcher. And, according to Respondent Six, ”Outsiders can transfer skills, lend credibility, and an external quality. They can poke their noses into things an in- sider can’t." The outsider must have a range of social skills. This individual must ex- hibit human qualities that facilitate interaction and the learning process while also conducting the task. He or she must be skilled at dealing with a wide range of people (from NFE program participants to ministers of education) in a variety of professional and social situations. Participant One stresses the "importance of skills of dealing with people of another culture. Also with a range of people. The human qualities of the person guiding the assessment are important.” Moreover, the NFE expert who is an outsider must be able to assess situations as they are developing, 106 taking advantage of opportunities as they present themselves, and heading off con- f licts before they become destructive. The tasks of the outsider are often complicated by a number of factors. Sometimes the outsider has to force the counterpart relationships. He or she also has to overcome the difficulties of being seen as an evaluator, a representative of inter- nal or external vested interests, and as a possible funding source. For example, Respondent Seven speculated ”that pe0ple saw me as a, well as a representative of USAID and possibly as a source of money, and so there was a certain kind of def erence.‘ 99.019151. Although seemingly obvious, it must be said that contexts vary from country to country and even from time to time in the same country. It is es- sential that this be taken into account when an NFE assessment is conducted. Moreover, it is important to recognize that some cultures are less forgiving of bad or inappropriate behavior on the part of specialists than are others. W Methodology is a critical element—it addresses the method by which an assessment is conducted. The methodology category has three properties, approach, organizational structure, and integrity of data, which are listed in Table 9. The properties of the methodology element are discussed below. Table 9. Properties of Methodology Element Property 1. Approach 2. Organizational structure 3. Data integrity 107 Ammeeh. The way data are collected and analyzed may be influenced to a great extent by the purpose of the assessment. For example, an assessment of ac- tivities for a limited client group such as out-of-school youth would be very dif- ferent from one for all NF E activities in a country. As Respondent One put it, 'If I went to India tomorrow [to conduct an assessment], it would be an enormous challenge. I would use a very different strategy.” The approach chosen will also be partially dependent on factors such as the size and population of the country, the availability of existing data, the structure and organization of NFE activities, and the availability of human and other resources to conduct the assessment. While most respondents generally believe that an assessment covering all NFE activities can be very useful, they also recommend in-depth studies of fewer activities with representativeness and geographic spread as criteria for selection and a goal of generalization from the specific. Respondent Six states, “Where an inventory would be very big, I might not recommend that. It might be better to do a smaller sampling." The amount of time allowed for the assessment is an essential factor when making decisions about the approach. For example, according to Respondent Four, "It takes time to develop ownership. I’m afraid this means more money rather than less.” In addition to wanting more time to prepare, assessors specify the need for more time for data collection and completion of the assessment. They indicated that increased time between data collection and completion would allow the benefits of more time to reflect, develop an environment for the creative process, and develop a better partnership study between insiders and outsiders. Some situations would be greatly improved if outsiders participated only during the beginning and at the end of the study, with perhaps a 6-month interval between for further data collection and analysis. According to Respondent Six, ''It might be best for the outsider to par- ticipate in primary data collection and especially to help with organizing tasks. 108 Then, let them do data collection and analysis. And again, for the outsider to help with organizing at the end.” The selected approach should allow for data gathering on specific tapics for substudies that can support and enhance the analysis, if possible. Though the asses- sors had worked in both centralized and decentralized assessments, the general con- sensus was that a centrally directed assessment seemed to be more focused and easier to interpret. For example, Respondent One states, “In Colombia we had a decentral- ized approach which didn’t work as well.” Regardless, the approach to an assessment must be both consultative and participatory. It is through consultation that the assessment starts off on the right footing and through participation that a sense of ownership is developed, advice is channeled into the process, skills are transferred, and new insights are brought in. W. A feature which distinguishes an assessment from an inventory is the presence of analysis in the f ormer—whatever the organiza- tional structure in an assessment, it must allow for analysis. According to Respon- dent One, "I distinguish between assessments and inventories", and Respondent Six, ”Inventories don’t add anything. They lack analysis.” Furthermore, this analysis should be conducted in-country and at least in consultation with insiders. Use of an existing structure or framework for an assessment is very helpful. This is especially the case when time is limited. But, "the framework must allow for examination of the relationships between data categories," emphasizes Respondent Five. If an existing framework is not to be used, it might be advisable to conduct a brief preliminary study of NFE activities to assist with the development of a struc- ture for the assessment. ”Preliminary work is necessary. It is especially useful if there is an existing national inventory or as a study of existing programs as was the case in Indonesia,” says Respondent Six. 109 Although the respondents used several very different frameworks for their work, they all found them useful—particularly those that allowed nonformal educa- tion to be viewed as part of a larger system of education. Use of the same framework in different contexts allows for comparison across countries. According to Respondent Six, however, "There is a trade-of f in sticking to a particular f ramework.‘ It may force the collection of unnecessary data and consideration of relationships that are not relevant to a given context. A flexible framework allows for a much more streamlined approach. W111i- Data collection often provides many more problems in non- formal education than it does in formal education. Even in cases where data appear to be abundantly available, there are often questions as to their accuracy. Inf orma- tion is only as good as its source, and in some cases biased data may be provided in an effort to protect certain interests. It is easy for assessors to be misled by en- thusiasts or by their own misinterpretations. It is also relatively easy to make reports look more scientific than they really are when it is better to provide decisionmakers with accurate information rather than large amounts of dubious data. Or, as Respondent Seven states, "Reports on content and methods in a program are not necessarily insurance of what messages were actually delivered.” To deal with problems of data validity, it is important that the methodol- ogy allow the researcher to triangulate findings. This enables the assessor to build conclusions that are more reliable. There are also some problems that NFE assessors have that may not have to be addressed by those concerned with the formal part of the sector. It is important that the methodology be flexible enough to allow the assessor to be creative in deal- ing with these difficulties. For example, often individuals or organizations are un- willing to provide information about the level of funding or how these funds are actually used. Respondent Four comments, "About the amount of money coming in. 110 People didn’t want to respond to the questions. We didn’t anticipate reluctance to respond to some questions.“ Or, if assessors are seen as evaluators of their work, they may be unwilling to share much information at all. Wuhan: NFE program descriptions are a basic element of an NFE assessment be- cause they illustrate the magnitude, variety, and texture of the subsector. The program description category has the largest number of properties, a total of fourteen. These are listed in Table 10. Although there are many properties in this category, they will be familiar to most readers and, in general, they consist of dis- crete and relatively simple concepts. The fourteen categories are described below. Table 10. Properties of NFE Program Descriptions Element Property Range Goals Program history Learners Content Methods and materials Outcomes Resource base Geographical spread Organizational structure Personnel Timing and Duration Facilities Research H PPFSQPdrPP!’ _ — fib— 99’3". Range. 'It is important to look at the range of programs that exist," says Respondent Five. Information on range includes the sponsorship of activities (government, nongovernmental, or community agency, for example), and their general approach. Information on this property indicates the extent to which lll various kinds of agencies are active in nonformal education and the structure of the NFE system. Q9113.- I'Questions about goals and objectives are important. But they don’t always yield. This is the weakest area of data. The hardest to describe,“ says Respondent One. This is a very difficult property on which to get data. Goals are hard to describe, but it is essential to examine achievements with respect to goals. Aspirations are important because they indicate how the future of these activities might evolve. Wm. According to Respondent Four, "An assessment should look at the history of the activity. Try to identify if there is a history—a con- sciousness within the staff of the activity, how and why it started, in response to whom and what, at who’s initiative, and with what participant involvement." His- torical data indicate how and why particular kinds of activities were started. They provide information about who initiated what aspects of the activities and what kinds of involvement have contributed to the success of the program. They might also give information on linkages. Learners. If one views the learner as the center of the learning process, as is the case with nonformal education, then it is expected that one would want exten- sive information about the learners. The first question asked is who are the learners. Information might be obtained on age, gender, ethnic group, SOCIO° economic class, caste, and whether the learner is rural or urban. Another important aspect of this property relates to how many learners are participating in a particular NFE activity. Although initial enrollment data may be available, this is seldom sufficient. Information on retention and completion rates is much more useful, and it is very important to have reasons why participants drop out of programs. It is sometimes useful to know if learners are participating as individuals or as groups. For Respondent Five there are several concerns, ”Data on learners are not always 112 sufficiently disaggregated. . . . We need to know how many complete a course, how many complete the cycle. Without this, we can’t evaluate success and completion. Retention is very important.” Cement. In addition to knowing the topical area, more detailed inf orma- tion on the curriculum may be desirable; for example, how the curriculum was developed and its relationship to needs for daily life of the target learners, and the entry level skills that it requires. "We need to take a good look at the curriculum and what it offers. A look at the curriculum in relation to the characteristics of the population,” says Respondent Five. Wale. A key concern here is how learners are engaged in learning; that is, what methods and materials are employed. According to Respon- dent Four, ”We need more descriptive information on methods, materials. . . ." and for Respondent Two it is important to look at the "appropriateness of materials and methods.“ Unfortunately, there are many cases where materials are not really avail- able to the extent they should be, even though they may be appr0priate. Quteemes. Outcomes of NFE programs are especially difficult to document; yet, it is this kind of evaluation (often in relation to costs) that planners frequently ask for. For example, Respondent Seven says, "even if there were data on cost, that doesn’t tell the whole story. . . . the efficiency of nonformal education is not only determined by spending the least amount of money for the maximum gain. . . .Well, it is but then you have to define what you mean by maximum gain or ef f ectiveness.” Measures of program effectiveness might include benefits to participants (such as impact of the program on quality of life, level of income, fertility rates, and health and agricultural practices), skill level reached, and relevance of skills learned. Another area where there is limited information is on the secondary benefits of nonformal education which are reflected in outcomes such as parental commitment to keeping children in primary school. ”We know that educational level of parents is 113 critical but we don’t know what contributions adult nonformal programs actually make,” says Respondent Two. W. Information related to the financial resource base for non- formal education has generally been very difficult to obtain. "Cost and financing data are the most difficult to get,“ says Respondent Five. Attempts have been made to obtain data on costs and financing, but it is still widely felt that a much better framework for this is needed. There are many indications that much financing comes from local sources—in many cases it would be useful for an assessment to produce a better description of the local resource base and its potential for expansion. Other issues concern the common view that centralized financing com- prises a better use of scarce resources than decentralized financing and the belief that pluralizing rather than centralizing implementation will make programs more responsive to local needs and will also give them a wider resource base. W1. Data on this property concern two separate but re- lated geographical concepts. The first is the general location of the program—does it serve urban or rural residents, for example, or is it located in an area that is populated by a majority or minority population? The second concept is its accessibility. Is it easy to get to and by what means. For example, is attendance de- pendent upon a river not flooding or being able to afford a bus fare? Respondent Seven notes, 'It’s much more difficult to get nonformal education to the poorest, or the people that need it most, and have it be successful. . . . And then also just be- cause of the distance and geography of places that the access issue is . . . how are people actually being reached and the cost of sending people out to outlying areas to try to bring people some form of education.” Were. ”Knowledge of the organization and managerial outline of a project are essential,” says Respondent One, “and, so are the dynamics, but it’s hard to know how much you can get into this at the national level.“ 114 Questions related to the management and administration of an NFE activity are en- compassed within this property. It might also include a discussion of the levels at which decisions are made and within-project dynamics if this was needed for the assessment. Sponsorship is sometimes considered in this property. Issues such as centralization, decentralization, and learner participation may also be considered. Reesennel. Information within a program is generally sought on facilitators and their trainers and supervisors. Data on the numbers involved and the content and methods of their training are often required. “We need to know about trainers, teachers . . . this is a very useful indicator of how the curriculum is being implemented,” says Respondent Five. This property frequently poses difficulties in a sector assessment. This is partly because of the range of personnel active in the NFE subsector. Paraprofessionals, volunteers, and short-term employees are often active in addition to f ull-time, long-term professionals, all of whom serve at many different levels. This range of types of personnel makes for very complex questions surrounding f ac- tors such as staff hiring, promotion, and evaluation. mm This property includes information on when NFE activities are made available—both in terms of the amount of time for a given course and its scheduling. For example, a learning program might be intensive during the dry season and have few meetings during harvest time for a community which is dependent on agriculture. One of the most difficult issues on which to ob- tain data is time on task. Yet this is important information, particularly as it re- lates to outcomes. Respondent One says, ”Duration versus intensity of a program is hard to describe. It is hard to get comparable data to use.” Eaeimjes. Information on facilities is especially important if coordination is planned. Data may be required on factors such as seating capacity, number of rooms, overnight accommodations, and kitchen facilities. In some having the llS mailing address is very useful. Says Respondent Three, "If I had to do it again, I would measure the buildings for accuracy because space is so important.” Reseereh. Information on research related to a particular NFE activity is useful to gather. Respondent One suggests that it is essential to know about "current, prior, and planned research.” assumes The linkages element addresses coordination, sharing of resources (institutional, physical, and human), and points of interaction. This element was difficult to distinguish from the others because it addresses the points of intersec- tion and interaction between nonformal education and other aspects of education and development. Such contact points are often not well documented and it is very difficult to obtain data that describe them. There are three properties that describe the element. These are linkages to general education and development goals, to the other activities in the NFE subsector, and to the rest of the education sector. These are listed in Table 11 and described in more detail below. Table 11. Properties of Linkages Element Property 1. Education and development goals 2. Other NFE activities 3. The education sector W. The relationships between education and development are of great concern for the developing nations, particularly in terms of investment of resources such as time, trained personnel, and funds. Ac- cording to Respondent Seven, what is needed is ”a kind of preliminary study on analysis of the development philosophy or the general development strategy of the 116 government and then how the different nonformal education activities fit into that or how they are aimed to fit in with the development strategy.” It is important, then, that an NFE assessment consider the nation’s development goals, the strategies that use education help meet them, and how NFE activities can support these strategies. The relationship between training offered through NFE activities and the labor market is an example of one way this property might be examined. W. This property addresses issues of coordination, cooperation, and redundancy. For example, separate NFE projects sometimes offer identical or highly similar learning opportunities to the same target group of lear- ners that are not taking full advantage of what is already available. Better coor- dination would encourage more efficient use of resources. Similarly, it might be advantageous for government agencies to leave certain responsibilities to non- governmental organizations, especially in cases where they already are functioning well in that capacity. There are many cases where NGOs could strengthen their programs through access to resources such as government-developed training programs, curricula, and materials. Respondent Two argues for ”Better coordination among agencies with a sharing of central resources. . . . The central service agency is a good approach because it improves efficiency markedly . . . .Central government could make better use of nongovernmental agencies, also." In general, the property is directed toward strengthening NFE activities in a country through an appropriate and efficient sharing of resources. We. The wide range of linkages to the rest of the educa- tion sector are considered under this property. In fact, this property is particularly difficult to describe because of the complex range of its components. Four examples cited in this research are mentioned. First, one area of potential mutual support and interaction is between nonformal skills training and vocational and technical education. Second, sharing of facilities and teachers may be desirable and is 117 common. Third, formal and nonformal education often borrow curricula, methods, and materials from each other in a given location even though this was never planned and might even be quite inappropriate! Finally, the relationship between the education of parents through adult nonformal education, and their willingness to support primary education for their children, may serve as an important way to obtain local financial support for formal education for children. Swami Chapter 3 described how grounded theory was used to identify the key ele- ments for national level NFE assessment. Application of the methodology to the data provided in this chapter resulted in eight elements, each with its own set of properties, ranging from three to fourteen. Each element is conceptually defined by these properties. Conclusions and implications of these findings are discussed in the following chapter. CHAPTER 5 FINDINGS, DISCUSSION, AND IMPLICATIONS This research was designed to identify the critical elements of a national- level NFE assessment conducted to support education planning in developing nations. This chapter summarizes the results, discusses the findings, and addresses the implications of this research. Finally, it identifies possible areas for future research. W The data collected from seven interviews were analyzed to identify the critical elements for NFE assessment and the properties that define them. Elements began to emerge during the interviews, each with a variety of components, descriptors, or properties. In some cases there was overlap—properties appeared to belong to more than one element. In others, it became clear that items that originally appeared to be elements were instead properties of a larger conceptual category, or element. The following eight key elements were identified by this research: purpose, preparation, policy environment, process, roles, methodology, NFE program descriptions, and linkages. Each element is conceptually separate from the others, and is defined by an individual set of concepts, or properties. These are discussed in detail in Chapter 4. The number of properties that define an element ranges from 3 to 14. 118 119 These eight elements should influence the planning and management of a national-level NFE assessment. One of the more interesting findings of the study is that most of these critical elements are intangible, such as the policy environment or the assessment process. In some cases the elements are difficult to identify and to describe, and thus present problems related to ensuring that they are included in an NFE assessment. It is stressed here, however, that this research has identified the elements and their component parts. This research was not designed to direct deci- sions regarding attention to these elements in a given assessment. Many of these elements are not reflected in the literature on NFE assessments. The literature has generally focused on NFE program descriptions, only one of the eight elements identified in this research. This present research is based on a wide range of actual assessment experience, however, and in most cases the respondents stressed the significance of categories such as the process, policy environment, purpose, and relationships with counterparts, indicating that a focus on the NFE programs alone is insufficient for an assessment. WWW Although the elements are conceptually independent, they are interrelated. Properties within an element may also be interrelated. Consistent with the response of the respondents, there is no element that is unrelated to the others. These interrelationships among the key variables contribute to the complexity of conducting an NFE assessment. Moreover, the respondents indicated that it was important, in their role as analysts, that they be able to examine the relationships among properties and elements. It is the analysis of these relationships that makes an assessment a useful document for planners and policymakers. This is not surprising because nonformal education operates within a system which is part of a larger educational system which itself is part of an even larger socioeconomic political system. Such interrelationships are intrinsic charac- teristics of complex systems. 120 Table 12 displays the relationships among the key elements. It illustrates that NFE assessment is complex because of these interrelationships that exist among variables and the elements they comprise. The columns in the table indicate that there is no element that is unrelated to at least one other element. Two elements, purpose and policy environment, are each closely related to eleven properties of six other elements. The program description element is related to five pr0perties, each of a separate element. And, the roles, linkages, and methodology elements are re- lated to six, six, and five properties respectively in three elements each. The table also serves as a basis for further discussion below of each of the critical elements. W. All respondents either mentioned or alluded to purpose in their first response. This is so important, in part, because the respon- dents recognized the need f or the purpose to be at least partially defined by those in the country where the assessment is being conducted, even if the assessment has ad- ditional purposes defined by the funding organization. A review of assessment documentation suggests that the purpose that gets priority attention is the one defined by the funding agency, suggesting that it is up to the individuals conduct- ing the assessment to obtain information on the possible multiple purposes of the assessment and to ensure that they are all addressed during the assessment process. The number of respondents who believed that a purpose of an NFE assess- ment was to focus attention on nonformal education was particularly interesting. Whereas those in formal education tend to be concerned that decisionmakers outside of education may not understand the importance of education, underlying these statements in the present research is a concern that the majority of educators and educational planners do not understand the methods and objectives of nonformal education. An NFE assessment is one way to get information about nonformal education to those who would not normally learn about the field, yet may be making decisions that affect it. 121 Table 12. Relationships Among NFE Assessment Elements and their Properties Elements Policy Elements and Prepa- Method- their properties Purpose ration enviror- Process Roles 010 0 Program descrlp- Linkages tion Purpose Learning process Policy change Attention to NFE Data provision Control of assessment Sponsorship Preparation Early preparation In-country x Briefing Planning the process Counterpart preparation Logistics Policy Environment Policy development 0 NFE system type at Decisionmaking structure it o x Process Learning at 3: Definition and data x Assessor interactions Insider involvement 1: at Information transfer at Roles Insider roles at x Outsider roles at x Context Methodology Approach x x o Organisational Structure 1: 0 Data Integrity o NFE Program Descriptions Range 1: Goals Program history Learners Content Methods and materials Outcomes Resource base Geographical spread Organisational structure 1: Personnel Timing and duration Facilities Research Links“ Education and de- velopment goals at o Other NFE activities The education sector it x 0 000000 000000 0 00000 X ”XX N8 00 8 x 3 00000000000000 0 Note: it = a related property from another element. o=apropertyofanelement. 122 The emphasis on a wide range of aspects related to preparation for an NFE assessment is worthy of comment. Every respondent was concerned about the issue of preparation. Many noted that preparation takes time—something they noted is usually scarce during an assessment. They also stressed the close relationship be- tween ample preparation and a well-received end product. It is interesting that no respondent mentioned appropriate technical train- ing of an assessor. It has already been noted that all respondents were both highly educated and experienced in nonformal education. Yet, it is somewhat surprising that these individuals did not describe as part of preparation the kinds of technical skills they possess—in nonformal education, analysis, writing, and data gathering, for example. It is possible that these individuals took these skills for granted. Per- haps if the pool of respondents had included individuals who had attempted NFE assessments but met with little success, then prior training in relevant technical skills might have been identified as a property of this element. The clear and early emergence of the policy environment element was not fully expected. Only one respondent stated that it was not the role of an assessor to be concerned about policy and the policy environment. An understanding of the policy environment can influence what data are gathered and how information is presented. It should be noted, however, that the degree to which the policy en- vironment is a critical element partly depends on the extent to which policy and implementation interact. If all NFE activities were conducted in total absence of government, then the policy environment would be irrelevant. It is unlikely that this extreme case would ever be found, but there are contexts where most of non- formal education occurs beyond the boundaries of government-funded activities. The strong emphasis on an NFE assessment as a learning process also was not expected. This was sometimes regarded as a major purpose of the assessment, al- though most have not been structured with that in mind. What was even more 123 interesting about the view of the assessment as a learning process is the degree to which the assessors themselves valued what they had learned from in-country coun- terparts and fellow team members. Moreover, the environment of the assessment al- lowed for a two-way learning process. The respondents were very aware of a variety of ways they had served as facilitators and teachers in a number of dif- f erent situations. The sensitivity of NFE assessors to their roles both as learners and teachers leads to a consideration of the roles category itself. Like the purpose category, this was one that all respondents commented on in one way or another, even if in- dividuals did not specifically identify it as an element. Working in an unfamiliar country demands extreme consciousness about the meaning of one’s behavior. A dilemma that NFE assessors often meet is that they have to work with a wide range of individuals—from field workers, to ministers, to representatives of international agencies—and each interaction requires a different kind of behavior. It is assumed that the assessor will know what behavior is appr0priate in a given situation. Again, this emphasizes the need for preparation that goes beyond technical skills in nonformal education and assessment techniques. It is important to conduct an assessment within a particular framework or with a given methodology, but there is no single methodology that is regarded as ideal. The respondents were quite responsive to alternative approaches to the assessment in relation to its purpose. Assessments that stress breadth were regarded as requiring trade-offs against those that permit in-depth studies of a smaller num- ber of NFE activities. Different kinds of planning requires different kinds of assessments. As noted earlier the NFE program descriptions element is the one with which professionals are most familiar. It is the single element that focuses on the nature of individual NFE activities. These assessments have mostly focused on 124 factors that describe NFE activities in one way or another. This research shows very clearly that this is only a small part of an NFE assessment—description of NFE programs is only one of eight elements. A focus on this element alone provides an inventory and not an assessment. With regard to this element, it is important to be aware of its properties in relation to each other—simple description is insufficient for an NFE assessment. For those who have been professionally involved in NFE assessment, for example, a major concern has been what data should be collected for planning purposes. These concerns have unfortunately been voiced only orally in meetings, so there is no f or- mal record of them. What has often been at issue is the lack of cost-related data and data on the relationships between costs and benefits on one side, and a concern, on the other side, that focusing on costs in relation to immediate benefits only ad- dresses part of the goals of nonformal education. Other points have concerned the need for disaggregated data for areas such as participants, program content, and methods. Of the eight elements identified by comparative analysis, the linkages one was the most elusive. Although it emerged immediately as a potential category, the extensive overlap with other categories initially suggested that it might not be a dis- crete category. In the final analysis it was regarded as conceptually discrete. The identification of linkages as an element emphasizes the need to regard nonformal education as part of the larger educational system. Although some practitioners might suggest that nonformal education would be more effective if separated from the remainder of the education system and left to develop on its own, the findings of this research suggest that one simply cannot ignore the existing linkages. This does not imply, however, that NFE activities should be fully included with the rest of the sector. It does raise significant issues about the roles of the different parts of 125 an educational system and how coordination and harmony can be achieved without violating the purposes and characteristics of the different parts of that system. W In addition to identifying the eight critical elements for NFE assessment, this research clarified that some of the issues that affect implementation of NFE ac- tivities will also influence an NFE assessment. Three recurrent themes in the inter- views were participation, NFE planning, and ownership of information. Regardless of the element discussed during interviews, questions were raised about who was involved and the nature of that involvement. Participation is a key issue for nonformal education, and it is clear that those professionally in- volved in the field regard it as a fundamental part of any process. A focus only on content is not suf f icient—any task also involves a process. As respondents were dis- cussing different elements, they would always raise questions about who was par- ticipating and in what ways. There is a general sense that all elements, from pur- pose through linkages, must have the issue of participation addressed within them. There is a degree of ambivalence with regard to the relationships between planning and the operations of nonformal education. There is general acceptance of the need for planning individual NFE activities, but most respondents believed that planning is only appropriate when the intended learners participate fully in the planning process. It is also widely agreed that it is important not to expend resources unnecessarily for education in developing countries where such resources are especially limited in supply. Related to this, there is a recognition that resource conservation is partially dependent on coordination. This seems to be where any agreement ends, however! Coordination can easily lead to control, and this concerns many NFE assessors. A valued characteristic of nonformal education is that its activities have 126 a tendency to be innovative. If these activities are coordinated within a highly con- trolled system, then it might become less easy, or even impossible, to make even small changes in programs. The concept of national planning for nonformal education concerns a large number of NFE experts for several reasons. First, there is a concern that the in- dividuality of separate NFE activities may be lost with focus on the general picture rather than on each individual activity. Second, there is a concern that those responsible for education planning do not understand nonformal educa- tion—especially the principles which distinguish it from formal education. And third, many believe that it violates a fundamental principle of nonformal education to have planning done by those who will not participate in the NFE activity. Regardless of who conducts or participates in an assessment, the respon- dents believe that both the process and the information produced belong to the host country. Ownership of information is an issue that is becoming increasingly sensitive, especially in developing countries. Whoever owns the information gathered for an NFE assessment can control how and which information is given to which decisionmakers and when. A commitment to place information under the control of host country nationals is a statement regarding where decisions should be made. ! . E 1' B 1 Initial attempts to identify the areas of inquiry focused on questions of the methodologies employed and the need to improve them. This was a result of a na- tional NFE assessment conducted in Botswana (September-October 1983) using a framework that was designed in 1982 and based on even earlier work in formal education. This activity was later discussed with colleagues who had performed a 127 1 Through the discussion it became apparent that this similar assessment in Niger. framework lacked appropriate structure and methodological flexibility to provide a detailed analysis of the components of nonformal education in Niger, a view of the complex interactions among components of the NFE subsector and between that sub- sector and the various components of formal education, and a recognition of the special characteristics of nonformal education. It became clear, moreover, that there was no general agreement on what components should comprise an NFE assessment, and that there was very little in- formation in the literature on this topic. It was obvious that an important first step in the development of a more appropriate methodology was the identification of the fundamental elements for an NFE assessment—the result of this research. Although a number of national NFE assessments have been conducted, they have not been systematically based on any theory or research. Nor is there much evidence that they have been closely linked to the existing literature describing the nature of nonformal education. This research is important because it represents the first attempt to identify the essential elements for an NFE assessment. This research not only addresses a need but also links theory and practice. It is solution oriented and addresses a real world problem in the education sector. It also concerns a subsector of education that traditionally has received very little at- tention even though adult and nonformal education are often expected to provide the majority of educational opportunities that formal education either cannot or has failed to provide. The research is built upon the experience of those who have con- ducted national NFE assessments. Although the respondents expressed some reserva- tions about planning nonformal education at the national level, there seemed to be a 1Discussion with Victor Barnes for the Bureau of Africa of the United States Agency for International Development, December, 1983. 128 general acceptance of the need for a better understanding of what exists within a country and for some coordination, especially given the limited resources for non- formal education. The focus on USAID-sponsored NFE assessments is appropriate for three reasons. First, the assessments were conducted recently so they should reflect cur- rent thinking in the field. Second, the names of the individuals who conducted these assessments, and the assessments themselves, were readily accessible. And third, much of the international donor community in education is closely monitor- ing USAID’s approach to education in which any project support for a country is based on the findings of an initial sector assessment. This community is hopeful that USAID’s work will serve as an example of a new approach to investment in education in developing countries. Thus, research that improves the assessments conducted by USAID will have positive effects far beyond the work supported by this single donor agency. W The seven respondents were an extremely rich source of data. They are highly educated and well trained with regard to nonformal education. Each brought a wealth of experience to the NFE assessment work—experience which would be drawn upon as they responded to the questions posed in this research. They were eager to share their opinions and experiences. All provided suggestions for how NFE assessment might be improved. During the design of this study, there was some concern was that open- ended questions might be responded to very briefly and that the questionnaires might not elicit sufficient data for analysis. This concern was unfounded! The initial question alone would often elicit as much as 20 minutes of continuous com- ments from the respondent. At this time the individual would identify key elements 129 and indicate why they were important. In many cases it was not necessary to ask most of the remaining items on the questionnaire, as the respondent had addressed them in one way or another in response to the first two questions. It also became clear that some of the questions were often either unnecessary or irrelevant. Thus, the precise structure of the questionnaire and the interview questions were not as important for data collection as originally anticipated. As early as the first interview it became apparent that the introductory statement was very important and not an aspect that could be glossed over. There were two sensitive areas with regard to this research. These related to sponsorship of the research and possible evaluation of the expert’s work. A relatively small number of individuals in the field of nonformal educa- tion have been involved in NFE assessment. Many individuals in this field are en- gaged in the private sector and so have to closely monitor the kinds of work that their other private sector counterparts are attracting and are engaged in. Once in- dividuals recognized that this research was not leading to some funding about which they were unaware, they were willing to speak openly— even pleased that their work had been noticed and their opinions were being sought. As with most professionals, these individuals recognized that their field work was probably not perfect. Many were initially concerned that the assessment that they had produced was going to be compared with others. Once it was clear that they were free to discuss assessments in terms of the best possible situation, rather than focusing on problems they had faced or covering up perceived weaknesses, they were eager to share their opinions and their experiences—both dif- ficulties and successes. It readily became apparent that trying to push for responses in areas that the respondents considered not relevant simply was not effective. For example, early in the interviews an attempt to get one respondent to elaborate on items such 130 as program content, duration, participants, and methods met with little success, even though these items eventually emerged as fundamental properties of the NFE program descriptions element. This particular respondent wanted to elaborate on the involvement of citizens in their own national NFE assessment—the conceptual area that eventually emerged as the roles category. Data collection for this research has two possible limitations to the research in addition to those noted in Chapter 1, both of which relate to non-independence of the data sources. First, four of the seven respondents had received their advanced academic training in nonformal education at the University of Massachusetts. This introduces a possible bias into the findings to the extent that that institution may emphasize particular issues or concerns in nonformal education. Of the three respondents who did not attend the University of Massachusetts, only one has received advanced level training in nonformal education. The other two have ad- vanced degrees in the social sciences. Second, the field of nonformal education in the United States is not large, and many professionals know each other personally. Even if they are not personally acquainted, they are probably know one another’s work. As a result, these NFE ex- perts may have been influenced by each other’s work. 3. [l . l B l D . The absence of any theory relating to NFE assessment provided an interest- ing challenge to the research design because there was not a set of assumptions or criteria to be tested. The grounded theory approach proved very useful. It provided the conceptual framework for work in an area without any developed theory and al- lowed the experience of experts to contribute to the initial development of substan- tive theory. In their interview responses, the respondents would usually list a few of the elements and then continue on to describe difficulties and obstacles that they 131 encountered in the conduct of their own work. Comparative analysis was used to restructure these difficulties as elements, each with identifiable properties. If these problem areas had been perceived as elements in the assessors’ work, and taken into consideration, they might not have emerged as such significant problems. Although it might be argued that the introduction to the questions would bias any responses toward support for national-level planning, the respondents felt quite free to express their concerns about planning nonformal education at the na- tional level. It was interesting that some of the reservations included concerns about protecting individual NFE activities from planners! Moreover, much of the information related to the policy environment and purpose categories stressed the need for more inidividuals than just planners to be involved in the process to ensure that nonformal education was not planned and managed by individuals or organiza- tions who were unaware of the characteristics that distinguish nonformal from f or- mal education. I 1' . There are several major implications of this research. These are discussed below. The research process demonstrated that individuals who have been respon- sible for NFE assessments have not often thought about their work in terms of the identified elements. Even though these eight elements were identified through the work of NFE assessors, these same assessors did not necessarily regard these as the elements. Their comments and observations suggested that they had not stepped back from their work to get a sense of this larger picture. Through the use of grounded theory, this research identified eight elements critical for effective national-level NFE assessment for planning purposes. Several of the elements identified through this research have not thus far been consistently incorporated into NFE assessments. A comparison between the eight assessments 132 conducted by the seven respondents and the eight elements indicates that no assess- ment covered all eight elements with equal effort. Most of the assessments covered several of the elements well, with the other elements receiving little or no attention. The NFE program descriptions element received the most consistent and detailed attention across assessments. Elements such as linkages and policy en- vironment received the least. The assessors are aware of this and it is quite ap- parent from the publications that resulted from the assessments. The respondents occasionally recognized the need to include some of these elements and did so during the assessment process. Information on categories that did not receive sig- nificant amounts of attention is more difficult to obtain, but respondents indicated the extent to which attention was given to such factors as roles of counterparts and the assessment process, even though questions were not specifically directed towards these topics. The limited attention to these elements, combined with the respondents’ statements about the need for a framework for conducting an NFE assessment, sug- gests that questions surrounding NFE assessment methodology are valid. It has been emphasized above that the identification of these essential elements is a separate task from the development of an assessment methodology. This research has defined what should be included in an NFE assessment, but it does not identify how to in- clude these elements. In fact, given the nature of the elements, it would seem likely that a variety of approaches could ensure attention to all elements. Existing methodologies have not produced equal consideration of all elements. Recent NFE assessments have covered only some of the elements in their reports, with a major emphasis on NFE program descriptions, even though the NFE experts indicate that other elements were an essential component of the assessments they conducted. The results of any assessment based on methodology that does not take all eight critical elements into account should be interpreted with caution. 133 These results should be generalizable beyond this research and beyond the field of nonformal education. The elements identified represent general conceptual categories for an NFE assessment. There is no reason to believe that they are valid only for the assessments that provided the data base for this research. For example, review of the NFE assessments described in Chapter 2 but not conducted by this study’s respondents indicates that at least some of these elements are present even though they may not have been so explicitly identified. These elements also may be essential for other areas of assessment. The re- searcher has been responsible for assessment of other subsectors (preprimary, primary, and vocational/technical education) in addition to nonformal education and, upon reflection, it appears that these elements are a fundamental part of any assessment. Given this, it is highly likely that these elements would hold for a field even more closely related to nonformal education, such as adult education. Watch This research was designed as an initial incursion into a domain where limited information is available. As a result there are several possible directions that are suggested for future research. The first step in follow-up research should be to cross-validate the identification of the eight critical elements. One approach would be to develop hypotheses related to the eight elements and then to test them. An alternative would be to replicate the research process using a different group of experts or to go beyond these assessments to non-USAID-sponsored activities. Per- haps the most important next step would be to test the generalizability of the f ind- ings with regard to other domains of study. Because of the close relationship be- tween nonformal and adult education, it would seem apprOpriate to make this the next focal area. It might then be advisable to continue with research on other sub- sectors of education. Should such tests yield additional elements, it would be 134 necessary to identify whether any of these additional ones should also be included in N FE assessments. Knowledge about the eight critical elements should provide helpful inf or- mation for the development of methodological improvements for NFE assessment. Knowledge of these elements and their conceptual properties makes it possible to assess and identify the degree to which existing methodologies can incorporate them. It might be easy to refine or adapt existing approaches to make them responsive to the eight elements. Or, it may be possible that NFE assessment should be reexamined and a different methodology developed. Should the latter be necessary, the new methodology should be flexible so that it can be applied to a variety of contexts. Results of the research have emphasized the need for better indicators of the outcomes and benefits of nonformal education, even though this was not the in- tended focus of the research. Areas in particular need are achievement, benefits to learners, secondary benefits, and indicators on the texture or richness of the learn- ing experience. It may not immediately be apparent why research on these topics could contribute to NFE assessment. Earlier, the importance of examining the inter- relationships within the NFE program category was stressed. Research in these areas would facilitate an understanding of these interrelationships. Information on these interrelationships would make it much easier for planners to understand NFE activities and to evaluate them for planning purposes. Recommendations As noted in the first chapter, this research is a preliminary step and there- fore its results should be interpreted with some caution. Nevertheless, it is possible to use the findings to guide both research and practice. The following recommenda- tions are suggested. 135 B 1.0.2. "I III' l'EEI set 9! resnendents. In particular it would be important to include developing country nationals as respondents. Their inclusion could present a very different perspective on national NFE assessment. Such a study would provide an oppor- tunity to test whether the elements identified through this study are also viewed as either some or all of the critical elements by insiders. B 1°IW'EI 11.11”]!- 91" ~11 1 _‘1:=ll 1m ~ _ l-o u .... c: . Forexample, a carefully constructed questionnaire could be developed and administered to education professionals familiar with NFE assessment. This would serve the pur- pose of identifying critical elements through a methodology that is less dependent on the researcher’s interpretation at the analysis stage than is grounded theory. This approach would also provide an opportunity to explore such things as the prof es- sional training of the NFE assessor which was not identified as either a property or an element but which may be important to NFE assessment. WNW Conductins this kind of study would present some significant research challenges, as it would prob- ably require a fairly large sample size which might be difficult to find in the fairly limited field of nonformal education. Recommendatianim: MW ”Av“. ....._.:'.. ny'M This would allow for better delimitation of each of the properties, and it would also provide more information on the nature of the relationships among elements and properties. “11105.1. 1‘ III'WWII'E'III'ii'x 136 though these may not be all the critical elements, they have been identified as im- portant elements to NFE assessment by those who have actually conducted NFE assessments. This research provides some evidence that attention to these elements could improve the practice of NFE assessment and through that national planning of nonformal education. B l' S'3E l . . °IIIEE _ ,,,,..-. .5: 1......"..”.‘.:" ”....M this reseereh. The properties are the conceptual aspects of each element. Attention to the properties should afford NFE professionals a much better understanding of the wide range of factors that affect their work. In many cases this understanding could improve practice significantly. Summers This research is a significant initial stage in knowledge building about planning as it relates to nonformal education. It has identified eight elements essen- tial for an effective national-level NFE assessment. These are purpose, preparation, policy environment, process, roles, methodology, NFE program descriptions, and linkages. The research was conducted through the use of grounded theory, an ap- proach designed for practical fields of study where little theoretical work has been done. The findings of this research can improve the practice of NFE assessment immediately. Moreover, it is a basis for additional research. B [I . l . S l The initial decision to focus on the topic of NFE assessment was a result of the author’s reaction to and wish to improve upon some of the work in which she and colleagues were engaged in several developing countries. As a consultant in 1983, she was a member of a team that conducted an education assessment in 137 Botswana. Her responsibility was to assess nonformal education. Her approach to NFE assessment differed significantly from earlier attempts. Nevertheless, she remained dissatisfied with the methodology, feeling that it did not allow sufficient latitude to assess the variety of approaches, content areas, organizational structures, and target learners that constitute nonformal education in a given nation. At this point dissatisfaction was directed at the existing methodologies for NFE assessment. Even though nonformal education often has low priority within ministries of education, it was recognized that NFE assessments would continue to be conducted and that decisions would be made based on these assessments. An ap- propriate methodology was necessary, therefore, or decisions would be made on par- tial or incorrect information. Initial problem definition for this study focused on developing an improved methodology for NFE assessment. As problem definition became increasingly frustrating, the committee chair suggested that perhaps it was because there was ap- parently no body of knowledge that could guide decisions regarding what should be incorporated into a more appropriate methodology. The logical outcome of this conversation was that perhaps the methodology question should not, and could not, be addressed until some initial work had been done regarding critical elements for inclusion in a national-level assessment. A qualitative research methodology was selected for two reasons. It was selected, in part, because of the renewed interest in and recognition of the value of qualitative research in the field of education. It also provided the researcher with an alternative approach to quantitative analysis which she had employed in her master’s thesis. The methodology was used with full recognition that it could be seriously biased by the author’s previous involvement in an NFE assessment and interpretation. This also provided an important challenge. Was the methodology as 138 rigorous as Glaser and Strauss claim, and was the author able to interpret the data without being wedded to the approach and interpretations she had employed earlier? The methodology appears to have been rigorous enough to withstand significant bias, as the research identified elements that the author had not expected to find. This suggests that grounded theory can serve the purposes for which it was designed if the research procedures are followed carefully. The exercise required patience and an ability to deal with ambiguity on the part of the researcher, however. Fol- lowing the procedures carefully meant that there were often data that did not seem to immediately fit into neat categories and that they had to be kept in abeyance as other data were collected, and then analyzed at a later time. In many ways, this qualitative approach made the research task much more difficult than a very struc- tured or quantitative one. Completion of this research fulfills at least one of its purposes—it should improve the researcher’s practical work! In retrospect, the work she did in Botswana, which was considered innovative and a major improvement in NFE assessment, was somewhat limited as it did not pay sufficient attention to all the elements identified through this research. It is also recognized that this research is a preliminary step and there might be additional elements that should be considered for NFE assessment if methodological approaches are to be improved. BIBLIOGRAPHY BIBLIOGRAPHY Academy for Educational Development. Guatemala Education Sector Assessment. Washington, D.C.: Academy for Educational Development. 1978. Academy for Educational Development. Nonformal Education: A Decade of Experience. Washington, D.C.: Academy for Educational Development. n.d. Academy for Educational Development. Paraguay Education Sector Assessment, 1977. Washington, D.C.: Academy for Educational Development. 1977. Adams, Emily Vargas and M. Joseph Bastian. Non-Formal Education in Lesotho: A National Study for Policy Planning and Program Development. Washington, D.C.: Agency for International Development, Bureau of Science and Technology. 1983. Agency for International Development, Bureau of Latin America. Educacion No Formal. Documento Metodologico de Trabajo #7. Washington, D.C.: Agency for In- ternational Development, Bureau for Latin America. 1977. Ahmed, Manzoor and Philip H. Coombs, eds. Education for Rural Development: Case Studies for Planners. New York: Praeger Publishers. 1975. Axinn, George and Nancy Axinn. Non-Formal Education and Rural Development. Supplementary Paper No. 7. East Lansing, Michigan: Institute for International Studies in Education, Michigan State University. 1976. Bereday, George Z. F., ed. Essays on World Education: The Crisis of Supply and Demand. New York: Oxford University Press, 1969. Blair, Philip M. Comments on Education Sector Analysis: Methodological Working Documents Nos 18, I9, and 20. Discussion Paper No. 7. Washington, D.C.: Agency for International Development, Bureau for Latin America. 1974. Bock, John C. and George S. Papagiannis. Nonformal Education and National Development: A Critical Assessment of Policy, Research, and Practice. New York: Praeger Publishers. 1983. Botswana, Department of Non-Formal Education and Creative Associates, Inc. An Assessment and Analysis of Non-Formal Education in Botswana. Gaborone: Govern- ment of Botswana. 1983. Botswana, Ministry of Finance and Development Planning and the Interministerial Steering Committee with USAID. Botswana: Education and Human Resources Sector Assessment. June 1984. Gaborone: Government of Botswana. 1984. Brembeck, Cole S. Formal Education. Non-Formal Education. and Expanded Concep- tions of Development. Occasional Paper No. 1. East Lansing, Michigan: Non-Formal Education Information Center, Michigan State University. 1978. 139 -140- Brembeck, Cole S. New Strategies for Educational Development. Lexington, Massachusetts: D. C. Heath and Company. 1973. Cain, Bonnie J. Nonformal Education Assessment/Analysis Model. Washington, D.C.: Creative Associates, Inc. n.d. Case, L. Harry and Richard O. Niehoff. Educational Alternatives in National Development: Suggestions for Policy Makers. Supplementary Paper No. 4. East Lansing, Michigan: Institute for International Studies in Education, Michigan State University. 1976. Cieutat, Victor J. Planning and Managing an Education Sector Analysis. Washington, D.C.: United States Agency for International Development. 1983. Chang, Paul. Comparative Analysis of Non-Formal Education in SEAMEO Member Countries. Unpublished paper presented at the SEAMEO/SEADAG Seminar on Non- Formal Education. 1971. Coles, Wendy Lowe. Development of a Survey Method to be Used for the Identification of a Community’s Lifelong Education Activities. Dissertation for the degree of Doctor of Education. Michigan State University. 1978. Coombs, Philip H. The World Crisis in Education: A View from the Eighties. New York: Oxford University Press. 1985. Coombs, Philip H. with Manzoor Ahmed. Attacking Rural Poverty: How Nonformal Education Can Help. Washington, D.C.: The World Bank. 1974. Coombs, Philip H. with Roy C. Prosser and Manzoor Ahmed. New Paths to Learning for Rural Children and Youth . New York: International Council for Educational Development. 1973. Creative Associates, Inc. Final Report: Nonformal Education Assessment/Analysis Project. Washington, D.C.: Creative Associates. 1983. Darkenwald, Gordon G. “Field Research and Ground Theory” in Long, Huey B., Roger Hiemstra and Associates, eds. Changing Approaches to Studying Adult Education. San Francisco: Jossey-Bass Publishers. 1980. pp. 63-77. Diebold de Cruz, Paula, ed. Executive Summary of the El Salvador Education Sector Analysis. Washington, D.C.: Academy for Educational Development. 1978. de Oliveira, Homero F. and George E. Lewis. Alternatives for USAID Policy and Assistance in Non-Formal Education in Brazil. Mimeograph. 1973. Education and World Affairs, Committee on Human Resource Development. Nigerian Human Resource Development and Utilization. New York: Education and World Affairs. 1967 Evans, David R. The Learning Opportunities Inventory: Preparing Comprehensive In- ventories in the Field of Education. Paris: International Institute for Educational Planning. 1983. ~141- Evans, David R. The Planning of Nonformal Education. Fundamentals of Educa- tional Planning-30. Paris: Internatioanl Institute for Educational Planning. 1981. Federal Ministry of Education. Investment in Education. Report of the Committee on Post-School Certificate and Higher Education in Nigeria. Lagos: Federal Minis- try of Education. 1960. Freire, Paulo. Cultural Action for Freedom. Harmondsworth, Middlesex, England: Penguin Books Ltd. 1972. Freire, Paulo. Pedagogy of the Oppressed. New York: Herder and Herder. 1971. Gideon, Zohra Miriam. Emergent Status Passage: A Case Study of Transition, Change and Adult Learning in a Religious Order of Women in the Catholic Church. Dissertation for the Degree of Doctor of Education. Michigan State University. 1985. Gilette, Arthur Lavery. Beyond the Non-Formal Fashion: Towards Educational Revolution in Tanzania. Amherst, Massachusetts: Center for International Education. 1977. Glaser, Barney G. and Anselm L. Strauss. The Discovery of Grounded Theory. Chicago: Aldine Publishing Company. 1967. Government of Tanganyika. Five-Year Plan for Economic and Social Development. Dar-es-Salaam: Government Printing Office. 1964 Gross, Ronald, ed. Invitation to Lifelong Learning. Chicago: Follett Publishing Company. 1982. Grandstaf f, Marvin. Alternatives in Education: A Summary View of Research and Analysis on the Concept of Non-Formal Education. East Lansing, Michigan: Institute for International Studies in Education, Michigan State University. 19743. Grandstaf f, Marvin. Historical Perspectives on Non-Formal Education. East Lansing, Michigan: Institute for International Studies in Education, Michigan State University. 1974b. Harbison, Frederick. National Development and Non-Formal Education. Unpublished paper presented at the Seminar on Non-Formal Education conducted by the Southeast Asia Development Assistance Group. 1971. Harbison, Frederick H. Education Sector Planning for Development of Nation-Wide Learning Systems. OLC Paper No. 2. Washington, D.C.: American Council on Education, Overseas Liaison Committee. 1973. Harbison, Frederick H. The Development of Nation-Wide Learning Systems: A Sector Approach for Assessment of National Development from a Human Resources Perspective. Washington D.C.: Agency for International Development, Technical Assistance Bureau. 1973. Harbison, Frederick H. Human Resources as the Wealth of Nations. New York: Ox- ford University Press. 1969. 173 p. -142- Harbison, Frederick H. A Human Resource Approach to the Development of African Nations. Washington, D.C.: American Council on Education. n.d. 24 p. Heyneman, Stephen P. The Evaluation of Human Capital in Malawi. World Bank Staff Working Paper No. 420. Washington, D.C.: The World Bank. 1980. 102 p. IDRC. Teaching Yourself in Primary School. Ottawa: International Development Re- search Centre. 1981. 108 p. Institute for International Research. Haiti: Education and Human Resources Sector Assessment 1985 (Draft). McLean, Virginia: Institute for International Research. 1985. International Institute for Educational Planning. "The Diversification of the Educational Field” in The Prospects of Educational Planning Related to Contemporary Development Problems. Working Paper IIEP/S92/1E. Paris: International Institute for Educational Planning. 1983. International Labour Office. Employment, Incomes and Equality: A Strategy for In- creasing Productive Employment in Kenya. Geneva: ILO. 1972. International Labour Office. Matching Employment Opportunities and Expectations: A Programme of Action for Ceylon. (2 Volumes). Geneva: ILO. 1971. International Labour Off ice. Towards Full Employment: A Programme for Colombia. Geneva: ILO. 1970. Kidd, J. R. How Adults Learn. Chicago: Follett Publishing Company. 1973. Kindervater, Suzanne. Nonformal Education as an Empowering Process. Amherst, Massachusetts: Center for International Education. 1979. Kleis, Russell J., Charles L. Lang, John R. Mietus, and Fia T. S. Tiapula. ”Toward a Contextual Definition of Non-Formal Education.” In Non-Formal Education: The Definitional Problem. pp. 1-7. East Lansing, Michigan: Institute for International Studies in Education, Michigan State University. 1974. Knowles, Malcolm S. The Modern Practice of Adult Education: From Pedagogy to Andragogy. Chicago: Follett Publishing Company. 1980. 400 p. Knowles, Malcolm S. Self-Directed Learning: A Guide for Learners and Teachers. Chicago: Follett Publishing Company. 1975. 135 p. Krueger, Chris and Jeanne Moulton. DS/ED Nonformal Education: A Retrospective Study I970-1980. SF-l47-PE-86. Unpublished paper. 1981. 77 p. Lappe, Francis Moore, Joseph Collins and David Kinley. Aid as Obstacle: Twenty Questions about our Foreign Aid and the Hungry. San Francisco: Institute for Food and Development Policy. 1980. Lesotho, Ministry of Education, Sports and Culture. The Education Sector Survey: Report of the Task Force . Maseru: Government of Lesotho. 1982. -143- Liberia, Ministry of Finance and Economic Affairs: Joint Committee for the Government of Liberia/USAID Education, Training, and Human Resources Assessment. Liberia: Education and Training Sector Assessment. Monrovia: Government of Liberia. 1983. Method, Francis J. and Saundria K. Shaw. AID Assistance to Education: A Retrospec- tive Study. Washington, D.C.: Creative Associates, Inc. n.d. Merriam, Sharan B. and Edwin L. Simpson. A Guide to Research for Educators and Trainers of Adults . Malabar, FL: Krieger Publishing Company. 1984. Morgan, Robert M. and Clifton B. Chadwick. Systems Analysis for Educational Change: The Republic of Korea. Tallahassee, FL: Florida State University. 1971. Niehof f, Richard 0. Educational Alternatives in National Development: Suggestions for Policy Makers. Supplementary Paper No. 4. East Lansing, Michigan: Institute for International Studies in Education, Michigan State University. 1976. Niehof f, Richard O. and Bernard Wilder. Non-Formal Education in Ethiopia: Literacy Programs. Discussion Paper No. 5. East Lansing, Michigan: Institute for Interna- tional Studies in Education, Michigan State University. 1974. Niehof f, Richard O. and Bernard Wilder. Non-Formal Education in Ethiopia: The Modern Sector. Discussion Paper No. 6. East Lansing, Michigan: Institute for Inter- national Studies in Education, Michigan State University. 1974. Overseas Liaison Committee. Tanzania: A Nation-Wide Learning System. New York: American Council on Education. 1971. Perraton, Hilary ed. Alternative Routes to Formal Education: Distance Teaching for School Equivalency. A World Bank Research Publication. Baltimore and London: The Johns Hopkins University Press. 1982. Robinson, Brandon. On Methodology for Education Sector Analysis. Washinton, D.C.: Agency for International Development, Bureau for Latin America. 1975. Robinson, Brandon. Some Special Problems and Characteristics of an Education Sector Analysis. Methodological Working Document No. 18. Washington, D.C.: Agency for International Development, Bureau for Latin America. 1973. Robinson, Brandon. A Brief Description of an Educational Sector Analysis Methodology. Methodological Working Document No. 19. Washington, D.C.: Agency for International Development, Bureau for Latin America. 1973. Robinson, Brandon. Questions and Answers Concerning Methodological Working Docu- ment #19, "A Brief Description of an Education Sector Analysis Methodology". Methodological Working Document No. 20. Washington, D.C.: Agency for Interna- tional Development, Bureau for Latin America. 1973. Robinson, Brandon. A Preliminary Examination of a Limited Collection of Statistics for the Analysis of the Education Sector in Panama. Analytical Working Document No. 7. Washington, D.C.: Agency for International Development, Bureau for Latin America. 1972. -144- Robinson, Brandon. Some of the Limitations of Education Sector Analysis. Methodological Working Document No. 21. Washington, D.C.: Agency for Interna- tional Development, Bureau for Latin America. 1974. Rogers, Daniel C. An Overview of Past Education Sector Loans and Some Points for the Future. Washington, D.C.: USAID, Bureau of Program and Policy Coordination. 1972. Rural Non-Formal Education: Inventory and Assessment of Non-Formal Education in Paraguay. Mimeograph. n.d. Ryan, J., ed. Planning Out-of-School Education for Development. Paris: International Institute for Educational Planning. 1972. Sheffield, James R. and Victor P. Diejomah. Non-Formal Education in African Development. New York: African American Institute. 1972. Somalia, Ministry of National Planning with USAID. Somalia: Education and Human Resource Sector Assessment. Mogadishu: Government of the Somali Democratic Republic. 1974. Southeast Asian Ministers of Education Organization. A Study of Non-Formal Education in the SEAMEO Region, 19734974. Preliminary Report. Bankok: SEAMES. 1974. Spector, Paul. Dissemination of the Interactive-Radio Model. Unpublished mimeo. n.d. Srinivasan, Lyra. Perspectives on Nonformal Adult Learning. New York: World Education. 1977. Tough, Allen. Intentional Changes: A Fresh Approach to Helping People Change. Chicago: Follett Publishing Company. 1982. Unesco. International Standard Classification of Education. Vols. 1 and 2. COM/ST/ISCED. Paris: Unesco. 1976. Unesco. Manual for the Collection of Adult Education Statistics. CRS/E/lS. Paris: Unesco. 1975. United Republic of Tanzania. Second Five-Year Plan for Economic and Social Development. Dar-es-Salaam: Government Printing Office. 1969. Velandia, Wilson B., Emily Vargas Adams, and Gilberto Bello D. La Educacion No— Formal en Colombia: Hacia un Diognostico de su Realidad. Bogota, Colombia: Centro Para el Desarrollo de la Educacion No-Formal. 1975. Wheeler, David. Human Resource Development and Economic Growth in Developing Countries: A Simulation Model. World Bank Staff Working Paper No. 407. Washington, D.C.: The World Bank. 1980. World Bank. World Development Report I984. Washington, D.C.: The World Bank. 1984. -145- World Bank. Accelerated Development in Sub-Saharan Africa: An Agenda for Action. Washington, D.C.: The World Bank. 1981. World Bank. Education Sector Policy Paper. Washington, DC: The World Bank. 1980. Yemen Arab Republic Educational Research and Development Center. Yemen Arab Republic: Education and Human Resources Sector Assessment 1985 (Draft). Govern- ment of the Yemen Arab Republic. 1985. APPENDICES APPENDIX A APPENDIX A INDIVIDUALS INTERVIEWED Emily Vargas Adams, Center for the Development of Nonformal Education, Austin, Texas Stephen Anzalone, Institute for International Research, McLean, Viginia Bonnie Cain, B. J. Cain and Associates, Inc., Washington, D. C. John Comings, World Education, Boston, Massachusetts David Kahler, World Education, Boston, Massachusetts May Rihani, Creative Associates, Washington, D. C. Lela Vandenberg, Michigan State University, East Lansing, Michigan A-l APPENDIX B APPENDIX B INTRODUCTION AND INTERVIEW QUESTIONS I am calling you because of your expertise in nonformal education. I am aware of the NFE assessment you did in for AID in l98_, and I’d like to ex- plore with you some of your reactions to that work. In particular, I’m interested in elements which you believe are essential to an assessment. As you know, a number of national-level assessments have been conducted since 1980. They display a rich variety of approaches, purposes, and results. I am engaged in some independent re- search (that is, nonsponsored research) to identify the critical elements in NFE assessment for purposes of national planning. Clearly, one of the best sources of in- formation regarding NFE assessment and its critical elements are those who have both knowledge and firsthand experience in the area. I have a few questions to ask you. Responding to these questions does not require any special preparation. Each one of us who has conducted an assessment has our own perspective and a sense of what are the most important aspects to be considered in a national-level NFE assessment. I want to take a few minutes of your time to think through what are the most important elements in a national -level NFE assessment. Please remember, we’ll be talking about NFE assessment at the national level. I am not evaluating the assessments that have been conducted so far or interested in the evaluation of a particular NFE activity or program. 1. Based on your experience, what are the critical elements in NFE assessment at the national level? 2. Let’s go over the list, and I’d like you to tell me why you think each of these is important. 3. You’ve identified . . . as key elements and given reasons for their importance. What other elements can you think of? 4. Why are these others important? 5. Now you have given me elements. Organize them for me. (Do you see them falling into any particular kind of order or structure?) 6. Please categorize the elements in terms of their importance for others who will be doing NFE assessments. 7. Why are these the key ones? 8. Of these that you have identified as most important, which ones did you use in your work? What data did you use to indicate each element? B-l 10. ll. 12. 13. B-2 Are there any key elements which are hard to use? (For example, are there some for which it is particularly hard to get data?) Which ones? What were the difficulties you encountered? How might these be changed? Do you see other NFE experts using elements other than those you have noted? What are they? Do you have any comments about these elements? What was your role in the assessment in relation to other professionals and host country institutions, for example? If you had to do the NFE assessment over, how would you change it? APPENDIX C APPENDIX C INTERVIEW TRANSCRIPTION Interviewer: What I mean by an element is, those components on which you collect data or are a key to a process of doing the assessment. For example, a factor that would really affect the credibility and the quality if you didn’t it into account. Respondent: Or validity even? Interviewer: Yes. Respondent: So you want to know not just what elements should have been included that weren’t included in the one I did but, in general, list— Interviewer: Yes. A list of components or a list of key items or key categories you think should be addressed. Respondent: It would probably be easiest for me to think of what we did in Haiti. I mean that’s a framework that’s already provided and maybe that gives some kind of bias though, but— Interviewer: That’s OK because I’m going to ask you questions about what didn’t you use, what didn’t you do that you would do differently, so the bias is fine. Respondent: Maybe you should turn it off while I think. I guess there was descrip- tion of the various types of nonformal education programs being done and there was, then, the analysis. Issues was the second part of that. The first part was the needs, constraints, problems, all of that, and that seemed like a reasonable format. You’re not asking me to talk about format, though. I guess the issues may be where I had some trouble and where I might have done it differently. I may have included different issues and divided the issues differently. I’ll go back to the issues part later. The description would have to include things like the organization and management of each program, the planning of the program, and that would include the curriculum development and I think that’s something that wasn’t included when I did Haiti. Something I didn’t think of, or maybe I didn’t have any information. Interviewer: When you say curriculum development do you mean how the cur- riculum is developed, or what it is, or both. Respondent: Both. How would include by whom, and what the content, and why that content given the audience, and why those people planning the content, given the audience. Maybe focusing on the curriculum development process a little bit more in the planning might have helped focus on how participants were or were not C-l C-2 participating in the decisionmaking. That is, I don’t think I had much of that in the descriptive part. Though I got a feeling for that and talked about it in the issues part. But it would have been better to focus more on it in the descriptive part. First planning, then participants. Who they are. And the trainers or the facilitators. Who are the pe0ple teaching the nonformal education and how are they trained so we look at the facilitators and the trainers. How are the facilitators trained? That would be another aspect of facilitators. And then how are the trainers trained and what is their education? What are their qualifications as trainers? It’s a three-level thing, I think. And then, the content of the NFE and the content of the facilitators’ training. I think that’s important too. And the methods both of the nonformal education which the facilitators engage in with the participants and also the method of training the facilitators. Then evaluation is another element. Interviewer: Evaluation of the NFE program? Respondent: Right! Of the programs in the descriptive part. What kinds of evalua- tions have been done and what have they found. Somewhere I should have put more focus on benefits. I don’t know if it would come here, describing the kinds of benefits that people said occured for participants or if it would be better in the issues section. Who benefited and how. A little more focus on the benefits. What else would be in description? I probably forgot some things. I can’t think of them now. Interviewer: Basically you’ve got then, who participates at what level, who are they and how are they participating, what are they participating in, and what are they gaining from it, and how is any gain being measured. Respondent: Right. Also, what kinds of programs; that would be another aspect. I guess its determined by the country itself. Right? Like in Steve’s case they just wanted him to look at that one nonformal education program. But it seems like you did in Botswana, that nonformal education is so broad that you can’t consider it without considering agriculture and health and literacy or basic education. And you did a lot more in Botswana. Well, you did vocational/technical, and that was separated out, and I guess that’s a good thing. I guess I‘m talking about the struc- ture again, too. Income generating occurs in all of those, or could occur in health, agriculture, especially agriculture. But, maybe there should be some separate com- ponent in addition to agriculture, health, literacy, and basic education called income generation, or economic gains, or whatever gives a description of kinds of programs that actually help people economically. And these programs could fall under health, agriculture or— 80 this component could be distilled out of the other sections to stress the economic advantages that certain projects help pe0ple gain. Another thing that I think is important which probably could be done by somebody else but it seems to me that if fits with the nonformal education assess- ment would be a kind of preliminary study or analysis of the development philosophy or strategy—general development strategy of the government— and then how the different nonformal education activities fit into that, or how they are aimed to fit in with the development strategy. I felt the lack of that in Haiti, anyway. I guess I’m answering two of your questions at the same time. Probably I should have done some kind of overview of development strategy and the way that nonformal education was working in support of or against it. And then part of the recommendation section, or maybe an issue could have been on the connection be- tween nonformal education and development. To consider the role nonformal education is playing given the government’s state of development and approach, and G3 the role that it could be playing. Because it can play either an adversary role or a supporting role. I think it was doing both in Haiti. Another element—I’ve just been talking about descriptions of specific programs. I guess the description of the NFE subsector itself would include the or- ganization and management of say the agriculture component, the health component, the literacy or whatever, the basic education, community development. Oh that’s another I should have listed up there, community development. I think is another area of nonformal education. At least the way I see it. I think it should in- clude those separate subsectors of nonformal education and the general organization of those subsectors, each of them, in both government and nongovernment which you did in Botswana. But I didn’t do in Haiti. And then maybe a section on the linkage between govenment and nongovernment, how they work together or don’t work together. And probably a section on how the formal and the nonformal work together or don’t and the linkages between them. And that might come of out of a section on the general organization of the subsector. I don’t know about the analysis. I guess after description comes the section on needs, constraints, and problems. I think that’s important but I guess when I was writing it I felt a little bit like I was just trying to fill in the blanks. For me most of the analysis actually came out in the issues section. I’m not sure that the needs and constraints part really brought it out. Well, I guess it was just a way of analyz- ing it in a way, and I suppose that was good. Maybe if I went back and looked at that again I would see that that was important. But I think the analysis happens in the issues section, too. And I guess its supposed to. Somehow the conclusions and recommendations have to lead from the dis- cussion in the issues section, and all the previous description has to kind of lead to the issues section. So it seems like the issues section is the big one, the important one, anyway, that ties it all together. I would include participation as a separate issue. I struggled with trying to fit it in both under external efficiency and internal efficiency, but I think it really should be a separate issue to be analyzed by itself. And that would include lots of things, like how participants were participating in all the different steps—organization, management, planning, training, the education process itself, choice of content, choice of methods, the evaluation, and benefits. And then the relationship of the different levels of participation to the benefits gained because I don’t think it’s necessary that all participants participate in every level and every aspect of the project. And then the issue related to that of people participating as individuals versus group participation. But all that is part of the participation issue. Access and equity is an important issue in nonformal education. And there I guess— Should I explain what I mean? Interviewer: Yes, go ahead. If you want to give a why. Why you consider it so critical to nonformal education. Respondent: Its critical really to any education but its critical to NFE because I think its much easier to design NFE programs for people who are a little better off than the majority. It’s easier to get their participation, for one thing, and it’s easier to get a successful program if you’ve got those kinds of participants. Its much more difficult to get NFE to the poorest, or the people who need it the most, and have it be successful. So I think there is a big tendency, or there could be a tendency to reach the upper crust of the local level people. And I think that happened in Haiti. And then also just because of the distance and geography of places access is an issue. Concerns include how are people actually being reached and the cost of send- ing people out to outlying areas to try to bring people some form of education. And then, is it for women, or is it for men, or is it for children? You know, the problem (3-4 of programs being designed for women when it’s actually the young unmarried girls are the ones who come to the classes. Or programs are designed for male farmers when its really the women who are doing the work. That’s why its important to have nonformal education. Cost and financing would be something important to discuss just because so little money is available for nonformal education. It’s important to talk about where that money comes from and how. You know, to think of creative ways of financing nonformal education. So analyzing the ways that nonformal education is financed in a country, either locally or a combination of funds from other sources, would be important. About internal efficiency and external ef f iciency— I think that you can use those terms to get at some of the important issues. But I think they are a little misleading just because they are used in a much more economic sense when you look at the other subsectors of education than in nonformal education. I think that un- less somebody carefully reads those sections, they might misinterpret what we’re trying to do when we’re analyzing the internal and external efficiencies of nonfor- mal education. Interviewer: By using the words, “they might misinterpret what we’re trying to do,“ are you making the argument that we are not as scientific as the other subsectors, that we aren’t as concerned about money, or something else? I just want a clarifica- tion on that. Respondent: It’s not that we aren’t concerned about money or that we don’t want to be scientific, but I think we can’t be. Interviewer: We can’t be scientific. Respondent: Yeah. We can’t be as number oriented, just because there aren’t the data. But also because even if there were data on cost, that doesn’t tell the whole story. I think efficiency, and that’s what I think both of us tried to say in those sections, in nonformal education is not only determined by spending the least amount of money for the maximum gain. Well it is. But then you have to define what you mean by maximum gain or effectiveness. Interviewer: So one of the questions that we have to deal with, if I understand what you’re saying, in terms of internal efficiency is the whole question of achievement and what we have gained? Respondent: Yeah. Interviewer: And essentially the frameworks we have now don’t provide the rest of the equation by any stretch of the imagination. Respondent: There has to be some thought put to that and a little more analysis of what achievement means. Because it could mean that reaching a handful of people, which is much more expensive than reaching a lot, is really much more efficient be- cause of the results—the benefits gained by those few people that may be not gained by a whole bunch of people who could be reached by radio or something. Interviewer: You’re not the only who has mentioned this. It’s a real problem. Respondent: There was a fifth issue. Can you— Management, that was it. C-S Interviewer: But you have talked about that before. In the descriptive piece. And I guess a question I would ask is this. Using this particular framework to discuss issues, is it possible to address management and administration issues or are we really talking more about the policy environment? I don’t think in many of the assessment we’ve necessarily done that. But its something that has come out as I’ve talked to people. Respondent: Say that again. Interviewer: Are we really talking about . . . Using this framework, management and administration has often looked at the level of training the people need and the bureaucratic structure within programs, but to a very small extent the bureaucratic structure in which decisions are made. And the question I would raise is, would it be more useful to look at the policy environment and how decisions are made? And who makes decisions given that nonformal education is usually implemented by a lot of government and nongovernment organizations? Respondent: Yes. I think that would be much more useful. That would be really complicated. I mean to try to see how those decisions are made in the different parts of government, where they are made, and then how those decisions get coor- dinated or whatever, if they do. But yes, I think that would be a much better way of looking at the policy environment. Looking at just the management of each sub- component of nonformal education like agriculture and health seems to be sort of internal. How they are managed on a more global level would be that policy en- vironment that you are talking about? Interviewer: Yes, although there are some that would argue that that really is coun- ter to nonformal education. Respondent: Yeah, that’s true. I guess it all just depends where the money comes from. Maybe that cost and financing issue is a real important one because if it is big dollars coming from the World Bank or USAID through the government, or even through some large NGO’s, then I think the policy environment decisionmaking is important. I mean the money has to be managed and distributed somehow, and put into programs. Programs have to be planned and then the money has to be ac- counted f or. But, if the money is obtained more creatively, more locally, and from different smaller sources, then the issue of the policy environment might not be as important. And that’s the way I would really like to see nonformal education work and I think its more successful. It’s just an opinion. When it does work that way, that’s more haphazard too. Interviewer: Can you let me just summarize what I think you’ve said? And you can correct me. Basically, you plan to use the Haiti example because it provides a framework which, although it may have some problems, nevertheless, was a framework. You argued that the description was important within the NFE subsector: the range of programs that are in operation; how they are organized managed; and how the planning of the curriculum is done, by whom, how, what areas, and in relation to the audience. And, your sense is that perhaps by looking at the curriculum, you have a much better sense of how people participate which is an issue you raised later as well. Who the participants are, who the facilitators are, who the trainers are, and in each case, how these are trained—not only in terms of the content of the training but also the methodology. Then to look at the content of the NFE program and the content of the training. And to pay particular attention to evaluation, and related to this is the question of benefits. C-6 Respondent: Also, well the methods, the methods of training, but I guess you al- ready said that. Interviewer: And then, although you recognize that a country might determine what it wants looked at in terms of the assessment, you believe its important to look at the range of activities in various content areas such as agriculture, health, basic education, community development, and income generation. And, in particular, that perhaps more attention might be given to income generation and the kinds of programs that help people financially. Respondent: Across all of those other four. Cause I don’t think its something separate. There can be income generating programs in agriculture, health, basic education and community development. Interviewer: You thought it would be important in the context of the assessment, to have a preliminary study of what the development strategy is and then to make sure, in looking at the various programs, how they link or how they are either in concert or at odds with the existing strategy. And you also saw as important to look at the linkages in nonformal between nongovernment and government. And also to look at the linkage between nonformal education and formal education. In terms of the needs and constraints you saw those as an important area to get at, but hard to get at contextually. And you said perhaps this is easier to do in the issues section. You felt that the issues section is really, in the IEES framework, the critical section, that tied the beginning and the end together. You would have liked to have seen participation in there as a key issue—one that is hard to deal with in the existing framework. How participants participate in all the steps of the program, the relationship that links the type of participation to benefits gained, and the issues of individuals versus group participation. You argued that access and equity is very important and that one of the real critical axes on which to analyze something is the extent to which NFE programs are designed for and reaching the hard-to—reach groups. Within any com- munity there is a structure or hierarchy and often its easiest to reach those at the top of the hierarchy than the bottom. And then to look at such things as access, vis- a-vis geographic location, gender, age, and associated costs. You gave examples of programs designed for women when girls are actually participants and for male farmers when females are the farmers. Cost and financing its important because so little money is available for nonformal education and we need to talk about where the money comes and find a creative way to finance the nonformal education. You argued for, as I understand it, a better way of analyzing the financing of NFE activities. Especially, we need to pay more attention to how nonformal education is financed. You saw as useful terms internal and external efficiency. They get at some of the important issues but mislead because in the other subsectors of education they are used in a much tighter economic sense than we use them in nonformal education. Basically because we aren’t as number oriented, there aren’t as many data for us. And then the issue of what is a maximum gain and what is a maximum effectiveness in relation to what we’ve paid for. Which gets to the issues of achievement. Then I raised the question about the policy environment. I indicated that that was something raised earlier in the interviews and whether you thought that was a useful element to consider. You thought it would be useful, though difficult, particularly in situations where there’s a lot of centralized funding for NFE activities. But in areas where there is either local, or smaller source, or more diverse sources of financing, the overall policy environment might be more or less important. C-7 Respondent: Right. It would be difficult, but important to look at. I don’t know if this is what you said. It’s important to look at when funding is from big sources outside. That would be very important to see how decisions were being made and things were being integrated. But it wouldn’t be as important, I don’t think, when things are more locally financed. Interviewer: OK. Are there any others you want to add to it? Respondent: Not that I can think of right now. Interviewer: OK. One of the questions that I had, related to asking people to or- ganize the elements. Now, essentially you already gave them to me in a structure. So what I’d like to do is ask you about the structure you used. I think I’d like to start with is a question, is it important to have a structure when you go into some- thing like that or would it have been equally as useful to go in and design your own structure given the context? Respondent: If I had a lot of time I probably would have preferred to design my own structure. But the structure was very helpful to me and I know there are reasons for having a similar structure across projects and that’s so that they can be more easily compared. But I think the structure was general enough that I wasn’t forced, except maybe a little in the issues section. I didn’t feel forced to fit things into categories. I think it was general enough that you would think any assessment of nonformal education would want to look at those categories of information that were given in the outline that we had. So, I was grateful for the structure and I think in 6-8 weeks, that’s a really big help because it kind of gives you direction right from the beginning. But I’m not sure that on my own I would have come up with that structure. And I probably wouldn’t have thought of everything that struc- ture included. It’s probably better to have structure, at least for someone like me, but I’m not so sure that that would be true for every person. Interviewer: You have identified some that you didn’t use in your own work. One of those was participation—one of the elements. Are there any other elements that you can think of that— Respondent: That should be included that weren’t included. Well, participation. More discussion maybe of benefits. Maybe that’s a little different than just evaluation. Interviewer: Yes. OK. One should study benefits as distinguished from evaluation studies. Respondent: But really that should come out in the evaluation. But it didn’t always come out in an evaluation that was done. I think if I had be thinking in terms of benefits I would have been able to ask people, "Well, how do you think participants actually benefited?” which isn’t a question that I thought of asking. Also, I think I would have asked about the curriculum. How the curriculum was developed? By whom, and what process was used to make the decisions about content? Though, I think that came out in questioning people. But having that as a particular element I think would have been good. I said this before too, that the development philosophy of the programs, not only the government’s development approach. Given the program, what was the assumed approach to development underlying it. That would have helped me analyze it a little bit better. C-8 Interviewer: Some of the things that you have identified as key elements are rela- tively easy to get information on like gender. I say relatively easy. Some are a lot more difficult to get information on, and it’s obviously going to require some further analysis or research to get information. What are some of the areas you think are very difficult to get a handle on at this point. Respondent: Because of lack of information? Interviewer: Or lack of measurement criteria. Respondent: Difficult, but still important? Interviewer: Yes. In other words, you’ve mentioned a lot that are important, and I would suggest that from your list, probably participation is a very hard one to get a sense of. How do you, in an assessment, get not only at the concept of, but also the fact that there might be relatively different levels and types of, participation. I would suggest that participation is probably one that may need some work on. My question is are there others? Respondent: Well, cost. That is an obvious one. Just because you don’t know if there’s much information most of the time. And related to participation, participa- tion at the various levels and who is participating, too—by gender, age, and for how long. And how many people actually begin and then drop out, and reasons why people drop out. I don’t know if this is what you’re getting after, but if there were information on who started and then who didn’t continue and why they didn’t continue, that would give a lot of information. Another thing is the content area; I mean, someone can tell you what the content was of either the training of the facilitators or the nonformal education itself, and they could tell you what methods were used, but that may not really tell you actually what content was communicated and by what methods. Like from so many government people, I heard about the content of these health programs and also the new literacy campaign—What they were intending to do, which isn’t bad content, but then the methods were going to be participatory and discussion oriented and all that. So that sounds really good; but then when I talked to XXX near the end—this American person who actually tried to train the people to be participant type facilitators and actually observed what they did—he said they ended up lecturing. So by having people report to you what the content is and what methods were used may not actually be the truth about the content or methods used. Interviewer: So then validity and verification become issues. Respondent: Yes. And I think that’s one of the problems with the sector assessment, at least the way I did it, is that you’re talking to people who are responsible for programs and they often don’t see themselves the way things are in reality. You have to try to talk to people who do see things the way they are, but you yourself, I myself, never got to actually see the way things were. That’s because nothing was actually happening anyway. And there isn’t that much time. Measuring the benefits, I guess, is difficult, or what you call the achievements. Interviewer: Those may be two different things. Respondent: Yes. What the project actually achieved. Is that what you mean by achievements? C-9 Interviewer: Well, because somebody could achieve a specific level of literacy but it not be beneficial to them. Respondent: Right. Interviewer: Or they might not have achieved a level of literacy, but going through the process in and of itself may have been beneficial. So that’s why I separate out the benefits and achievement. Achievement is more measurable. Respondent: Or visible. Interviewer: Yes, and more concrete. Respondent: Maybe more saleable. Interviewer: Whereas benefits are more saleable to the learners. Respondent: Yes. Interviewer: I wonder whether you’ve seen NFE experts using different elements than those we’ve been talking about, or ones that strike you as being unusual or surprising or— Respondent: In assessing nonformal education? Interviewer: Yes. Respondent: Seems like I read an article once, by someone who said that a true measure of consciousness-raising-type NFE would be increases in people’s political participation. That if the nonformal education were really effective people would increase their political activity or their participation in public affairs. So that would be unusual, as we’re looking more at economic gain. Interviewer: What was you’re role in the assessment in relation to other profes- sionals and host country institutions? Respondent: What was my role in relation to other professionals? Interviewer: Yes. How did you see your role? Would it have been enhanced by a counterpart? Was it enhanced by being part of a team or not? Because NFE assess- ments have been carried out by people both in teams and people working individually. Respondent: Oh, If I had been part of an NFE assessment team. Interviewer: Or if you had to do an NFE assessment by' yourself—without having the rest of the crew around. I’m sort of curious as to how you saw yourself in rela- tion to others, and what were the strengths and shortcomings of the professional en- vironment in which you found yourself? Respondent: I don’t know exactly how I saw myself in relation to others. I felt that being part of a team was a real help, that it gave me more legitimacy to approach people explaining that I was part of a team and that we were doing this overall assessment of all the different subsectors, that I was responsible for this one. And I C-lO felt that the camaraderie and support of team members and sharing of daily ex- periences was very helpful. And also, there was a lot of sharing of information, both people to see or to talk to, I mean, we each seemed to find out things for each other as we were talking to different people. We got leads that we could share with each other. So, that was very valuable and I think being alone would have been much more difficult in those terms. I felt that because I was an outsider with no real background in Haiti, that people saw me as a, well as a representative of USAID and possible money. So there was a certain kind of deference, and also a sense that somehow they were being evaluated. Even though I made a big point of explaining what the assessment was, here was this outside person connected with USAID wanting to know what we’re doing, what our problems are, and how we succeeded or failed. I think it would have been that much better to have had a Haitian counterpart team and a Haitian NFE counterpart. I think it would have given me a lot more depth. I think I would have learned a lot faster and I think in the eyes of the pe0ple I had to interview, maybe we together would have interviewed, I think we would have been seen less as a outside evaluator and more as someone actually connected to Haiti. I think, then too, recommendations would have seemed a little more grounded, that they would have seemed more legitimate if we had a Haitian counterpart. We did work with AID closely and one of the things that I felt there then was that would have helped me to have a small advisory com- mittee of professionals, both Haitian and maybe other Americans. Ira, would have been a good member of that advisory committee. That committee would meet with me periodically and give me advice. I think I would have liked to have that kind of formally set up contact with other professionals working in and concerned about nonformal education. Interviewer: With the professionals working in-country! Respondent: Yes. In country. I don’t think it would have taken a lot of their time either. And it would have given people who actually are there working more of a feeling that the assessment was something that they contributed to and belonged to them and maybe more chance that the recommendations would be supported, or reflect what they felt also. Interviewer: If you had to do it all over, how would you change it? Respondent: Oh, I guess I’d reiterate. Interviewer: You’ve given me some ideas. What I mean is, are there any other things that we haven’t talked about that you would do differently? Respondent: Yes. I know I thought of things like that earlier. I would do more background reading before I went. But I just didn’t have the preparation time. More advanced warning, I guess, but I that couldn’t be helped. I ended up reading there, things that I had collected here. I think it would have been much more efficient if I had been able to read them here. I think I would start more quickly. There was no choice there, I mean, we had to kind of wait until we met the AID people, and then there were some holidays and all that, but I think that next time I would be more agressive about the need to begin interviewing people right away. I think I could have done a lot more if I had started sooner and got into interviewing people. Because once I got over the initial fear of actually going out and presenting myself, then things went kind of fast but I didn’t actually get to the point of going out to meet people until maybe a week or two after we were there. So I think I could have gotten a lot more information. I would meet with the AID people concerned, or C-ll whoever was really concerned with the outcome of the assessment the most. By myself. Which I didn’t do then. I could have asked to do that, but I just didn’t, to really clarify with them what it is they wanted. Which programs? What government? Nongovernment? And get it very clear right from the beginning what my— Interviewer: Purpose? Respondent: Yes, and what do you call it? Scope of Work was. And I think because I didn’t do that, that led to some misunderstanding. What I did versus what they seemed to want. Interviewer: In that situation it was their fault. Respondent: Well, yes, but if I had met with them and got it from their mouths, it would have been harder for them to say, ”Well that’s not what we wanted.“ Interviewer: Yes. OK. Respondent: Probably there are other things that I cannot think of now. Interviewer: A couple of things that have sort of emerged as I have been talking with people and one of them is what should be the role of LDC planners in the development or the process of the assessment? Respondent: Planners. Interviewer: I’m talking about high-level ministry people as opposed to the people that we talk with and get information from about the description of the program and the outcome and the achievements of the program. Basically, we deal mostly with functionaries. What about the planning level people? For example, the group that we met with twice, with the minister of education and his cabinet, and sort of the next layer down. What should their role be? Respondent: I think— like I mentioned some kind of NFE advisory committee. Interviewer: You would see them fitting in there? Respondent: Yes. Maybe there would have to be two. I don’t know. Maybe there would have to one that I would work with more closely. And maybe a little more openly and honestly. I’m not sure how. If I were working in Haiti, for example, you couldn’t be totally open with a government upper-level advisory committee be- cause so much of what I learned was anti-what they were doing. But, yes, I would see somebody like XXX, the main head of the literacy and community development division of the ministry of education. He and several other people, like the head of IPN literacy research group, GREAL. And maybe XXX, who headed different sec- tions of the community development section, and someone from the literacy section, and maybe materials development section. Perhaps the heads of those sections and XXX forming sort of an advisory committee that I would keep up to date on what I was doing and get their advice and direction too. It might be hard to manage all that, you know. Different kinds of input. But yes, I think that would be really im- portant for the same reasons I gave for an advisory committee on the AID side of it. It would be to help, have them feel that it was their assessment, which is what I thought would be one of the goals of this—that the country itself would feel like it C-12 was their assessment, and not an evaluation. It might have been good in Haiti to have people from USAID plus those people together on the same committee because it might have helped with the dialogue and their relationships, because I think they didn’t have much. Interviewer: If people are going to continue to do NFE assessments, who do you think should do them? Respondent: Who? Interviewer: You’ve talked about the insider and outsider and I guess that’s getting a that question. Respondent: I don’t know. I see the reason for having an outsider and reasons for having an insider. But having someone from the country you’re bound to have some kind of bias. I mean if you have someone appointed by the government you’re going to def inately have a bias and probably not going to get an honest assessment of what NFE is doing. If you have somebody like XXX, who probably could have done it, or even XXX, I mean, somebody from USAID, well especially like XXX, there’s also going to be a bias. A negative bias—everything the government does is bad, and they might not even want to look at what the government is doing. Be- cause there’s is a lot of goodwill on the part of government too. So, I don’t know how you would find an objective insider to do it. But that would be the best pos- sible situation. Maybe somebody from one of the NGO’s who’s kind of a disinter- ested third party not analyzing his or her own organization, but assessing nonformal education in general. Maybe that would be a solution. But whether then it could be' a Haitian, a local country native, or a nonnative is another question. I don’t know. But I would say a disinterested insider would be much better than someone who knows little about the country and hasn’t worked here before coming and trying to do it. Interviewer: What are the advantages of an outsider? Respondent: Well, the advantages are the objectivity and, like for me, just the lack of any preconceived ideas about what needed to be done. Yes. I was fairly objec- tive in that I really didn’t have much knowledge. I guess you can be objective when you don’t know much. And the outsider is not answerable to any organization like the government or any other organization and they have no vested interest in any of the projects. You know, even an AID person would have a vested interest in certain AID NFE projects. 80 that’s the advantage, I think. And I think an outsider could promote cooperation if there were some kind of committee for each of the subsec- tors and that committee consisted of government, nongovernment, native people, non-native people, and expatriots. And, if the person doing it were skilled, that per- son could learn a lot from the discussion by promoting a dialogue between the people on this committee. I think this person could learn a lot from that kind of a discussion and could also help promote cooperation just by getting people to talk to each other. And I felt that was really needed in Haiti, there was such mistrust of the government. Who am I to say that there shouldn’t have been, I mean, if I had been working there maybe I would have thought someone coming in trying to promote this kind of dialogue was naive, but that could also be a benefit of an outsider. Interviewer: I don’t have any more questions. Maybe you do. C-l3 Respondent: I guess would just wonder what kind of things you have come up with. Interviewer: Well, what is interesting is that as I’ve gone through this with dif- f erent people, I’ve come up with a group of elements and each has components. I’m not sure how they are all going to fall out. They sort of end up being the purpose of the assessment, the preparation for the assessment, the policy environment, and issues of bureaucracy and centralization, and those kinds of things come up. The assessment process, that is, who does it, and well you talked a lot about the committees, really fits into this whole question of how you go about doing the assessment. The roles of insiders and outsiders and the team. Then the NFE programs themselves which, I think, as where most prior assessments have focused is on the components of the NFE pregrams. And its interesting that that came down really far on my list. The methodology of the assessment and then there is one which is sort of emerging which is the relationship of nonformal to formal education. Linkages both within the NFE sector, subsector and formal education. That’s sort of the way things are coming out, then within each, there’s just a whole melange of questions and issues. But one of the things that it is sort of telling me is that the work that I did in Botswana was less than ideal. There were a lot of things that were missing in terms of addressing the preparation, addressing the purpose of the assessment, getting some participation at the beginning in terms of how that process should be conducted and some of the things that have come out in the methodology have been very good. It funny, listening to you talk, you would really like to talk with XXX and XXX. You would really love to talk to them. It’s really interesting. XXX did work in Somalia and XXX did work in Liberia, but its so in- teresting because now I said I have this sort of small group of people— Respondent: Who said similar things? Interviewer: Yes. Very similar sets of concerns and I think— END OF INTERVIEW "7'1111111111111111111115