7H ES a :3 13-5.? This is to certify that the __ i-r m (.2)? ‘ .( 1"!— 5“ vfi— HIS n. m i 1‘ -/n‘_.L% "3:15.”? ___uv__“_ ’- W'tfr‘rw - -._ 'J A'- _4‘.‘ - .‘_ WWI 1W 3 I J. .... ”mte U233 _f“ dissertation entitled TESTING THE APPLICABILITY OF DATA CLASSIFICATION AND SYNTHESIS TECHNIQUES FOR USE IN PLANNING AND EVALUATING HEALTH PROMOTION PROGRAMS presented by Kathleen L. Akpom has been accepted towards fulfillment of the requirements for Ph.D. Date Nov . 11, 1981 MS U is an Affirmative Action/Equal Opportunity Institution ____.——————_ ____ . ___. degree in J J ’\ ‘JIIHITIIWIHIIHII L 23 01058 0573 Continuing Education Major professor ~ 0-12771 OVERDUE FINES: 25¢ per day per 1m RETURNIMS LIBRARY MATERIALS: Place in book return to remove charge from circulation records cv‘v 7..-? m-" _w ' l I::0 TESTING THE APPLICABILITY OF DATA CLASSIFICATION AND SYNTHESIS TECHNIQUES FOR USE IN PLANNING AND EVALUATING HEALTH PROMOTION PROGRAMS By Kathleen Lucille Akpom A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY College of Education 1981 ABSTRACT TESTING THE APPLICABILITY OF DATA CLASSIFICATION AND SYNTHESIS TECHNIQUES FOR USE IN PLANNING AND EVALUATING HEALTH PROMOTION PROGRAMS by Kathleen Lucille Akpom The purpose of this study was to test the applicability of data classification and synthesis techniques for use in planning and evalu- ating Health Promotion programs. Six specific research questions were developed related to the purpose of the study. A subset of one identified Health Promotion priority areas was selected as the focus for the study: community nutrition education programs for non-patient, non-institutionalized adults in the U.S. A sample of 46 studies was selected from journals published in the U.S. from 1960 through 1980. A review of existing classification instruments revealed that none met the needs of this study so an instrument was de- veloped. Studies were analyzed using the instrument which classified them according to program and evaluation content categories; the studies were also scored according to quality criteria. The studies were then screened for appropriate inclusion in selected data synthesis procedures. The assumption and data requirements for each of four data synthe- sis techniques were specified and the sample of studies was examined in light of these. It was found that no two studies in the sample satisfied the assumptions and data requirements of the synthesis techniques. In order to synthesize outcomes several similar studies must exist and at this time studies of evaluation of outcomes of community nutrition Kathleen Lucille Akpom education programs vary greatly by type of program, target population and outcome variable. Therefore, data synthesis techniques were not appropriate for the studies which represent the focus of this project. The usefulness of the Classification Instrument developed for this project combined with the identified limitations of the selected data synthesis techniques led this researcher to further develop the classi- fication approach into a Health Promotion Classification Manual. The Manual is an attempt to develop a methodology for classifying, according to multiple categories, the content, methods and quality and results of Health Promotion so that this data can be systematically retrieved and used in developing health promotion programs based upon existing evalua- tion research findings. The Manual can also help to further the under- standing of Health Promotion as a multi-strategy, multi-focus and inter- disciplinary entity. Two general guidelines that have emerged from this dissertation can be summarized as follows: first, relevant past program experience should be systematically reviewed by Health Promotion professionals in order to more efficiently plan both their programs and the evaluation of those programs; and secondly, whatever designs are identified in the literature will have to be tailor made to some extent to accomodate the resources of each particular program. Data classification and appropriately conducted data synthesis of evaluation research findings are promising initatives that have poten- tial for enhancing the success of Health Promotion Programs in increas- ing positive health behavior among Americans. For Amechi and Akosa ii ACKNOWLEDGMENTS The author wishes to express special and sincere graditude to Dr. Larry Lezotte, dissertation director, for his guidance, support and encouragement throughout the many different phases of this work. Many thanks are extended to Dr. Larry Borosage, committee chairman, for his encouragement and enthusiasm, as well as to committee members Dr. Howard Hickey and Dr. Don Tavano. A very special thanks goes to my family: my parents for their encouragement; and to my husband, Amechi, for his unfaltering moral support, valuable critic and hours of babysitting; and to our sons, Amechi and Akosa, who should each receive honorary doctorates for their patiencei Finally, the author wishes to express appreciation to do Cornell for her expert editing and typing. Chapter I: II: III: IV: TABLE OF CONTENTS Page INTRODUCTION TO THE STUDY Purpose of the Study .................... 1 Background ......................... 2 Need for the Study ..................... 6 Scope and Focus ...................... 11 Research Questions ..................... 14 Objectives ......................... 15 Methodology ........................ 16 Overview of the Study ................... 17 Limitations of the Study .................. 17 Significance of the Study ................. 18 Summary .......................... 20 REVIEW OF RELATED LITERATURE Data Synthesis Techniques ................. 21 Development of the Classification Instrument ........ 27 Summary .......................... 38 PROCEDURES Development of the Classification Instrument ........ 39 Pilot Study ........................ 4o Scope of the Data ..................... 41 Source of the Data ..................... 43 The Data Set ........................ 44 Procedures for Data Analysis ................ 45 Summary .......................... 45 PRESENTATION OF DATA Data Relating to the Content of the Studies ........ 46 Data Relating to the Quality of the Studies ........ 59 Data Relating to Synthesis of Outcomes ........... 63 Summary .......................... 67 SUMMARY, CONCLUSIONS, RECOMMENDED GUIDELINES, SUGGESTIONS FOR FURTHER STUDY AND REFLECTIONS Summary .......................... 69 Conclusions ........................ 70 Recommended Guidelines ................... 79 Suggestions for Future Study ................ 86 Reflections ........................ 87 iv Appendix A. Specific Nutrition Objectives for 1990 or earlier in the U.S ........................... 91 8. Classification Instrument .................. 93 C. Bibliography of Potential Data Set and Disposition Relative to Inclusion in Study and Reasonsfor Non Inclusion .......................... 94 D. Description of the Content of the Selected Studies ..... 104 E. Description of the Quality of the Selected Studies ..... 114 F. Classification Manual for Studies of Health Promotion Programs and Evaluations .................. 120 6. Health Promotion Experts' Opinion Questionnaire and Accompanying Letter ..................... 148 Bibliography ............................ 153 General References ....................... 158 4.10 4.11 4.12 LIST OF TABLES A Health Promotion Model ............... Year of Publication by Journal in which Studies were Published ............... Frequency Distribution of Categories of Program Purpose ....................... Cross Tabulation of Profession of Instructor by Target Group ................... Categories of Variables Assess by Study Number. . . . Specification and Frequency Distribution of Structural Variables by Study Number ......... Specification and Frequency Distribution of Process Variables by Study Number .......... Specification and Frequency Distribution of Outcome Variables by Study Number .......... Specification and Frequency Distribution of Impact Variables by Study Number .............. Techniques for assessing Behavior and Results . . . . Techniques for Assessing Knowledge and Results. . . . Techniques for Assessing Attitudes Including Satisfaction and Results .......... Frequency Distribution of Quality of Study Scores . . . Quality of Studies Criterion Categories by Ranges in Scores and Mean Scores ......... Mean, Mode, Median and Range Scores by Type of Evaluation Design ................. Mean, Mode, Median and Range of Scores by Evaluation Design Groups ............... vi Page 48 49 51 52 53 53 55 56 57 58 59 61 62 62 63 LIST OF FIGURES Figure Page 1.1 Health Promotion Priority Areas ............ 7 1.2 Education and Information Measures for Promoting Health/Preventing disease .......... 13 vii CHAPTER 1 Introduction to the Study There is nothing more different to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the intro- duction of a new order of things. Machiavelli Purpose of the Study The purpose of this study is to develop and test a method for sys- tematic and analytic review, and classification of existing public health- prevention, and related literature which, when used with appropriate data synthesis techniques, can be useful to health planners in building upon the "best" of what is known from past research to work towards achiev- ing the national priorities of Health Promotion. Although the proposed study is primarily concerned with analysis of the quality of the evaluations in this literature and how these evalu- ations can contribute to new programs and policies for Health Promotion, it must also concern itself with the content and quality of program stra- tegies in order to assess the appropriateness of evaluation designs, methods, results and conclusions. As prevention programs proliferate, policy makers, health planners, health insurance carriers, practition- ers and consumers are asking: (I) are existing programs effective? and specifically, (2) what are the most effective strategies that have been and are being conducted, and what results do they offer for further de- veTOpment of Health Promotion strategies and policies? 1 Background The most readily accessible and broadly representative source of information concerning previous program experiences are printed journals. Given today's technology of both printed and computerized Indexes and abstracts, journal articles concerning a particular topic can usually be identified easily. The limitations of this present technology are several: 1) the appropriateness of the article to the problem of inter- est must be determined by review of the article, since titles may be mis- leading, and because Index and Abstract categories are usually broader than a problem or area of interest for which the literature search was conducted; 2) the Indexes and abstracts usually do not necessarily in- clude all relevant references on a subject or problem; 3) access to the original articles may not be possible because of limited library facili- ties available to a practitioner, and 4) all articles are reported equally, with no evaluation of or reference to the quality of each study. Wellness goals for Health Promotion for the year 1990 or before were identified in the 1979 Surgeon General's Report entitled Healthy People, It is the assumption of this project thatin orderfor the field of Health Promotion to achieve these goals, successful methods must be rapidly implemented and be focused at multiple targets using a wide var- iety of activities. Turner (1975) has stated that to advance a given state of the art, a two pronged approach is necessary: (1) identification of the schemata or conceptual characteris- tics of an entity, and (2) identification of instances of its success and non success, that is, an understanding of what has been done and its conse- quences. Implicit in understanding the success of a program is the need to understand the quality of the measurement of its suc- (2655. Defining Health Promotion Duane Block, in a 1980 address to the American College of Preven- tive Medicine, entitled Health Promotion: Prospects for the Future, described health promotion as "an idea whose time has come." He goes on to state: Hardly a day passes without seeing an article re- lated to health education, health promotion, di- sease prevention, or life style change in the pub- lic press or in numerous lay periodicals. Pro- fessional journals regularly carry articles... which fit under the rubric of health promotion. The federal government has issued several recent publications.... Many business organizations have initiated various types of programs that are viewed as health promotion in an effort to reduce health care costs, improve health of their employees, or other wise have a favorable effect on the enterprise. Numerous professional conferences are being held on the subject of health promotion and disease prevention.... These are but a few of the examples that...illus- trate the wide-ranging interest in health promo- tion at this time and the importance which the health professions and the general public ascribe to the activities and programs that are grouped under the rubric of health promotion. Despite the frequent use of the term health promotion there has been ambigui The fi ty and speculation voiced concerning its meaning. rst recognized definition of health promotion was provided during the 1976 U.S. National Conference on Preventive Medicine: All health promotion involves...in balanced propor- tion, all the means or strategies of promotion: research, education for the health professions, public health, environmental protection, occupa- tional health, consumer health education, health care (diagnosis and treatment of illness and dis- ability), and health economics (organization and financing). Unfortunately, that definition is no more concrete than the high- ly criticized World Health Organization's definition of health, which can be summarized as "total well being," and therefore, lacks a meaning- ful basis by which it can be operationalized. The American Hospital Association House of Delegates, in August of 1979, defined health promotion as: ...the process of fostering awareness, influenc- ing attitudes, and identifying alternatives so that individuals can make informed choices and change their behavior in order to achieve an optimal level of physical and mental health and improve their physical and social environment.... The goal of health promotion is to encourage and aid each individual to take greater responsibil- ity for his or her own health, not to coerce any- one into adopting specific health practices. The objective is to set an atmosphere conducive to positive health, to provide information to support it, and to offer education about how to change negative practices into positive ones. Lauzon (1977) stated that despite the fact that most health pro- fessionals seem ”comfortable with the substance and implications of health promotion," there still is a lack of clarity about what health promotion means and can be. Lauzon goes on to state that, ”Unfortun- ately there are indications that the rhetoric surrounding health pro- motion is nothing more than a semantic game, a form of wizardry in which health education activities of yesterday are transformed into the health promotion programs of tomorrow." But Lauzon (1977), Divore and Krueter (1980), and Block (1980) all recognized that health education is but one of the methods that will contribute to health promotion. The U.S. Department of Health and Human Service (1980) reinforce this motion in their definition that: "Health Promotion refers to any combination of health education and related organizational, poli- tical and economic interventions designed to facili- tate behavioral and environmental adaptations that will improve or protect health." (p. 7) Lauzon (1977) has gone beyond defining Health Promotion. He has developed an "Epidemiological Approach to Health Promotion," which iden- tifies selected health promotion activities and their targets according to host, agent and environment spheres. According to the model, the host variables (defined as low risk individuals, average risk individuals and excess risk individuals) can be influenced through the Health Promotion activities of instruction, education, persuasion, behavior modification, proselytizing, screening and advertising. The agen variables (defined as alcohol, automation, automobiles, food, health services, illicit drugs, and tobacco) can be impacted upon by the Health Promotion strategies of marketing, product modification, engineering, substitutes, regulation, and legislation. The final component, environment variables (defined as physical, social- cultural, economic and mass media) can be impacted upon by physical in- fluence, social-cultural influence, economic influence, and media influ- ence. Lauzon (1977) explains that “the epidemiological model enables health professionals to acquire an econogical or systems perspective for various problems. Such a perspective facilitates the identification of potential courses of action which may be useful in moderating or elimi- nating health risks. The dynamic relationships among the three compo- nents must be recognized in order to conduct risk reduction programs in an effective and efficient manner and to evaluate them appropriately." (p. ) Lauzon's (1977) model is the most definitive to date for moving health promotion from an ideal to tangible program components. It helps to identify goals, objectives, and activities, which aid in identifying specific previous efforts from various disciplines which can contribute to a comprehensive health promotion strategy. In discussing the need for his model, Lauzon quotes T. S. Kuhn, who stated that: In the absence of a paradigm or some candidate for a paradigm all the facts that could possibly per- tain to the development of a given science are likely to seem equally relevant. This project is an attempt to test out data classification and synthesis techniques which will be useful in sorting out previous re- levant research consistent with the Lauzon paradigm. Need for the Study To illustrate the need for this project, the following scenario is offered. Assume that you are the person responsible for develOping a Health Promotion program for adults. You want to develop the best program possible. You are committed to Health Promotion, which has been well described by Divore and Krueter (1980) as: The process of advocating health in order to enhance the probability that personal (individual, family and community), private (professional and business), and public (federal, state and local government) support of positive health practice will become a societal norm. The process of advocating health may be conducted by a variety of modalities, including but not limited, health education. You understand the extend of need for program deliver if the well- ness-goals (as identified by the 1979 Surgeon General are to be achieved by 1990. (See Figure 1.1 for identification of the priority area.) Preventive Health Services High blood pressure control Pregnancy and infant health Sexually transmitted diseases Toxic agent control Occupational safety and health Accident prevention and injury control Fluoridation and dental health Surveillance and control of infectious diseases Smoking and health Misuse of alcohol and drugs Physical fitness and exercise l. 2. Family planning 3. 4. Immunization 5. Health Protection 6. 7. 8. 9. l0. Health Promotion ll. l2. l3. Nutrition l4. l5. Control of stress and violent behavior Figure l.l Fifteen priority areas for "promoting health/preventing Note: diseases" Promoting Health/Preventing Disease: Objectives for the Nation, U.S. Department of Health and Human Services, Public Health Service. Fall l980. 74. You agree with Turner for the need to understand the conceptual characteristics of a field; and that ques- tions concerning appropriate research foundations for Health Promotion programs are contingent upon a de- termination of the parameters of the field of Health Promotion. You recognize as Lauzon (1977) has stated that "Ef- fective health promotion programs for the future must be founded upon sound principles which recognize the multiplicative dynamics of health behavior." and you accept the "Epidemiologic Approach to Health Promo- tion" proposed by Lauzon as an appropriate schemata for this field. (See Table 1.1 for a diagram of the Model.) You recognize that in addition to the Lauzon model identification of target groups and activities, you can further contribute to development of successful Health Promotion programs for your target gruop (adults) by incorporating into your planning the information concerning adult learning theories and research findings. Because almost all host (or adult-targeted) Health Promotion strategies can be considered as non formal education, you also want to incorporate into your planning, experience from the fields of non-formal education. Because the literature concerning Health Promotion and realted fields is so inclusive, you are faced with the enormous task of pulling together the most successful strategies from many thematic and program fields. You are not a researcher, but rather a practitioner, and you do not have vast technical research skills. The person in this senario will in all liklihood find that she/he is severely handicapped in meeting her/his responsibilities for develop- ing Health Promotion programs. In fact, without a systematic approach for analysis, classification and synthesis, she/he will probably find re- ported results from the literature to be equivocal. Consequently, the usefulness of the review in program planning efforts will be limited. Just as a model or schema is required to understand the components and Table l.l A Health Promotion Model Epidemiological Selected Health Concept Targets Promotion Activities Host Low-risk individuals Instruction Average-risk individuals Education Excess-risk individuals Persuasion Behavior Modification Proselytizing Screening Advertising Agent Alcohol Marketing Automation Product Modification Automobile Engineering Food Substitutes Health Services Regulation Illicit drugs Legislation Licit drugs Tobacco Environment Physical Physical influence Social-cultural Social-cultural influence Economic Economic influence Mass media Media influence Note: From "An Epidemiological Approach to Health Promotion" by R. Lauzon, Canadian Journal of Public Health, August 1977, 68, and reprinted by Bureau of Health Education Focal Points, U.S. PHEW, Public Health Service, Center for Disease Control, Atlanta, Georgia, May l979. 10 conceptual characteristics of Health Promotion, so too there is a need for a method by which previously conducted research about each component can be extracted, analyzed, classified and synthesized to contribute "to a data bank" to further clarification of the Health Promotion knowledge .basg. Despite the temptation, there is not a need to develop all new methods and techniques of Health Promotion, but rather there is a need to creatively combine in a new. coordinated way the most effec- tive strategies that have been used for eliciting positive health behavior. New methods will then only be developed as needed to supple- ment existing ones to create a coordinatedsocietal strategy for pro- moting norms of positive health behavior. Identifying those successful strategies may seem to be a straight- forward task, but considering the volume of studies for each of the host, agent, and environment variables and their broadtyidentified health promotion target activities and research from related fields, combined with the variable quality of studies, the task is great. A method which could assist in analysis and classfication of ex- isting relevant literature, according to content, and quality may prove useful in advancing the knowledge base of health promotion programs. The method would not only provide useful information for planners, it could identify comparable studies upon which data synthesis techniques could be performed. Supporters of data synthesis claim that the techni- ques can yield valuable data for program planning. Pillemer and Light (1980) have described data synthesis as "the science of discovering what we know". They define it as "formal procedures for combining the results from several empirical studiesh 11 They assert that all too often literature reviews on a certain question of inquiry are unsystematic. Using such unsystematic methods, two researchers reviewing literature to answer the same question-~is a specific program effective?--may well come to different conclusions. In order to enable unbiased and scientific conclusions to be drawn from ex- isting literature about program effectiveness on a specific problem or subject, Pillemer and Light suggested the need for data synthesis. hfith the increased emphasis on Health Promotion, along with the budgetary constraints being placed on all health and human service programs, it seems that the time has come for a systematic evaluation, description and synthesis of "what we know" about the efectiveness of prevention programs that are aimed at improving health behavior, so that health promotion programs can have a firm basis on the best of relevant program and research experiences. Scope and Focus Because a review of all prevention literature is beyond the scope of this or any single study, one sub area of the Health Promotion pri- ority areas identified earlier was selected in order to test the classi- fication and synthesis methods for this study. The specific area selected was community nutrition programs for non-patient, non-institutionalized adults in the United States. The categories of potential prevention/promotion measures for improved nutritional status by 1990, as identified by the U.S. Public Health Service in Fall of 1980 are: 3 4. 5. 12 Educational and information measures Service measures Technologic measures Legislative and regulatory measures Economic measures (The specific nutrition objectives for 1990 or earlier in the U.S. are presented in Appendix A.) The category of interventions,which are the focus of this project, are educational, and information measures. Figure 1.2 displays a listing of specific education and information interventions for Health Promotion. (See Figure 1.2.) The time frame for programs under review was 1960—1980, primarily because, as has been identified by Flaherty and Morrell, few program evaluations were reported prior to 1960. 13 Increasing awareness of ideal weight ranges and safe weight reduction and weight control strategies based on energy balance concepts. Increasing awareness of the science base regarding relationships between diet and heart disease, blood pressure, certain cancers, diabetes, dental care, and other conditions. Providing information and behavior skills to select and prepare more healthful diets. Development more effective means of communicating nutrition information to people in different age and ethnic groups. Providing nutrition information and education about healthy food choices in the home (via the media), in schools, at the worksite by and to health care providers, at the point of purchase, as a part of government food service programs (such as Project Head Start, school lunch, and HIC Programs) and by appropriate advertising. Providing appropriate information on the advantages and techniques of breast feeding and when appropriate, alternatives, particularly for low income women. Figurel.2 Education and information measures for promoting health/ preventing disease Note: Promoting HealthlEreventing Disease: Objectives for the Nation, U.S. Department of Health and Human Services, Public Health Service, Fall l980, 74. 14 Research Questions The specific research questions addressed in this study are: 1. What is the state of the art of program deliviery and evaluation of community nutrition education for non- institutionalized, non-patient adults, in the United States? Are any of the existing literature review or meta-evaluation (secondary evaluation) methods or instruments appropriate for classification and assessment of public health studies for the purpose of planning programs and evaluation methodologies, and for screening study data for appropriate inclusion in data syn- thesis. If not, can such an instrument be developed in a format that can be easily understood by, and useful to, those who plan, ad- minister and evaluate Health Promotion programs? Are the selected data synthesis techniques appropriate for use with a sample of Health Promotion programs? Does the type of evaluation design affect the usefulness of the evaluation findings--that is, do any systematic evalua- tion efforts, despite level of experimentation (i.e., pre ex- perimental, quasi-experimental or true experimental) generally yield useful results? Is there a minimum set of guidelines that can be suggested (based upon this dissertation research) for use in planning and evaluating Health Promotion programs? 15 Objectives: 1. Identification of as complete as possible the set of commu- nity nutrition education studies from 1960-1980 for non-pa- tient adult p0pulations in the United States, as reported in U.S. journals. Development of a comprehensive classification instrument for determining the content, results, and quality of program strategies and evaluation designs and methods of each study, and to identify information needed in order to determine which data synthesis techniques is appropriate for each study. Systematic review and classification of the set of studies using the classification instrument. Testing the usefulness of the selected data synthesis techniques by synthesis of a subset of studies of true experimental, quasiexperimental and preexperimental designs using each of the selected synthesis techniques. Analysis of comparability of information derived from synthesis of each of the three sets of studies (true, quasi, and preexperimental designed studies). Analysis of the usefulness of the classification system and revision, as needed. Based upon results of analysis of the articles, determination of whether there are guidelines by which Health Promotion programs can be planned and evaluated. 16 Methodology The research involved secondary analysis of published literature in U.S. journals concerning evaluation of community nutrition educa- tion for adult non-patient populations in the U.S. Through a process of reviewing literature on evaluation research, program evaluation generally and program evaluation literature in prevention, medical care, education, health education, and nutrition education, a classification instrument would be developed by which the data for this study would be analyzed. In this way, the variables, strategies, and methods addressed in the nutrition education studies would be compared with those identified by authoritative sources. The nutrition evaluation studies would be analyzed, classified, and evaluated using the Classification Instrument. A score reflecting the overall content and quality of each study would be assigned. The studies would then be classified into preexperimental, quasiexperi- mental, and true experimental design categories. The studies in each category would be sorted according to synthesis techniques appropriate for the type of variables and statistical analysis performed. It was anticipated that synthesis techniques would be conducted on appro- priate groups of studies, within the experimental design categories and synthesis results compared across experimental design categories to study the effects of quality or type of initial data on synthesis results. Based upon this pilot study using the classification instrument combined with the synthesis techniques, the procedures would be com- bined to form a resource manual that could be a useful tool for 17 those who plan, administer, and evaluate Health Promotion programs. A set of guidelines will also be developed for use in planning and evaluating Health Promotion programs. Overview of the Study In Chapter Two a review of literature which contributed to devel- opment of the Classification Instrument will be presented as well as descriptions of selected data synthesis techniques. In Chapter Three the data set, along with the instrument and the procedures for data analysis, both classification and synthesis, will be discussed. In Chapter Four, results of the classification and synthesis will be presented. Chapter Five will contain a summary of the study, conclusions drawn from interpretation of the data, recommended guidelines for plan- ning and evaluation of Health Promotion programs as suggested by the study, suggestions for further research concerning the use of classi- fication and synthesis techniques and their potential contributions to the development of Health Promotion programs. Reflection concerning policy needs and influences concerning realization of a national strate- gy for Health Promotion will be discussed. Limitations of the Study The study will be limited to an examination of selected published reports of community nutrition education programs for non-patient adults in the United States that appeared in U.S. journals from 1960-1980. 18 Although similarities may exist between community nutrition education and health education, and prevention generally, it is not possible to draw conclusions about the general practice of health education or prevention programs from this study. The topic of nutrition education program for adults was selected to "pilot test" the classification/screening method and synthesis techniques, and was not meant to appear representative of other thematic prevention programs. Significance of the Study The significance of the proposed study is that the methods tested will provide information concerning the appropriateness of data classification and synthesis techniques to those who plan, conduct, and evaluate Health Promotion programs, in helping them in building upon the best of what is known from a variety of fields and thematic programs. The study should contribute to building the knowledge base for Health Promotion. Those who will, and do plan, conduct, and evaluate Health Promo- tion programs out in the field often lack the skills,timecn:resources to search out from among the great numbers of studies (Which vary greatly in qual- ity and breadth), those studies which have conducted successful interven- tions which also have used appropriate, valid, and reliable evaluation methods. For those who lack analytical skills, the study will provide: l. tested methods for extrapolating data, methods, and strategies, and classifying and synthesizing these; 2. a recommended set of program planning and evaluation guidelines. 19 The needs of those practitioners who lack time to thoroughly review the breadth of literature relevant to Health Promotion can be addressed in a number of ways from this study: 1. Those concerned with community nutrition education will have identified for them: a. The content of existing studies of non-institutionalized. non-patient populations in the United States for the past 20 years; b. The quality of existing studies; c. An aggregation and synthesis of evaluation findings. 2. Those concerned with other thematic areas of Health Promotion may be able to influence policy makers to commission future studies to further develop techniques to classify, analyze, aggregate, and synthesize a broader body of previously conducted research relevant to Health Promotion, so that Health Promotion strategies can include, be limited to, and based upon, "the best of what we know" about prevention program strategies and their evaluations. The fifteen National Health Promotion priority areas identified in the l979 Surgeon General's report are those for which substantial improvements in health status can be achieved by changes in behavior of individuals at risk for each problem area. Specific health status and disease prevention goals for each area have been identified and can be achieved if specific unhealthy behaviors are changed. These are wellness goals; the strategies for achieving the wellness goals will require multiple strategies and interdisciplinary, coordinated planning 20 and programs. These will require some restructuring in the practice of preventive services to facilitate interdisciplinary coordination, and to enable consumers and patients to learn, participate, and take initiative in a new way in promoting, improving, and sustaining their wellness. 1 Health Promotion programs can most efficiently and effectively be planned and implemented if the most successful strategies from various disciplines can be brought together, built upon, and targeted on a problem area. In other words, by looking back at relevant, successful program experiences, and refining and applying these to specific Health Promotion goals, the methods developed and tested in this study should contribute to the knowledge base of Health Promotion theory and practice by providing tested methods for identifying and organizing successful program experiences. Summary In this chapter, the need for conducting this study, as well as the purpose, have been stated. The methodology to be used has been described, and the scope and overview of the study have been identi- fied. The limitations have been cited and significance of the project has been proposed. In Chapter Two, a review of literature which con- tributed to development of the Classification Instrument will be pre- sented as well as descriptions of selected data synthesis techniques. CHAPTER II REVIEW OF RELATED LITERATURE "...experimentation occurs in an intellectual en- vironment marked by diverse, competing and non- comparable theoretical traditions. Consequently there is only weak guidance for inquiry: it is difficult to decide when treatments have been either implemented or successful, for there are no solid theoretical or emperical anchors for measure of either." David K. Cohen In order to consider the applicability and adequacy of classifica- tion and synthesis techniques as tools for planning Health Promotion program strategies and evaluation methods, it is necessary to review ex- isting classification and synthesis techniques, and theoretical bases for comprehensive program evaluation. As suggested in Chapter I, no classi- fication system was found to be appropriate for this study, so this chapter will also include a review of literature which contributed to develop- ment of the Classification Instrument for this study. The Chapter is presented in two sections: (1) Literature concerning the selected data synthesis techniques, and their data requirements and (2) Literature related to development of the Classification Instrument. Data Synthesis Techniques The two basic purposes of data synthesis, as described by Pillemer and Light(1980) are to provide a systematic technique, with explicit "com- binatorial strategies" for: (1) focusing on research findings in order to "...provide the systematic information a researcher needs to design more powerful future investigations," and (2) "...rendering scientific research useful to public policy." The authors discuss the fact that 21 22 complex policy questions require more than narrative discussion of several selected research studies relevant to the question; rather they require a synthesis of results of comparable studies in order to elucidate the setting by treatment interactions and effects. "...This can lead to guide- lines about where and how to implement a particular program, improving the chances of its success." Pillemer and Light identify four benefits of data synthesis which are generalizable across content areas, these are: (1) increasing power; (2) obtaining a precise estimate of the size of an effect; (3) describing the form of a relationship; and (4) harnessing the benefits of contradictions. Four categories of data synthesis techniques will be described. (1) Conducting a combined significance test Several conceptually similar techniques that have been identified for conducting a combined significance test are summarized by Rosenthal (1978) and include weighted z scores; adding 2 scores and dividing by the square root of the number of studies combined; adding t scores; adding logs; and by adding probabilities. The purpose of these techniques is to draw an overall "grand" conclusion from results of several comparable in- dependent studies by combining the separate significance tests to address the common null hypothesis which is: both groups have the same popula- tion mean. In order to use a combined significance test technique, the several studies to be compared must be independent of one another, com- pare two treatments which are similar across studies and in which the group differences in each study were statistically treated. The major advantages of this category of data synthesis techniques are: the limited 23 information required (the sample size, value for the test statistic and exact probability); simple computations; and the only formal assumption being that the studies being combined should have tested the same direc- tional hypothesis. A strength of the combined significance tests is that they generally accomplish the goal of increasing power...In general, techniques for conducting a combined significance test are most helpful when the separate studies can be considered indepen- dent and essentially random samples estimating a single 'true' difference between populations, so that variation among study outcomes is attri- butable to chance. In this case, when the treat- ments are in fact differentially effective, an over- all comparison will often detect this difference because it increases the effective sample size used in the test (Pillemer and Light, 1980). (2) Computing an Average Effect Size Glass (1977) is the major proponent for the synthesis technique of computing an average effect size, a technique which Pillemer and Light (1980) describe as a "useful descriptive statistic"; they go on to com- ment: (3) When comparing a treatment to a control, a common definition is simply the difference between the two groups' average standard deviation...There are several elaborations on this basic idea, some of which in- corporate the treatment group's standard deviation, and others are based on the idea of changes over time...Effect size averaging has few formal assump- tions and relatively low informational require- ments. Using this procedure requires only that we know the group means and the control group standard deviation... Estimating an average effect size is most clearly valuable when a group of study outcomes seem neatly, perhaps normally, distributed around the mean. In this case an average gives a useful single summary of results. Investigating Interactions Unlike the two above described techniques which try to gain further insight into a research question by combining similar effects, several 24 techniques instead look for variations and how to explain these. Lesser (1974) identified the policy needs for this type of synthesis across studies and states "Summative evaluations must go beyond assessing simple overall effects...('Does it work?') and attempt to answer these more mean- ingful questions ('For whom and under what conditions does it work?')" (Pillemer and Light, 1980). Glass (1977) developed a method of study- ing variations in outcome using regression analysis and Rosenthal (1978) has developed a procedure for this purpose called the "blocking technique" which uses a combined ANOVA, with studies regarded as the blocking variable. (4) Comparing Similarly Labeled Treatments: the Cluster Approach. One criterion for obtaining results of data synthesis is that com- bined variables actually are the same. One method for determining differ- ence or sameness in treatment variables has been identified by Light and Smith (1971) as the cluster approach - a method which analyzes subgroups within treatment groups. They identified a set of "hurdles" or criteria of comparability against which they measure subgroups. "While the cluster approach does not by itself supply the reason for differences among subgroups, it provides a signal that an atypical program exists... It requires an analyst to recognize that seemingly similar groups may in fact be different." A limitation in use of this technique is the detailed information required about treatment groups and subjects. This technique is usually most useful when combined with one of the above described me- thods, because it only examines comparability of treatment groups and not treatment effect, i.e., effect size, or significance. The authors hate that no one data synthesis procedure is always best and that which technique(s) to use depends on the question a re- searcher*is asking, as well as the particular data contained in the studies 25 being reviewed. A thorough review of the quality and comparability of designs and studies will help to determine the method of data synthesis most appropriate to a given set of studies. Pillemer and Light argue that "Synthesis deals with urgent issues, and the answers it can provide would be useful to many constituencies." It cannot only enhance research in a specific area and improve program planning for greater program success, it can also be a useful technique for studying characteristics of studies and their outcomes across fields. On the latter point they comment: One can ask whether there are general rules per- taining to the relationship between study charac- teristics and outcomes across areas. For instance do observational studies tend to demonstrate larger or smaller effects than randomized experiments... It (data synthesis) enables us to explore whether knowledge in various content areas converges over time...Generating some empirical data about convergence in research findings would not only inform philosophical inquiry, it would suggest which approaches are likely ultimately to be more productive. However, caution must be taken when using data synthesis methods so that the results are valid. Light and Smith (1971) identified sever- al costs to incorrectly combining studies; weakened inferences,overlooked inferences and wrong inferences - each which gives a biased view of actual program effects. To get the least biased combined measures of effect from a synthesis of several evaluation studies, it is important to have confidence in the quality of data available. Whenever a researcher re- views literature on a topic or problem it should be evident that not all evaluation results have equal validity because of the quality of design and procedures used to measure program effect. Eysenck, a critic of data synthesis, (also referred to as meta -evaluation) refers to the process as Mega-Silliness in great measure because of the differing quality of 26 data which is synthesized. In a critique of an article published by Smith and Glass in 1971, Eysenck stated: It is noted that 'subjectivity of the outcome measure' has much the highest correlation with effect size; this alone would invalidate all the complex sta- tistics (presented in the article)...A mass of re- ports - good, bad and indifferent - are fed into the computer in the hope that people will cease caring about the quality of the material on which the conclusions are based. Using the classic approach of Campbell and Stanley (1966) to ex- amine each study's design for threats to internal and external validity is one very useful method of determining the likely errors and types of error inherent in the study's results. Another concern in data synthesis is that of combination of variables. Studies must be thoroughly reviewed to determine if variables are opera- tionalized in the same way. Evaluation results may also not be comparable if the time frame in which effects were measured varies considerably (long term versus short term studies). The more error that can be eliminated from studies before data syn- thesis is performed the higher the chances that the synthesis will yield the most valid results about program effects. It is possible that two studies conducted by using equally rigorous designs and methods, the same variables, time frame and for the same purpose, may yield different re- sults. George (1979), Light and Smfith (1971) and Pillemer and Light iden- tify the effect that situational factors can have on the results of simi- lar, even identical studies which must be considered and identified in the synthesis to help determine under what conditions a program will be effective or most effective. Pillemer and Light report that Glass (1976) has suggested that 'apprehending the meaning of the collected research on educational 27 problems has become a technical problem.‘ (and they add) ...conceptual issues are no less im- portant. For example, is a synthesis approached with the assumption that a program tried in many places has the same effects everywhere, or rather that it may have consistently different effects in different settings? This analysis of data within a single study involves both technical and conceptual issues; the same is true for data from several studies. In fact conceptual issues in dealing with groups of studies are likely to be ever more critical since conclusions may be generalized to a broad field of study rather than to a more narrowly defined program or single data source...0ur conclusion is that a major growth area for both research and policy for the next few years will be systematically developing a science of discovering what we know. Analyses for this study are not limited to data synthesis technique for two reasons: (1) Information about program effects are but one type of evaluation data needed for program planning. Studies con- cerning the context in which programs have taken place, the structural arrangement that support the program, the process or interventions of the program and its cost effectiveness impact are all relevant factors in program planning. Pre- viously conducted studies have much more to offer the plan— ning process than only as contributors to synthetic estimates of effect. A method for classifying. assessing and aggrega- ting studies concerning context, structure, process and impact is also needed. (2) Further analysis is also necessary before the synthe- sis part of this study can be conducted. Studies must be thoroughly reviewed to ensure the quality, compatibility and comparability of pro rams, evaluation designs and methods (Pillemer and Light, IQBO) Development of the Classification Instrument A” Contribution of public health evaluation theory to thedevelopment classification/screening instrument--classification of variables: In 1966 Donabedian published what has now become the classic theor- etical conceptualization of medical care (and now many related areas of health care) evaluation. Donabedian maintains that evaluation of only outcomes of medical care (then the model practice) was insufficient for understanding all of the variables that can and do affect the quality of medical care. 28 Donabedian identified the need to evaluate structure and process variables as well, and when possible using explicit rather than implicit criteria for all variables, including outcome. Structure variables refer to all resources that support a program including but not limited to staff, physical arrangements in which a service is provided, and fiscal and administrative arrangements; process variables are all of the pro- cedures and interactions a provider performs in actually delivering the medical care. (Outcomes were identified as standard morbidity and mor- tality rates.) Since Donabedian developed the structure-process-outcome theory of medical care evaluation, it has become the basis for all medical care ser- vices;medical audits and peer review, and has formed the basis of the Profes- sional Standards Review Organizations (PSRO's) established by the Social Se- curity Amendment of 1972 for review of basic medical services to Medicaid recipients and now for review of hospital based care to all populations. The Donabedian theory has been expanded in use to professional groups Such as nurses, social workers, Occupational therapists and physical therapists in their quality review programs. It has also been made more specific by works such as that by Freeborn and Greenlick (1973) which explicitly identifies evaluation questions for each category of variables for ambulatory care. Other researchers have expanded the theory to include impact variables. Green (1979) states that "...the most succinct statement of the standard of acceptability against which to evaluate impact is cost effectiveness." The process, structure, outcome theory is also used in educational evalu- ation, particularlythrough the contributions of Stufflebeam (1967) and Longest (1975) who in addition to Britan (1978) and others add the cate- gory of context variables. Britan describes contextual evaluation 29 variables as those "...which holistically examine particular program operations...The first step in contextual evalu- ation is a description of what actually happens in program treatment. Analysis then considers why such treatment occurs and how it relates to formal rules, informal goals, varying partici- pant understandings, external pressures and mul- tiple program results. Finally, variation in pro- gram results are linked to variations in the treat- ment process and provide a basis for future pro- gram improvements." Britan quotes Foster (1969) as providing a theoretical anu methodological foundation for evaluation of context; Foster generally studied evalu- ation of social change. "To understand such change, Foster argues, one must consider the nature of both innovating organizations and target groups as well as the setting in which they interact...Contextual evalu- ation uses multiple methods and data sources to gain a holistic perspec- tive on program activities." The categories of variables reported in each evaluation study to be reviewed will be classified as Process, Structure, Outcome, Impact or Context, individually or by some combination of some or all categories. For the purpose of this study, the following definitions have been adapted from Freeborn and Greenlick(1973),Longest (1975)auuiGreen,(1977). Structure: the properties of the available resources that are used by a Health Promotion Program, and the manner in which these resource re- quirements are organized. Process: the activities of those who conduct Health Promotion Programs; program operation, process and procedures. Outcome: the intended and unintended effects or consequences of the pro- gram; it also focuses on how well program objectives have been met. Impact: narrowly defined as cost effectiveness and policy related vari- ables. 30 Context: the nature of the social structure and how it relates to the program being conducted. B. Contribution to developmentcfl the Classification Instrument format fronlgeneral program evaluation theory: The purpose for which an evaluation is conducted as well as the time perspective under which it is conducted effect the comparability of re- sults. A study which has as its purpose comprehensive evaluation of a program, will include both formative and summative evaluation as well as short term and long term effects. Most published studies, however, pre- sent summative, short term results. Formative evaluation, as Worthen and Sanders (1973) put it "serves to improve the product." It is ongoing during a program, and an integral part of good management. It contributes to the "formation" or running of a program consistent with its objectives and needs, and its purpose as Stufflebeam (1970) put it is "not to prove, but to improve" a program's effectiveness. Summative is a term credited to Scriven (1972) which refers to evalu- ation at the conclusion of a program against predetermined goals, and has as its purpose judgement of the effectiveness or worth of a program. The dichotomy of long term versus short term program effects also is important to consider when comparing studies because as Green (1977) puts it "Most of the benefits of health education are time- dependent. These raise problems of behavioral change that must be taken into account in assess- ing program effectiveness and benefits. Most of these have to do with the timing of measurement of outcomes following the educational inputs. Some effects of health education are immediate and temporary, others are slower in developing but longer lasting. C. 31 Contribution of Research Theory to the Classification Instrument The design and methodology part of the classification screening for- mat is based on that developed by Bertram and Brooks-Bertram (1977), and Green and Figa-Talamanca (1974) which are both in part based on the classi- cal work of Campbell and Stanley: In Experimental and Quasi-Experimental Designs for Research (1966), Campbell and Stanley identified three types of designs: 1) pre-experimental designs, which are the least scientific 2) 3) in their design, drawing no comparisons, or comparisions from one time to another on only one group of subjects; quasi-experimental designs, which are attempts to use scienti- fic methods of inquiry using non-equivalent comparison groups, because the use of a true experimental design (which utilizes control group) is not possible; and true experimental designs, which are scientific experiments using random assignment of subjects, to experimental and con- trol groups. Specific type of designs that have been identified by Campbell and Stanley are: 2) Pre-experimental designs - The One-Shot Case Study - The One-Group Pretest-Posttest Design - The Static-Group Comparison Quasi-experimental designs The Time Series Experiment The Equivalent Time - Samples Design The Equivalent Materials Design The Nonequivalent Control Group Design 32 The Counterbalanced Designs The Separate-Sample Pretest—Posttest Design The Separate-Sample Pretest-Posttest Control Group Design The Multiple Time Series Design The Recurrent Institutional Cycle Design The Regression-Discontinuity Analysis 3) True experimental design The Pretest-Posttest Control Group Design The Solomon Four-Group Design The Posttest-Only Control Group Designs Factorial Designs Knowledge of the type of design used enables a researcher to form a series of questions about the quality of the study in relation to the threats to internal and external validity inherent in the design. Factors which Campbell and Stanley have identified as being threats to internal validity are: 1. History, the specific events occurring between the first and second measurement in addition to the experimental variable. Maturation, processes within the respondents operat- ing as a function of the passage of time per se (not specific to the particular events), includ- ing growing older, growing hungrier, growing more tired, and the like. Testing, the effects of taking a test upon the scores of a second testing. Instrumentation, in which changes in the calibra- tion of a measuring instrument or changes in the observers or scorers used may produce changes in the obtained measurements. Statistical regression, operating where groups have been selected on the basis of their extreme scores. 33 Biases resulting in different selection of respon- dents for the comparison groups. Experimental mortality, or differential loss of respondents for the comparison groups. Selection - maturation interaction, etc., which in certain of the multiple- -group quasi-experimental designs. .might be mistaken for, the effect of the experimental variable. Sources of external validity are: 9. 10. 11. 12. The reactive or interaction effect of testing in which a pretest might increase or decrease the respondent's sensitivity or responsiveness to the experimental variable and thus make the results obtained for a pretested population unre- presentative of the effects of the experimental variable for the unpretested universe from which the experimental respondents were selected. The interaction effects of selection biases and the experimental variable. Reactive effects of experimental arrangements, which would preclude generalization... Multiple-treatment inferences, likely to occur whenever multiple treatments are applied to the same respondents, because the effects of prior treatments are not usually erasable. D. Contribution of Existing Classifications or Review Methods nadevelOp- .ment the Classification Instrument: In 1976 Sanders and Nafziger published a monograph in which they presented "A Checklist for Judging the Adequacy of Evaluation Designs." The monograph describes the development of the checklist, based on three sets of criteria: (a) (b) (C) those derived from "Guidelines for Evaluation Designs" those derived from "Essays about Evaluation Qaulity," and those derived from "Checklists that Guide the Application of Standards to Evaluation Designs." The two Guidelines for Evaluation Designs they considered were those of Stoke (1969) and Worthen and Sanders (1973). 34 Stoke's work was a suggested format for final (education) evaluation reports, and contained the following five sections: Section I - Objectives of the Evaluation A. Audiences to be served by the evaluation B. Decisions about the program, anticipated C. Rationale, biases of evaluators Section II - Specifications of the Program A. Educational philosophy behind the program 8. Subject matter C Learning objectives, staff aims D. Instructional procedures, tactics, media E. Students F. Instructional and community setting G. Standards, bases for judging quality n Sectio III - Program outcomes A. Opportunities, experiences provided B. Student gains and losses C. Side effects and bonuses D. Costs of all kinds Section IV - Relationships and Indicators A Congruences, real and intended B. Contingencies, causes and effects C. Trend lines, indicators, comparisons Section V - Judgements of Worth A Value of outcomes B. Relevance of objectives to needs C Usefulness of evaluation information gathered. Worthen and Sanders have identified a similar format that is general and probably applicable to all types of program evaluations. It also consists of five sections: 1. Rationale (Why is this evaluation being done?) II. Objectives of the Evaluation Study A. What will be the product(s) of the evaluation study? B. What audiences will be served by the evaluation study? III. Description of the Program Being Evaluated Philosophy behind the program Content of the program Objectives of the program, implicit and explicit Program procedures (e.g., strategies, media) Students Community (federal, state, local) and instructional context of program valuation Design Constraints on evaluation design General or anizational plan (or model for program evaluation Evaluation questions Information required to answer the questions Sources of information; methods for collecting information IV. FIDO oa>rr1 'fll’flUOW) 35 F Data collection schedule G Techniques for analysis of collected infomation H Standards; bases for judging quality 1. Reporting procedures J. Proposed budget V. Description of Final Re art A. Outline of report(s) to be produced by evaluator B. Usefulness of the product of the study C. Conscious biases of evaluator that may be inadvertently injected into the final report The essays about evaluation quality that were reviewed including Stufflebeam, et al's 1971 work on evaluation standards, and Worthen 1973 "A Look at the Mosaic of Educational Evaluation and Accountability." Stufflebeam, et al's standards contain 11 factors, which are: Internal validity External validity Reliability Objectivity Relevance Importance Scope Credibility Timeliness Pervasiveness, and Efficiency Hommwmmnwmi—o HH Worthen's similar set of standards also contains 11 factors: Conceptual Clarity Characterization of Program Recognition and Representation of Legitimate Audiences Sensitivity to Political Problems in Evaluation Specification of Information Needs and Sources Comprehensiveness/Inclusiveness Technical Adequacy Consideration of Program Costs Explicit Standards/Criteria Judgements and/or Recommendations reports Tailored to Audiences. ' I—IOSOCDNOSU'I-PDWNH HH Four checklists were reviewed, and Sanders and Nafziger state: "Each existing checklist seems unique in form, content, and purpose; nevertheless, many share common characteristics. Generally, checklists for judging evaluation designs include considerations of the scientific or technical adequacy of the evaluation, the practicality and cost efficiency of the 36 design, the usefulness of the data to be collected, and the responsive- ness of the design to legal and ethical issues." The checklist developed by Stoke in 1970 considers five major areas and several sub-areas. The five major factors considered are: (1) the evaluation itself; (2) specific information about the program being evalu- ated; (3) program outcomes; (4) relationships and indicators; and (5) the worth of the overall program. A checklist developed by Bracht in 1973 identifies questions for six categories: (1) communication; (2) importance of the evaluation; (3) de- sign for making judgements; (4) design for obtaining descriptive data; (5) reports; and (6) concerns. An administrative checklist developed by Stufflebeam in 1974 con- tains six components which questions the adequacy of evaluation design as well as contextual and process information. The six factors reviewed are: (1) conceptualization of the evaluation; (2) socio-political factors; (3) contractual/legal arrangements; (4) the technical design; (5) the manage- ment plan; and (6) moral/ethical utility considerations. A checklist by Smith and Murry (1974) is a compilation from other checklists and assess the adequacy of three major areas and several sub- areas of evaluation design. The three major areas are: (1) content des- criptions; (2) evaluation activities/results; and (3) document character- istics. Sanders and Nafziger considered previously mentioned work, in creat- ing their checklist, but developed a check list which has as its unique- ness, the purpose of not only assessing the adequacy of an evaluation de- sign, but also facilitating communication between an evaluator and rele- vant parties. The authors note that: 37 It is very important to remember that an evaluation design is a vehicle for communication between an evaluator and those whose role calls for reviewing the evaluation plan. The checklist helps organize that communication. In cases where an evaluation is conducted by a contractor, the design becomes a con- tract between the evaluator and the client. In such cases the checklist assists a client in judging adequacy of the design, and provides a basis for a client in judging adequacy of the design, and pro- vides a basis for giving feedback to the evalu- ator. If the evaluator is involved in the program being evaluated, the guidelines provide a basis for the evaluator and his or her colleagues to check the design." The Sanders and Nafziger Checklist contains four general criteria sections, each with specific questions that assess whether a specific criterion was met. The sections are: 1. Criteria concerning the adequacy of evaluation planning. II. Criteria concerning the adequacy of the collection of and process of information, III. Criteria concerning the adequacy of the presentation and reporting of information, and IV. General criteria (ethical considerations and protocol). Issac and Michael in 1974 published a checklist by Wandt, a "Form for the Evaluation of an Article,“ consisting of 25 items concerning the clarity of statements and appropriateness of procedures described. Each item can be rated in a checklist format, on one of four criteria: (1) Completely incompetent (2) Poor (3) Moderate (4) Excellent. Each of the checklists, standards sets, and guidelines that have been briefly reviewed helped to identify important factors to include in a Classification Instrument, but none of those reviewed was appropriate to the needs of this study because each was intended for review based upon 38 lengthy reports, documentation and interaction with program and evalua- tion staff. Most required more information than is generally available from a journal article and the focus of many was very decision oriented in relation to the context of specific programs. The review of existing checklists helped to identify more clearly the purpose of the Instrument - to assess and classify the content and quality of intervention strategies, evaluation methods and results in order to identify comparable and compatible studies for data synthesis of programs' effects (outcomes) and to aggregate program experience con- cerning other important levels of program planning and evaluation (i.e. structure, process, context and impact.) m For this Chapter, a literature review was combined with a descrip- tion of the theoretical basis of the Classification Instrument and data synthesis techniques. In Chapter III the Classification Instrument and data set will be described and the data analysis procedures (both classification and syn- thesis) will be discussed. CHAPTER III PROCEDURES In this chapter, a description will be given of the way in which thecflassification instrument and data synthesis techniques were planned for application to the data. The sources of data, and data set will be described, as will be the methods of data collection. The procedures for analyzing and presenting the data are also described. Development of the Classification Instrument No existing instrument which satisfied the requirements of this study could be identified, so a Classification Instrument was developed based upon the literature review of other similar instruments, meta evalu- ation criteria and guidelines, public health evaluation theory, educational evaluation theory, general program evaluation theory, and research theory as described in the previous chapter- It was pretested and changes were made to refine and improve it. The Instrument assesses both the content and quality of program ex- perience and was developed for use with journal articles and final pro- ject reports. (This specification is identified because several exist- ing instruments assessed contextual aspects of program evaluation that almost never are reported in journal articles.) The instrument content of each study by specifying the: objectives of the program target population intervention strategies used objectives of the evaluation type of evaluation specific variables measured methods of assessment evaluation design methods of data analysis 39 40 time frame for evaluation of effects (long term vs. short term) direction and significance of results The instrument assesses the qualityof each study by scoring the fol- lowing variables as to their adequacy: literature review evaluation questions hypotheses independent variables dependent variables instrument pretest instrument reliability and validity sampling size and procedures evaluation design statistical analysis interpretation and results Each variable (as identified above) was scored and then an overall score was assigned to each study by adding each item's score. The possi- ble scores were assigned from least to most desirable. Each of the items was scored as follows: 0 not present 1 present inadequate or poor 2 present adequate 3 present excellent The instrument is presented in Appendix B. One or more of the data synthesis techniques described in the previous chapter will be used. One of the purposes for analysis and classification of the literature is to identify which data synthesis technique for each study is appropriate by determining the purpose, pro- cedures and data and design limitations and quality. Because each des- cribed technique is based upon different underlying assumptions and data requirements the exact data synthesis techniques to be used could only be determined after the classification phase of the Study was completed. Pilot Study A pilot study was conducted on studies covering the five year per- iod of 1975-1979. The general topic of evaluation of non-patient health 41 Education programs was selected for the pilot study; a total of 151 stud- ies were identified using a manual search of Index Medicus. The headings searched were: Health, Health Education, Preventive Medicine, and Pub- lic Health Surveys. About 15% of the articles were in journals to which the MSU library system does not subscribe (and therefore those articles were not reviewed for the pilot study); about 10% of the 151 articles were from British or Canadian Journals, and about 10% had misleading titles. About 20% more articles were added to the list from references found when 50 selected articles (about one third of the original list) were reviewed and from review of selected journals in which several arti- cles were identified via the Index Medicus search. Scope of the Data From this experience, several decision rules were made concerning the identification of data for this study. Because of the breadth of literature encountered in the one sub area of health promotion/prevention activities reviewed for five years (namely non patient health education) it was decided that the topic (by which the Classification Instrument and Data Synthesis techniques would be tested out) for this study must be more limited. Limitations were placed by topic, target population and country. Community nutrition education studies for adult non-patient non-institutionalized populations in the U.S. as reported in American Journals were selected for testing out the combined Classification and Synthesis "System". The rationale for selecting evaluation studies of community nutri- tion education was threefold: (1) it represents an area in which there has been multi-disciplin- ary program approaches, -a program history which can contribute richly to the .multl-disciplinary program requirements of Health Promotion pro- grams; 42 (2) evaluation of community nutrition education programs have been reported less often than evaluation of other priority health promotion/ prevention areas, (like alcoholism, drug abuse and hypertension etc.) and there a comprehensive review of nutrition education program literature for non-patient adults over the past 20 years is within the scope of this study. (3) nutrition education programs represent one of the 15 priority areas of Health Promotion/Prevention as identified by the U.S. Department of Health and Human Services, 1980.and (4) some of the principle aSsumptions identified upon which improved nutritional objectives are based include the need for improved dissemination and growth of current research efforts to improve the scientific base with respect to diet and disease, and the need for identification of effective measures of nutrition edu- cation. This study could, in small part, contribute to both needs. The rationale for selecting adult populations is that that group is most appropriate for study by an adult educator. Non patient populations were selected to further narrow the body of studies so that a comprehen- sive review of one subset of Promotion/Prevention priority area could be reviewed. It was assumed that such a comprehensive review was necessary to adequately test the developed "system". The reason 1960-1980 time period was selected was because of trend in program evaluation over that period. Thaherty and Morrell (1978) reviewed the history of the field of program evaluation and identi- fied that although there were several early developments in program evalu- ation in a number of areas in the late 1940's and early 1950's, the field of program evaluation didn't begin to develop rapidly until the 1960's. They report that this development was due to a combination of four causes: 43 ...greater requirement for accountability in pub- licly funded programs, increasing interest among social scientists in social relevance, and increas- ing scarcity of resources for the traditional social sciences, and an expansion in the social science methodologies appropriate for research in applied settings. Obviously none of these causes operated independently, and some probably played a more im- portant role than others. A review of evalution of public health, health education and nutri- tion education programs indicates that they conform to the general trends identified above. Source of Data The decision rules developed concerning identification of evaluation literature for this study were: 1) 2) 3) 5) 6) only studies from U.S. journals concerning programs in the U.S. would be included; studies would be limited to those which concern evaluation of nutrition education programs for non patient adults; only studies from 1960-1980 would beincluded Index Medicus was the first source used to identify studies. Each of the following headings were searched manually: Health Health Education Nutrition Nutrition Education Nutrition Survey Preventive Medicine Public Health Administration Nutritional Review Abstracts were searched under the topic Nutrition Education. The following bibliographies were also used in the data search: Otterness, Eleanor G. Bibliography on Evaluation in Health Education. Group Health Plan Inc. 2500 Como Ave., St. Paul MN. Prepared for September 12, 1980 Third Annual 44 Meeting, Minnesota Chapter, Society for Public Health Education at the University of Minnesota Arboretum, Chaska, Minnesota. Theme: "Evaluation Implications and Applications for Health Educators." Owen, Anita Yavochek: Community Nutrition in Preventive Health Care Services: A Critical Review of the Literature. Health Planning Biographic Series #7, USDHEW Public Health Service, Health Resources Administration, May 1978. Wilson, Christine and Knox, Sharron. "Methods and Kinds of Nutrition Education (1961-72): A Selected Annotated Bibliography" Journal of Nutrition Education Vol 5, No. 1 Supplement 2, Jan. - March 1973. 7) The references from each article reviewed were cross checked against the Index Medicus derived list and added if not already there. 8) Journals which were cited more than once per year in the search were reviewed for additional titles that appeared to concern the topic of study and many articles were reviewed to determine if they were appropriate to this study. The Data Set A set of 111 studies were identified as potentially meet- ing the criteria for inclusion in the study. Each was reviewed for appro- priateness and only those which described a community nutrition education program, its intervention strategies and some statement about evaluation were included in the data set. Appendix C contains a listing of the ori- ginal potential data set and disposition of each regarding inclusion in the study. Forty-six articles make up the data set for the study - each being identified as a single subject. 45 Procedures for Data Analysis The data were classified through use of the Classification Instrument. Each article on the list was analyzed separately concerning its content, quality and whether it met any of the criteria for any of the selected data synthesis techniques. Then, the studies were grouped according to ex- perimental design, quality categories within those designs (i.e. ranking according to overall score), and then assigned to appropriate data synthe- sis technique. Appropriate data synthesis was then done, results of the Classificaton and Synthesis were then compared by experimental groups and quality sub groups. Summary In ChapterIII,,an explanation has been provided for the proce- dures used to analyze the data. The sources of data were identified, and a description was given about the ways in which the data were collected and studied. The classification Instrument and Data Set are presented. In Chapter IV, results of the Classification and synthesis will be presented. 46 CHAPTER IV Results Published studies of one type of the Health Promotion program, nutrition education for non-patient non-institutionalized adults, were analyzed using a classification instrument which assessed the content and quality of the studies as well as their ability to meet data synthe- sis requirements. The data resulting from the analysis is presented in three parts: the content of the studies, their quality, and their appropriateness for use in data synthesis. Data Relating to the Content of the Studies A total of 46 studies were analyzed. The content of each study was classified by the following factors: study number author and year purpose of the program target population educational approach used length of program setting instructor sponsor type of evaluation method of evaluation 47 categories of evaluation variables, and results Data concerning each factor will be presented. Table4.1 displays the year of publication by the Journal in which each study was published. The trend identified by Haherty and Morrell (1978) is somewhat evidenced, by few studies existing in the early 1960's, and a large increase in number of studies per year by 1969. The continued "growth trend" identified by Haherty and Morrell however, has not mani- fested itself,with the number of studies falling off from their peaks in 1969 and 1972 to only two in 1980. See Table 4.1. Table 4.2 presents a frequency distribution of categories of program purposes. The type of purpose most frequently reported can be categor- ized as general nutrition education; 19 studies (41.30%) were in this category. Two other categories were high, with programs to improve nu- trition behavior being reported 12 times (26.09%) and programs to im- prove nutrition knowledge, behavior, and or attitudes reported in nine studies (19.57%). Programs conducted to study cost effectiveness of specific programs as well as programs to learn more about program parti- cipants and instructors were each reported two times (4.35% each); pro- grams to improve nutrition knowledge and programs to assess the relative functioning of different methods of instruction were each reported one time (2.17% each). See Table 4.2. Programs ranged in length from one encounter only to repeated en- counters over 14 years, while the longest duration of evaluation data col- lection was for three years. The most common setting for the educational programs were indivi- dual consumer homes, followed by community centers and super markets. The most frequent sponsor of the programs was the Cooperative Extension Service for 22 (47.83%) of the studies; a variety of other sponsors 48 84 N H H a N m m N o m N N H N N N H , HHHQNI H H HHcheeOH .N N N cowmcmuxm ...b .n. N H H 52...: 3 He; 22:32.. c8225 H H mHaHHamoz HH m N H H H H N .8 55.:sz .H. N N N N N ...5: 5:332 H H 3.25:8“ 85: cw 3oz m.pm;3 a H H H H .>mN muH>Emm .qu m H N H H H muHsocouu 850: .a oH N H H H H H H H H .UOmm< UHHm lumwo snowcms<.a. ommH mNaH NNmH NNNH mNmH mNmH eNmH NNmH NNNH HNmH QNNH ammH NomH NmmH oomH mmaH HGNH umgmHHaaa tam» umgmwpnsa mew: mmwusum cows: cw Pacezoa An cowumowpazm we gmw> Hé open... Haggaoc 49 Table 4.2 Frequency Distribution of Categories of Program Purpose n = 46 Categories of Frequency Program Purpose # % General nutrition education 19 41,30 Improve nutrition behavior 12 26.09 Improve nutrition knowledge & be- havior and/or attitudes 9 1g 57 Cost effectiveness analysis 2 4.35 Understand more about program participants and instructors 2 4.35 Improve nutrition knowledge 1 2.17 Assess relative functioning of different methods of instruction 1 2,17 46 100.00 50 were reported between one to three times each. These were: multiple agency'Sponsorship ; the National Diet and Heart Study; the Dairy Council; the National Heart, Lung and Blood Institute; Social Service agencies; Universities; supermarkets, hospitals; and the Center for Disease Control. Table 4.3 presents a cross tabulation of profession of instructor by target group. The most frequently reported instructors were para professional nutrition education aides, 18 studies (40.00%); the second most frequently reported were nutritionists and home economists, each identified in five separate studies (11.11%) each. Instruction via the media was reported in four studies (8.89%); and teams of professionals and paraprofessionals accounted for another four (8.89%) of the instruc- tors. Dieticians were reported in three studies (6.67%) and self instruc- tion was reported in two studies (4.44%). In four studies the instruc- tors were referred to in general terms and their profession could not be identified. Health educators were reported in only one study and only as part of a team. Considering target group, low income was the most frequently identified group, being reported in 25 articles (55.56%). The general public was mentioned with next highest frequency, 10 studies (22.22%), followed by senior citizens and supermarket shoppers each being reported in three separate studies (6.67% each). Middle age men, police officers, employees of the National Institute of Health, and Cooperative Extension Service invited homemakers each were reported one time (2.22% each). The only definitive trend concerning target group and instruc- tor is between low income homemakers and paraprofessional aides. ‘All paraprofessional aides taught low income homemakers, and of all the stud- ies reporting on low income homemakers, in 72% of these, the homemakers were instructed by paraprofessional aides. See Table 3. .mcHucaoL Ho mmamuoa uoo.ocH aHuuuxo cu a: can we: meet Hope» Hcv muH>Lom concmaxm o>Huoemnoou ago mH muu Hmv HHHaaz co aHzHHHmcH HacoHHaz as» m_ 1H: HNH .uuuumaxo mo av Hnaoo Ho: moon mNHm «HaeNm ecu mnemocosu 51 .ENQH a mo occho: No: as; .couuzeumcH one case aces umueonwc Huuoh $8 a moHuHugo Hague .couoacumcH o: meowmcogu can Eucmocn HmcoHuuacNm a 30¢ Hogwaamcu u Hog .:H o: to: mmouoca use ocauuacum Ho moHuaum «Eon mmaouoa men: HHH a ass—cu coH me NN.N H 1~NHN H NN.N H NN.N H scum m sm.m m NN.NN oH mm.mm NN.N NN.N cc.¢ NN.N co.mN H oo.om N oo.mN H umHHHuuamca o oo.ooH oo.m oo.o o¢.¢ ee.¢ oo.ooH N ccHHoacumcH mHom N oo.m No.c No.c oo.ooH m mcaHuHHoHa m oo.NH NN.N NN.N No.8 oo.mN H oo.mN m manop v oo.e oo.NH NN.N NN.N NN.N NN.N NN.N oo.mN H oo.mN H co.mN H cc.mN H NHuoz e oo.ocH cc.ooH NN.Nm oo.e NN.N NN.N NN.N ee.¢ HH.HH oo.oN H oo.oN H oo.0N H oo.o¢ mumHEocouu «so: m nn.mn mm.nm oc.e oo.m N NN.N ee.e NN.N NN.N HH.HH oo.oN H oo.o¢ N oo.oN H cc.oN H mHchoHuchzz m oo.ocH No.00 mm.mm oo.e oo.o¢ mmuHo oo.e oo.coH NH HugonmoHoca use; NH H co.NN I N 0d WVW $51 .US .0 H1 MM mm mm, mm 3m Wm“ ...H mm mm .2... man mu 3 H um“... um. mm mm mu H.031 H.“ U “.49 W 80 mm. H 5 ”IN “w JU S «on O I, H Hme a c aaocw Homes» H: LouuagumcH Ho :onmoHoea mo coHNmHanN» mmocu H m . 5 ~32. 52 Concerning the evaluations that were conducted, 16 (34.78%) were summative with the majority formative: 30 studies (65.22%). (This is in part a reflection of the many developmental studies that were conduct- ed prior to and early into development of the Expanded Food and Nutrition Program (EFNEP) of the Cooperative Extension Service, which began eleven years ago). Evaluation results were categorized according to category of vari- ables assessed, methods of assessment and direction of reported changes. Table 4 presents a summary of the frequency distribution of the studies by the variable categories of structure, process, outcome and impact. (No studies concerning context variables were present in the sample.) Of the total of 46 studies in the sample, one (2.22%) assessed structural variables; three (6.67%) process variables; 43 (93%) outcome variables, and three (6.67%) impact variables. (Percents do not add up to 100% because some studies identified more than one category of variable.) Table 4.4 Categories of Variables Assessed by Study Number n = 46 Structure Process Outcome Impact 25 16 1-15 12 18 17-19 15 20 21-24 43 1 3 43 3 2.18% 6.52 93.48 6.52 Number do not add up to n=46 because some studies included more than one category of variables. 46 is the base figure for the table, therefore the percentages do not add up to 100%. 53 In the one study concerning structure, the specific variables investi- gated were socio-demographic characteristics of Expanded Food and Nutri- tion Education Program Aides and homemakers (clients). Table 5 presents this finding by study number. Table 4.5 Specification and Frequency Distribution of Structural Variables by Study Number Structural Variables Frequency Reference Socio-demographic characteristics of EFNEP* Aides and homemakers 1 25 * Expanded Food & Nutrition Education Program Table 6 identified the process variables that were assessed by study number. All three evaluated multiple educational approaches and techniques. Table 4.6 Specification and Frequency Distribution of Process Variables by Study Number Process Variables Frequency Reference Educational approaches/ techniques 3 16, 18, 20 Table 7 presents a frequency distribution of outcome variables in- vestigated by study number. The outcome variables were divided into three groupings: intermediate outcomes, direct outcomes and side effects. 54 Intermediate outcomes are those which can enable a change in knowledge, attitudes, values and beliefs or behavior, but in themselves do not re- present a change. A goal of a project may be "to increase knowledge a- bout nutritious foods" in a supermarket and the only measure of the pro- gram may be the number of people who stop at an information booth for handouts concerning facts about nutritious foods. AReceiving handouts in no way can be used as a proxy for a change in nutrition knowledge or behavior, bUt it may predispose such change. It therefore is considered an intermediate outcome. Sixteen intermediate outcomes were reported in 13 separate studies. Receipt of follow up mail from consumers was the 1 most frequent intermediate outcome, (five studies); four studies re- ported the number of people stopping at a display and/or taking handouts; four studies also reported other general feedback; two reported use of service and one study reported the increase in number of listeners to a radio program. Forty-seven direct outcomes were reported in 32 differ— ent studies. Direct outcomes were defined as knowledge, attitudes, values, beliefs and behaviors that did or were expected to change as a result of the intervention strategy. The most frequently reported outcome was change in behavior (27 studies); followed by no change in outcome (six studies); positive change in attitudes (five studies); no change in be- havior (four studies); participant satisfaction (three studies) and im- proved clinical tests (two studies) and positive change in knowledge (one study). Two studies investigated side effects, which were identified as out- comes not directly related to knowledge, behavior, attitudes or clinical studies concerning nutrition. Each of the side effects was reported once, in separate studies, and are: improved personal habits and atti- tude and community related behaviors. Specification and Frequency Distribution of Outcome Variables by Study Number 55 Table 4.7 n = 43 Outcome Variables Frequency Study Number Intermediate: -No. of people shopping at a display and/or taking handouts 4 1,3,5,6,8 -Use of a service 2 7,42 -No. of listeners to a radio program 1 15 -Receipt of follow-up mail from consumers 5 29,39,5,7,41 -Other general feedback 4 2,6,7,17 Direct Outcomes: -No change in knowledge 2 10,11 -Positive change in knowledge 5 11,24,30,36,4O -No change in behavior 4 10,11,12,33 -Positive change in be- 3,4,8,9,11,12, havior 28 13,14,18,19,21, 22,23,24,27,28. 30,32,33,34,36, 37,41,43,44,45, 46,40 -Positive change in attitude 4 8,37,38,40 eParticipant'Satis- faction 3 31,40,42 -Improved Clinical tests 2 23,34 Side Effects: -Improved personal habit and attitude 1 37 -Community related behaviors 1 33 56 Table 8 presents specific impact variables by frequency and study number. Three studies assessed impact variables: cost-effectiveness was investigated in two studies and policy changes were assessed by one study. Table 4.8 Specification and Frequency Distribution of Impact Variables by Study Number n = 3 Impact Variables Frequency Study Number Cost effectiveness 2 12, 43 Policy Change 1 15 The techniques used in collecting evaluation results were analyzed by frequency of being reported, outcome variables and their results . Findings are presented in Tables 9 - 11. Table 9 presents techniques for assessing behavior and results. Six methods were used in assessing behavior. The most frequently reported assessment method were survey interviews, reported in 11 studies or 39.39% of studies assessing behav- ior. Informal reports were the second most frequently reported method for assessing behavior; eight studies (28.57%) used this method followed by six (21.43%) reporting record review, five(17o86%) self-administered questionnaires, four (14.29%) observation and three (10.71%) informal comments by participants. All but one study claimed some positive nu- trition-related behavior changes. Table 10 reports the techniques that were used to assess knowledge and their results. The method most frequently used to assess knowledge 57 Table 4u9 Techniques for Assessing Behavior Results n = 28 Data Sources Frequency Number Claiming Some positive # % . Behavior Change Survey 11 39.29 10 Self-Administered Questionnaire 5 17.86 5 Observation 4 14.29 4 Record Review 6 21.43 6 Informal comments by participants 3 10.71 3 Informal reports 8 28.57 8 Totals *37 *36 Adapted from Bertram and Brooks-Bertran (1977) *Total exceeds n because more than one technique used in some studies. Percentages were based on 28 and therefore do not add up to 100% 58 outcomes were self administered questionnaires, reported in three studies (50%). Survey interviews were reported in two studies (33.33%) and re- cord review was reported once (16.67%). All but one study claimed an increase in knowledge as an outcome of their intervention. Table 4.10 Techniques for Assessing Knowledge and Results n = 6 Frequency Number Claiming Technique of Studies Increased Knowledge # % Survey Interview 2 33.33 1 Self-Administered Questionnaire 3 50.00 3 Record Review 1 16.67 1 Totals 6 5 Adapted from Bertram and Brooks-Bertram (1977) Table 11 displays the techniques used to assess attitudes (including satisfaction and their results. Of the four methods reported, the me- thods most frequently used to assess attitudes were self-administered questionnaires, and record reviews each reported in two studies. Survey interviews and informal reports were each used once. All of these studies reported positive attitudes. 59 Table 4.11 Techniques for Assessing Attitudes, Including Satisfaction n = 6 Number Claiming Number of Positive Attitude Technique Studies (including satisfaction) Survey interview 1 1 Self administered questionnaire 2 Informal reports 1 1 Record Review 2 6 6 Appendix 0 contains a detailed summary ofthe content of each study by study number, author and year of publication, program purpose, target popula- tion, educational approach used, length of program, program setting, instructor, sponsor, type of evaluation, and results. Data Relating to the Quality of the Studies The quality of studies was assessed by assigning a score to each study based upon whether or not several criteria factors were presented adequately and clearly. A score of one indicated that the factor was present but limited, inadequate or unclear; a score of two indicated that the factor was adequate and clear and a score of three indicated excellence. The factors which were scored were: Evaluation Design: Type of design Purpose Evaluation questions Hypothesis . Whether l1m1tations of the design were discussed; 60 Measurement Instrument Instrument Pretest Reliability of Instrument Validity of Instrument Whether limitations in the instrument were discussed; Statistical Analysis Descriptive Statistics Inferential Statistics Level of significance Whether limitations of the statistical analysis were discussed Sampling Sample size Sample description Population description Sampling methods Response rate d Whether limitation of the sampling method were discussed, an Literature Review. An adequate and clear study had the possibility of obtaining a score of 46. Table 12 presents a frequency distribution of the scores assigned to the sample. The scores ranged from one to 32, with a mean of 10.09, modes of one and six, and median of nine. See Table 12. Appendix E presents the summary listing of each study by each cri- terion category, criteria score and overall score. Table 13 displays each criterion category by range of scores and mean score. In the evalu- ation design category, the scores ranged from one to eight with a mean score of 2.67; the Measurement Instrument category had a mean of 1.48 and range of scores from zero to ten; the category of Statistical Analy- ses had a range of zero to ten and mean of 2.17; the Sampling category had a mean of 2.87 and range of one to ten, and the Literature Review category had a range of zero to two and mean of .87. 61 Table 4.12 Frequency Distribution of Quality of Study Scores Scores Frequency 1 6 3 5 4 6 5 1 6 2 7 2 9 3 10 2 11 3 12 2 13 1 15 2 16 2 17 1 18 2 20 2 21 1 22 1 24 1 32 1 46 X = 9.28 m = 1, 4 M = 9 62 Table 4.13 Quality of Studies Criterion Categories by Ranges in scores and Mean Score n = 46 Criterion Category Range in scores 'x Evaluation Design 1 - 8 2.67 Measurement Instrument O - 10 1.48 Statistical Analysis 0 - 10 2.17 Sampling 0 - 10 2.87 Literature Review 0 - 2 .87 In order to elucidate why such a range in scores exist, the scores were analyzed by type of evaluation design. Table 14 presents a grouping of scores by type of evaluation design. Seven studies were conducted using a quasi-experimental design and 38 using a pre-experimental de- sign. Only one true experimental design was reported. The mean, modeaand median for the quasi-experimental groups were respectively: 8.68, 1 and 4, and 6; and the scores ranged from 1-32. The range of scores for the quasi-experimental designs were 11-22 with a mean of 16.38 and median of 16. No mode is reported, because each score occurred only once. Table 4.14 Mean, Mode & Median & Range scores by Type of Evaluation Design n = 46 Type of Design n Y' m M Range Pre experimental 38 8.68 1,4 6 1-32 Quasi-experimental 7 17.14 17 13-22 True experimental 1 IO Because the range of scores overlapped and quality differences still not clear, the pre experimental scores were again 63 divided between those which used only case histories or informal report- ing to those which used some type of systematic pre-experimental design, such as post test only, or one group pre and post test. Table 15 dis- plays results of this analysis along with previously reported data from the quasi-experimental group, which is designated as Group C. Group B represents those pre-experimental studies using a systematic pre-experi- mental design; Group A is made up of the pre-experimental studies which did not use a systematic design. The range of scores for the 25 Group A are 1-15, with a mean of 5.04, modes of one and four and median of four; the range of scores for the 13 Group 8 scores is three through thirty-two, with a mean of 15.69, modes of 10, 16, and 18 and median of 16. Group C consists of seven studies with mean of 17.14, Median of 17 and score ranges from 13-22, By this breakdown, the extreme cases seem to be mostly in Group A, and Groups 8 (the systematic pre-experimental) and Group C (the quasi- experimental) designs yield very similar results. Table 4.15 Mean, Mode & Median & Range of Scores by Evaluation Design Groups n = 46 Design Group n 'x m M Range Group A 25 5.04 1,4 4 1-15 Group B 13 15.69 10,16,18 16 3—32 Group C 7 17.14 17 13-22 Data Relating to Synthesis of Outcomes Each of the 43 studies of program outcome were analyzed with re- gard to the requirements for each data synthesis technique. Each 64 technique's assumptions and data requirements will be summarized along with appropriate use with this sample of studies. The Cluster Approach (Light & Smith, 1971). The cluster approach has the following requirements: 1. Access to original data of each study 2. Review of quality of studies. "The studies accepted for further an- alysis would be only those which meet the following three standards: a) All subjects in the study must have been selected from a known and precisely definable population. b) A study's dependent variable and those independent variables which are measured must be measured in the same way as, or in a way subject to a conversion into, those employed in the rest of the studies. c) Overall, the instrumention and quality of the experimental work must be generally comparable to that in all the rest of the studies." 3. Examples of appropriate research questions using the Cluster method. a) Of the specific approaches being analyzed, which offers the greatest improvement in (specific primary dependent outcome vari- able)? b) Is the relationship between pre and posttest score identical in all clusters? c) Are the variations in the differences between (two subsets of subjects, i.e. men-women; young-old) on a criterion variable from cluster to cluster associated with the type of program? No published report provided access to original data and there- fore this technique was inappropriatefor use with this set of studies. 65 Investigating Interactions - The Blocking Technique (Rosenthal, 1978). The requirements for this technique are: The procedure is appropriate for studies that compare the same two programs, but not necessarily on similar groups of subjects. Data requirements for each study: a) mean b) sample size c) standard deviation Moderate computational difficulty: "The 'blocking technique' involves comparing the outcomes by casting the results into an overall analysis of variance ANOVA, with studies regarded as a blocking variable...mean squares can be constructed and a two-way ANOVA (treatments by studies) can be performed...Studying the main effects of treatments from the ANOVA provide an average measure of their differential effective- ness" Pillimer & Light (1980) p. Since no two studies compared the same two programs, the Blocking Tech- nique was not appropriate for use on any of this set of studies. Conducting a Combined Significance Test (Rosenthal, 1978). The requirements of this technique are: Computational simplicity: "Example for combining z scores "If two groups are compared in each study, there is a z score associated with each reported p value. The z's are added across studies, and their sum is divided by the square root of the number of studies that are combined. The prob- ability- value associated with the resulting overall score provides the level of significance for the combined statistical test." Pillimer & Light (1971) p. 66 2. No review of quality of potentially combined studies is recommended. 3. Requirements: for each study the following data is necessary a) sample size b) value of the test statistic (+, z, or F) 4. Studies must have tested the same directional hypothesis. 5. Example of an appropriate research question using the combined signi- ficance test method: What grand conclusion can be drawn concerning the comparability of two treatments? 6. Preferred conditions: a) the separate studies are independent, and b) random smaples were used in estimating group differences. No studies used a z or F test statistic, but six studies reported the t test statistic. Of those, five reported a sample size, and only one of those specified a hypothesis. The Combined Significance Test is also not applicable to any groupings of the studies. Computing an Average Effect Size (Glass, 1977). The requirements for this technique are: 1. Low informational requirements: a) group means and b) control group standard deviations 2. Few formal assumptions a) "Mean scores should be used to characterize a data set only after studying the destribution of the data, to determine if the mean is in fact a good descriptor." b) "...when the distribution of outcomes is unusual, procedures that focus on variation may prove more useful." c) same type programs must be compared. 67 3. Example ofén1appropriate research question using the average effect size method: On the average, is the specific broad type of program beneficial? Four studies reported means and standard deviations. Only one study reported means and standard deviations for a comparison group, and the means and standard deviations were all for different variables: in one study mean caloric intake, cholesteral levels, weight and satur- ated and polyunsaturated fats levels were reported. In another study, mean scores on pre and post nutrition workshop test scores were reported. Another reported mean food energy, protein, calcium, Vitamin A and Thia- min levels of diets, and the final study reporting means concerned weight loss. The Average Effect Size technique was also found not to be appli- cable to any combination of the studies in the sample. Summary In Chapter’TVthe sample of studies was analyzed using a classi- fication instrument, which assessed content, quality and appropri- ateness for use in data synthesis. The most commonly reported purposes for the programs were general nutrition education, programs to improve nutrition knoweldge, behavior and/or attitudes. The most often reported instructors were paraprofessional aides and the target group most fre- quently identified were low income homemakers. Most of the studies evaluated outcomes of their programs, although many of these were for formative rather than summative purposes. Most studies of outcome re- ported positive outcomes. Concerning the quality of studies, a wide range of quality was identified in evaluation design, measurement instruments, sampling, 68 statistical analysis and literature review. No true experimental designs were reported. The majority of studies were pre experimental and about 17% were quasi experimental studies. Two patterns emerged from the analysis: the studies reporting quasi experimental designs when consider- ed as a group, reported the highest overall mean score, when compared to pre experimental studies which used systematic evaluation designs (which had an overall mean score almost as high as the quasi experimental group); pre experimental studies which did not use systematic evalua- tion designs had the lowest overall mean score, much below the compara- tive group means. The studies were each reviewed according to the requirements for each of the selected data synthesis techniques and no studies were iden- tified that satisfied the requirement for any of the synthesis techniques. In Chapter V, a summary will be given, and each of the research questions for this project will be discussed and answered in light of the findings. Recommendations and conclusions along with further research needs will be presented. CHAPTER V Summary, Conclusions, Recommended Guidelines, Suggestions for Further Study and Reflections "Program evaluation is frequently thought of as a dry, arid, and fruitless endeavor extolled in theory, but ignored in practices. Under con- ditions of limited resources and limited require- ments, the application of program evaluation is more than an opportunity - it is a necessity. Rather than a wasteland, it is one of the last main frontiers in public administration. Al Loeb. In this chapter, a summary of the study, conclusions drawn from interpretations of the data, recommended guidelines for planning and evaluation of Health Promotion programs as suggested by the study are pre- sented along with suggestions for further research concerning the use of classification and synthesis techniques and their potential contributions to the development of Health Promotion programs. Reflections concerning policy needs and influences concerning realization of national Health Promotion goals are discussed. The purpose of this study was to test the applicability of data classification and synthesis techniques for use in planning and evalu- ating Health Promotion programs. Specific research questions were developed related to the purpose of the study. A subset of one of the 15 identified Health Promotion priority areas was selected as the focus for the study: community nu- trition education programs for non-patient, non-institutionalized adults in the U.S. A sample of studies for analysis was selected from journals 69 70 published in the U.S from 1960 through 1980. Studies were analyzed using an instrument which classified them according to program and evalu- ation content categories; the studies were also scored according to quality criteria. The studies were then screened for appropriate in- clusion in selected data synthesis procedures. Conclusions In presenting the conclusions, reference will be made to the ques- tions posed in the statement of purpose for the study. 1. What is the state of the art of program delivery and evaluation of community nutrition education for non-institutionalized, non-patient adults in the United States? The state of the art is well identified by the results described in the previous chapter. Most studies were formative, most used pre experi- mental designs, and most reported positive changes in behavior. When ex- amined from a quality of evaluation design standpoint, the studies for the most part lack scientific rigor, but from a descriptive programmatic sense, they are useful in developing a basis for understanding the methods that have been usedlKJimprove nutrition knowledge, behavior and attitudes and their results. It is readily apparent from the studies that nutri- tion education has been considered more of a service for low income peo- ple than as a Health Promotion strategy for the general public, and that the public health community has.demonstrated less leadership for commue. nity'nUtrition education programs than other involved fields. 2. Are any of the existing meta-evaluation (secondary evaluation) meth- ods or instruments appropriate for classification and assessment of pub- lic health studies for the purpose of planning programs and evaluation methodologies and screening for study data appropriate for inclusion in data synthesis? 71 Meta-evaluation techniques, checklists and review instruments de- veloped by Bracht (1973) Bertram and Brooks Bertram (1976), Green and Figa- Talamanca (1974), Smith and Murry (1974), Sanders and Nafziger (1976), Stoke (1969, 1970), Stufflebeam (1971, 1974), Wandt (1974), Worthen (1973) and Worthen and Sanders (1973) were reviewed and none of them was able to classify both the content and quality of studies along the parameters of interest of this study. Many of the reviewed methods were intended for review based upon lengthy reports, documentation, and interaction with program evaluation staff. Most required more information than is generally available from a journal article and the focus of many were very decision oriented in relation to the context of specific programs. None examined all variables of content and quality as required for this study and none'screened studies according to the assumptions and require? ments of data synthesis techniques. 3. If existing methods or instruments are not appropriate, can an in- strument be developed in a format than can be easily understood by, and useful to, those who plan, administer and evaluate Health Promotion programs? Each of the checklists, standards sets, and guidelines that were reviewed and found to be inappropriate or incomplete helped to identify important factors that were included in a Classification Instrument, tailor made for this study. The review of instruments, checklists and guidelines helped to identify more clearly the purpose of the Instru- ment - to assess and classify the content and quality of intervention strategies, evaluation methods and results in order to identify compar- able and compatible studies for data synthesis of programs' effects (out- comes) and to aggregate program experience concerning other important levels of program planning and evaluation (i.e. structure, process and impact). The developed Classification Instrument was useful for the 72 purpose of this study, but the usefulness to those who plan, evaluate and administer Health Promotion programs seemed limited in the format used for this study because no definitions or instructions were provided. Further development of the Classification Instrument will be discussed later in this chapter. 4. Are the selected data synthesis techniques appropriate for use with the samples of Health Promotion programs? The assumptions and data requirements for each of the four data synthesis techniques were specified and the sample of studies was exam- ined in light of these. It was found that no two studies in the sample satisfied the assumptions and data requirements of the synthesis techni- ques. Each technique's assumptions and data requirements will be sum- marized along with appropriate use with this sample of studies. The Cluster Approach (Light & Smith, 1971) The cluster approach has the following requirements: 1. Access to original data of each study 2. Review of quality of studies. "The studies accepted for further an- alysis would be only those which meet the following three standards: a) All subjects in the study must have been selected from a known and precisely definable population. b) A study's dependent variable and those independent variables which are measured must be measured in the same way as, or in a way subject to a conversion into, those employed in the rest of the studies. c) Overall, the instrumentation and qualityof the experimental work must be generally comparable to that in all the rest of the studies." 73 3. Examples of apppropriate research questions using the Cluster method. a) Of the specific approaches being analyzed, which offers the greatest improvement in (specific primary dependent outcome vari- able)? b) Is the relationship between pre and posttest score identical in all clusters? c) Are the variations in the differences between (two subset of subjects, i.e. men-women; young-old) on a criterion variable from cluster to cluster associated with the type of program? No published report provided access to original data and there- fore this technique was inappropriate for use with this set of studies. InvestigatingyInteractions - The Blocking Technigue (Rosenthal, 1978). The requirements for this technique are: 1. The procedure is appropriate for studies that compare the same two programs, but not necessarily on similar groups of subjects. 2. Data requirements for each study: a) mean b) sample size c) standard deviation 3. Moderate computational difficulty: "The 'blocking technique' involves comparing the outcomes by casting the results into an overall analysis of variance ANOVA, with studies regarded as a blocking variable...mean squares can be constructed and a two-way ANOVA (treatments by studies) can be performed...Studying the main effects of treatments from the ANOVA provide an average measure of their differential effective- ness" Phillimer & Light (1980) p. 74 Since no two studies compared the same two programs, the Blocking Tech- nique was not appropriate for use on any of this set of studies. Conducting,a Combined Significance Test (Rosenthal, 1978). The requirements of this technique are: 1. Computational simplicity: "Example for combining z scores "If two groups are compared in each study, there is a z score associated with each reported p value. The z's are added across studies, and their sum is divided by the square root of the number of studies that are combined. The pro— bability value associated with the resulting overall score provides the level of significance for the combined statistical test." Pillimer and Light (1971) p. 2. No review of quality of potentially combined studies is recommended. 3. Requirements: for each study the following data is necessary a) sample size b) value of the test statistic (+, z, of F) 4. Studies must have tested the same directional hypothesis. 5. Example of an appropriate research question using the combined signi- ficance test method: What grand conclusion can be drawn concerning the comparability of two treatments? 6. Preferred conditions: a) the separate studies are independent, and b) random samples were used in estimating group differences. No studies used a z or F test statistic, but six studies reported the t test statistic. Of those, five reported a sample size, and only one of those specified a hypothesis. The Combined Significance Test is also not applicable to any groupings of the studies. 75 Computing an Average Effect Size (Glass, 1977). The requirements for this technique are: 1. Low informational requirements: a) group means and b) control group standard deviations 2. Few formal assumptions a) "Mean scores should be used to characterize a data set only after studying the distribution of the data, to determine if the mean is in fact a good descriptor." b) "...when the distribution of outcomes is unusual, procedures that focus on variation may prove more useful." c) same type programs must be compared. 3. Example of an appropriate research question using the average effect size method: On the average, is the (specific broad type) of program beneficial? Four studies reported means and standard deviations. Only one study reported means and standard deviations for a comparison group, and the means and standard deviations were all for different variables: in one study mean caloric intake, cholesteral levels, weight and satur- ated and polyunsaturated fats levels were reported. In another study, mean scores on pre and post nutrition workshop test scores were reported. Another reported mean food energy, protein, calcium, Vitamin A and Thia- min levels of diets, and the final study reporting means concerned weight loss. The Average Effect Size technique was also found not to be appli- cable to any combination of the studies in the sample. 76 In order to synthesize outcomes several similar studies must exist and at this time studies of evaluation of outcomes of community nutri- tion education programs vary greatly by type of program, target popula- tion and outcome variable. Therefore, at this time, data synthesis techniques are not appropriate for the studies which represent the focus of this project. Because of the scope and focus of this project, limitations must be placed on the generalizability of this conclusion. Data synthe- sis techniques, although not found appropriate for these selected studies, may be appropriate for use with other types of Health Promo- tion programs. Testing the applicability of the selected data synthe- sis techniques for this project, however, has led this researcher to view data synthesis techniques more cautiously than before having worked with them. A surface familiarity with the techniqeus from the writings of their creators and proponents had raised the expectations of this re- searcher that the techniques could prove to be very useful secondary data analysis techniques. 77 After going beyond surface familiarity and examining the under- lying assumptions and requirements of the techniques, this researcher takes a very conservative viewpoint and suggests that their use be very cautiously considered. The two points that raise the most concern to this researcher are: the lack of review of and concern for the quality of studies that contribute data for the synthesis and the questionable mix of data concerning diverse outcomes. If data from poorly conducted studies are used together with data from well designed and conducted studies to form synthetic estimates of program effects, the accuracy of those estimates surely seems question- able. 'This issue has been the f0cus of debate between Glass, Smith, LightLand Pillimer who support data synthesis and critics such as Esyneck. The second point, the mixing of scores from different kinds of out- come variables has also been debated. The proponents of the synthesis methods view the diversity as a favorable factor which can only further the understanding of program effects; the critics disagree. Again, the experience gained from this project led this researcher to join the side of the critics. Use of diverse measures seems to be inappropriate - an example of comparison of apples and oranges. Upon careful review, the authors have not provided to this researcher's satisfaction, a rationale for their position, and an attempt to implement their suggested use of diverse outcome measures for this study seems conceptually impossible. 5. Does the type of evaluation design effect the usefulness of the eval- uation findings -- that is, do any systematic evaluation efforts, des- pite level of experimentation (i.e.. pre-experimental, quasi-experimental or thru experimental)generally yield useful results? 78 As reported in Chapter V, when the quality of studies scores were broken down by groupings of quasi-experimental, pre-experimental using a systematic design and pre-experimental using an unsystematic design, the quality scores for the quasi experimental group and systematic pre-experi— mental group were quite similar, with the unsystematic pre—experimental group having very dissimilar, lower scores. These findings seem to sug- gest that systematic evaluation efforts, despite level of experimentation, do yield useful results. Although there is a trend throughout the pro- gram evaluation field to try to make evaluations more "controlled" (i.e. encourage random assignment and other true experimental approaches), this researcher takes the view that given the limited state of the art in the in the documentation of various strategies and their effects (in the narrow field under study in this project) there is also a need to encourage systematic.program evaluation at all levels. If the goals of Health Promotion are to be achieved in their short time frame, rapid trials of new methods and materials will need to occur. For these newer untested and small scale studies, §y§tematic pre-experimen- tal designs could yield useful results. For programs that have been more fully developed and for which more evaluation resources are available, quasi and true experimental methods would be the preferred methods because of the fewer threats to internal and external validity they face as well as the stronger inferences that can be drawn from them. 6. Is there a minimum set of guidelines that can be suggested (based upon this research) for use in planning and evaluating Health Promotion programs? Two general minimum guidelines have emerged from this study and are: (1) "don't reinvent the wheel" - a great deal of research and pro- gram data currently exists upon which further program experimenta- tion and development can be built. Programs should spend more time 79 and make investments in learning from the experiences of others and integrating these into their planning than now currently exists. (2) fit the evaluation and the program to the resources and mil- leu of each specific program. Each of the two general guidelines will be discussed. The first guideline - discovering and building upon what is known, brings this discussion back to research question three which concerned the feasibility of a Classification Instrument for assessing the content, results and quality of studies. The usefulness of the Classification Instrument developed for this project combined with the identified limit- ations of the selected data synthesis techniques lead this researcher to further develop the classification idea to a form which suggests to have potential as a tool for planning Health Promotion Programs and their ap- propriate evaluations as well as helping to further the understanding of Health Promotion as a multi-strategy, multi-focus and interdisciplinary entity. While abstracting the sample of studies for this project using the Classification Instrument it became readily apparent that the Instrument was providing data that was dynamic in form. Data synthesis techniques yield only figures and (aside from the previously discussed questionable estimates represented by the figures) those figures represent program ef- fects based on a given set of studies, that can be combined in only one way. As more studies become available, the figures have to be recalcu- lated. Data synthesis techniques are static, one dimensional techniques. Systematic classification of studies, however, could provide a dy- namic, multi-dimensional data system. Once a study is classified accord- ing to numerous categories those can be combined in many ways with other 80 studies many times over. As new studies are added to a classification system, the previously stored studies could interact with the new data in an additiVe way, since the classification method does not synthesize by reducing each study to a numerical component of a statistical technique, but rather would ag- gregate studies according to multiple classification categories. To further develop the ideas about data classification, discussions were held with health promotion, evaluation research and computer spe- cialists. Further literature was reviewed and a decision was made to de- velop a "Classification Manual for Planning Health Promotion Programs and Evaluations." The manual was styled after the Patient Classification Mggggl (1972) edited by Jones and a reSult of team research efforts from faculty of four Universities including the M.S.U. College of Human Medi- cine. Having previously participated in research using the Patient Classification Manual, this researcher viewed the style of the Manual to be appropriate for the purpose and prospective audience of a Health Promotion Classification System. The manual is attached in Appendix F1. The Health Promotion Classification Manual is an attempt to develop aimethodology for classifying, according to multiple categories, the content, methods and quality and results of studies of Health Promotion/ Risk Reduction so that this data can be retrieved and used in de- veloping health promotion programs based upon existing evaluation re- search findings. The Manual offers program planners, administrators, evaluators and practioners a standardized, and inexpensive method- ology for organizing and retrieving information on programs via a var- iety of multiple categories. That is, if the planner of a smoking cessa- tion program wishes to review program experience to aid in developing 81 her/his program, this could be done using the Manual in a number of ways: 1. by review of literature in the thematic area (smoking cessa- tion); 2. by review of various program strategies that have been used with the intended target audience, from a variety of thematic areas; or 3. by review of evaluation designs that have been used from a variety of thematic areas and/or program strategies. Each type of review could be quickly and easily accessed using the Manual. The Manual is not an annotated bibliography; it does not contain summaries of studies from various thematic areas. Instead, it provides a framework -- a system —- for coding the content, methods, results and quality of programs (from journal articles, final reports, etc.). The Manual couldenable practitioners to extract, code and make available for organized and easy access, information from studies that Health Promo— tion specialists routinely read in professional journals. No statisti- cal ability is required by the Manual user. The Manual could be used via computer retrieval, or hand sorted using McBee data cards or stand- ard index cards. Like any retrieval system, it will only be able to "give out" data from that which has been fed in. The more studies class- ified and stored, the more useful this multiple category retreival system will be. Recause the Manual defines and uses standard terminology, a large "data bank" could be established through a cooperative effort of several programs sharing coded studies that they classify as a result of ongoing literature reading and review. In order to test out the clarity and potential usefulness of the Manual, it was submitted for review to three health professionals active 82 in Health Promotion programs. Each reviewer was mailed a letter of re- quest and explanation and a questionnaire (which appear in Appendix G) along with a copy of the Manual. Two of the reviewers have M.P.H. de- grees, one being an administrator and the other a program planner. The other has a B.A. degree and 20 years experience as a health educator. Each reviewer provided positive indications that a Classification Manual and system could be very useful to the Health Promotion field and only minor changes were suggested for clarifying the manual. Two reviewers suggested that the work involved in building up the data bank may be prob- lematic for some users, considering time and other constraints on pro- grams. One reviewer suggested that the "system" might most usefully be implemented in a university research setting. This researcher is encour- aged by the response of the reviewers, and believes that it is the most significant outcome of this study. To illustrate the usefulness of the Manual, it seems useful at this point to return to the scenario presented in Chapter One. Assume that you are the person responsible for developing a Health Promotion program for adults. You want to develop the best program possible. You are committed to Health Promotion, which has been well described by Divore and Krueter (1980) as: The process of advocating health in order to en- hance the probabilit that personal (individual, family and community , private (professional and business), and public (federal, state and local government) support of positive health practice will become a societal norm. The process of advocating health may be conducted by a variety of modalities, including but not limited, health education. You understand the extent of need for program delivery if the wellness-goals (as identified by the 1979 Surgeon General) are to be achieved by 1990. (See Figure 1 for identification of the priority area.) 83 You agree with Turner for the need to understand the conceptual characteristics of a field; and that ques- tions concerning appropriate research foundations for Health Promotion programs are contingent upon a de- termination of the parameters of the field of Health Promotion. You recognize as Lauzon (1977) has stated that "Ef- fective health promotion programs for the future must be founded upon sound principles which recognize the multiplicative dynamics of health behavior." and you accept the "Epidemiologic Approach to Health Pro- motion" proposed by Lauzon as an appropriate schemata for this field. You recognize that in addition to the Lauzon model identification of target groups and activities, you can further contribute to development of successful Health Promotion programs for your target group (adults) by incorporating into your planning the infor- mation concerning adult learning theories and research findings. Because almost all host (or adult-targeted) Health Promotion strategies can be considered as non formal education, you also want to incorporate into your planning, experience from the fields of non-formal education. Because the literature concerning Health Promotion and related fields is so inclusive, you are faced with the enormous task of pulling together the most suc- cessful strategies from many thematic and program fields. You are not a researcher, but rather a practitioner, and you do not have vast technical research skills. The person in this senario will in all liklihood find that she/he is severely handicapped in meeting her/his responsibilities for develop- ing Health Promotion programs. In fact, without a systematic approach for analysis, classification and synthesis, she/he will probably find re- ported results from the literature to be equivocal. Consequently, the usefulness of the review in program planning efforts will be limited. Just asamodel or schema is required to understand the components and conceptual characteristics of Health Promotion, so too there is a need for a method by which previously conducted research about each component 84 can be extracted, analyzed, classified and synthesized to contribute "to a data bank" to further clarification of the Health Promotion knowledge base. It may be unrealistic to expect that the person in the scenario could collect, classify and set up a data bank system, but the person could certainly make use of such a system if it were available at a Health Systems Agency or regional University or a National Clearing House. If theses Options were not made available a person might have to capability to set up a cooperative data bank with other similar pro- grams using selected classification variables from the manual. A mini- mum basic data set could be agreed upon by the group and studies could be classified by only these variables. Using all or parts of the manual to classify previous evaluation studies by their content, methods, quality and results by whatever means and on limited or large scale seems to be an option that would improve the state of the art. As further research beyond this dissertation plans are underway to make minor revisions to the manual according to reviewer suggestions and to try to obtain sponsorship for further development of the manual, namely: to conduct an inter-rater reliability test, and to build up a minimum data bank and to test the usefulness in selected Health Promotion programs, and to test out alternative sponsorship for a cooperative "data bank" which could result from use of the Manual. It may be that the most efficient way to encourage use of the proposed system would be through "data banks" access- ible through "Regional Health Eeucation Centers" which are sponsored in part by Health Systems Agencies (HSA's) throughout the country. The second general guideline for Health Promotion practitioners that emerges from this dissertation concerns fitting the evaluation de- sign to the resources and milleu of the program. 85 One of the studies in the sample for this project was written by the respected researcher Virginia Li Wang (1977). For this particular study Wang used a post test only design and as arationale (almost an apology) for this design, she stated: "In the real world of programs, evaluation may have to accept certain limitations if it is to take place at all (p. 109). Many of the studies in the sample could not have been conducted if higher level experimental standards had been imposed upon them, Since the findings of most of these Studies are useful, program planners and evaluators must be cautious to nevertx>fail to conduct a sys- tematic evaluation because a true experiment is not possible. They should be equally as cautious,however, never to try to conduct a design that exceeds their resources, conditions or abilities. The following more specific guidelines are based in part upon glean- ings from this project combined with program evaluation experience of this researcher, and are based on the belief that the key elements to a suc- cessful program evaluation are negotiated planning, communication, docu- mentation and utilization. a) Careful planning by an interdisciplinary team about design is necessary. A design must be negotiated. The most rigorous design that can be carried out by available staff, staff abili- ties and resources should be agreed upon. A true experimental design with planned high level statistical analysis does not provide useful data if it is not correctly carried out. A design should be explicitly spelled out and agreed upon only if it can actually be implemented. A lower level design may be more useful to a program given its resources. b) Continuous communication among members of the interdiscipli- nary planning team must be maintained. The evaluator, 86 administrator and planner and practitioners must continue to be informed about the progression of the evaluation. c) The process of the evaluation should be documented by all in- volved partiesin order undocument any deviations from the ori- ginal design, as "red flags" for emerging problems and to pro- vide a record for replication of the design. d) If the above three points are achieved, they will hopefully enable utilization of results. If evaluation results are not used then they are meaningless. The two general guidelines that have been discussed as emerging from this dissertation can be summarized as follows: first, relevant past program experience should be reviewed by Health Promotion profes- sionals in order to plan more efficiently both their programs and the evalu- ation of those programs; and secondly, whatever is found in the litera- ture will have to be tailor made to some extent to accomodate each parti- cular program. Suggestions for Further Study Further testing of data synthesis techniques is suggested to deter- mine if they can be useful in the development of thematic areas of Health Promotion other than that selected for this study. Another area for further study suggested by this project is comparative analysis of the usefulness of results of data synthesis techniques as compared with re- sults of other methods of aggregating findings, such as the Classifi- cation Manual. ‘ Further study concerning the Classification Manual is also indica- ted, and as previously stated, the specific research needs are: a study of interrater reliability of the Manual; development of a small "data 87 bank" resulting from the Manual; pilot testing the usefulness of the "data bank" by selected programs; and testing out the Manual system in alternative settings, such as individual agencies and Regional Health Education Resource Centers. Reflections This dissertation research could not have been conducted without a conviction by the researcher that Health Promotion is possible and nec- essary for the nation's health. It is believed that the 1990 Health Pro- motion goals are realistic given appropriate leadership and support from health professionals all over the country and from the federal govern- ment. And there in lies the threat to the forward movement of Health Promotion. The national strategy and goals for health promotion were devel- oped under the Carter Administration. In 1979, in an article entitled "Nutrition in the Humphrey Tradition" Bray gave "...an authoritative at- tention and emphasis at DHEW." (p. 116) These included a national Nu- trition Coordinating Committee, new national leadership for nutrition education, including development of an inventory of nutrition education materials programs, mandated improved FDA and nutrition labeling, in- creased national nutritional status monitoring, review of dietary guide- lines and a commitment to improve international nutrition. Bray con- cluded his paper by commenting "We would welcome your help now and in the fu- ture as we attempt to make our programs in nu- trition education and information more effective, more available, and more responsive to the needs of the American people. Nutrition in the Humphrey tradition is alive and well in DHEW." (p. 121) Unfortunately, the Humphrey tradition is not alive and well in the Reagan Administration and the Health Promotion goals for 1990 may now be 88 avianathema Even without considering the change to conservative poli- tical leadership, the movement towards Health Promotion -- promoting healthy lifestyles -- has faced difficulty in the United States. A com- parison with Norway highlights the source of some of these difficulties. In 1975 Norway developed plans for a comprehensive national food and nutritional policy. Winikoff (1977) has compared the Norwegian ap- proach to that of the U.S. She has stated that the Norwegian proposal merits being called a "milestone" because "...of its comprehensive nature and because, al- though advocacy of a nutritional policy has a long history, the development of such a policy by government is relatively rare. In fact, in relation to the United States' efforts in this same area, there are at least three reasons why this document is remarkable: 1. The goals of the proposed nutrition and food policy are based on scientific understandings of nutrition/health relationships. Real numbers, based on nutritional knowledge, are the founda- tion of the proposal. This is a document which does not rely only on humanitarianism, morality, or pious hopes for the future. 2. The stated goals themselves are desired end points against which policy must be measured. These goals take primacy over the institution- al or functional arrangements of government struc- ture designed to deal with nutrition. 3. The national government takes itself ser- iously as an institution for setting overall policy directions. The government's responsi- bility is first to the wellbeing of its own citi- zens, and as far as possible, in congruence with the aspirations and needs of other citizens of the world." (p. 552) Winikoff concludes her discussion by a comparison of Norway and the United States: "...the nutritional goals of the two nations are not so different and neither are the prob- lems of diet. Norway has taken the first step in doing what the United States has often said it would like to do: attend to the nutritional and health needs of its citizens in congruence with international, national, and regional devel- opment goals, in a rational, organized and well 89 structured way. The United States, with more con- certed attention to the problem, could also fol- low along this general path. The struggle might be harder, but the rewards might be greater. At least, let us not say "It can't happen here" until we have tried." (p. 557) Since Winikoff made these statements the U.S. national Health Pro- motion Strategy has been defined and goals established. Informed indi- viduals can now not challenge the link between life style and health status and risks. The challenges are whether recognition of the rela- tionships and federal support in combination will be strong enough for- ces to lead to lifestyle changes. And so the question remains: What are the pro- spects for the future in health promotion? It seems that health promotion is indeed an idea whose time has come. The time is right, the soil is fertile. But there are large unmet needs. Solutions must be found to existing manpower needs. There must be an expanded prevention know- ledge base, including improved data collection and cost-benefit analyses. Research priorities must shift to provide increased emphasis on di- sease prevention research. Professional schools must place increased emphasis on teaching di- sease prevention. Financial incentives must be utilized to accomplish necessary prevention- oreinted projects and programs. The federal gov- ernment must provide leadership and coordination. The private sector must participate in a meaning- ful manner in the initiation and implementation of health promotion programs. Linkages must be established that will enhance the success of health promotion programs in all sectors of so- ciety. Despite these and other unmet needs, in my Opinion the prospects for the future will be bright if those of us who are preventive medi- cine practitioners take the lead in identifying and implementing initiatives that will meet these needs. Certainly it will not be easy. As Machiavelli, that sage observer of humanity, once said, "There is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things" Duane Block, 1980 (p.12) 90 Data classification and cautiously conducted data synthesis of evaluation research findings are promising initiatives that have poten- tial for enhancing the success of health promotion programs in increas- ing positive health behavior among Americans. APPENDIX A APPENDIX A Specific Nutrition Objectives for 1990 or earlier in the U.S. Improved Health Stutus: 1. Improvement in nutrition may yield reduced rates of infant mortality, cardiovascular disease, dental carries and possibly some cancers. Reduction by 50% of pregnant women with iron deficiency anemia. Elimination of growth retardation of infants and children caused by inadequate diets. Reduced Risk Factors: 1. The prevalence of significant overweight among adults should be decreased to 10% of men and l7% of women, without nutritional impairment. 50% of the overweight population should have adopted weight loss regimens, combining an appropriate balance of diet and physical activity. The mean serum cholesterol level of adults aged 18 - 74 should be at or below 200 mg/dl. The mean serum cholesterol level in children aged 1 to l4 should be at or below l5 mg/dl. The average daily sodium ingestion by adults should be reduced from l to 4 grams. The proportion of women who breastfeed their babies at hospital discharge should be increased to 75% and to 35% at six months of age. Increased Public/Professional Awareness: 1. The proportion of the population which is able to identify the principal dietary factors known or strongly suspected to be related to disease should exceed 75% for each of the following diseases: heart disease, high blood pressure, dental carries and cancer. 91 92 APPENDIX A (continued) 2. 70% of adults should be able to identify the major foods which are: low in fat content, low in sodium content, high in calories, good sources of fiber. 3. 90% of adults should understand that to lose weight people must either consume foods that contain fewer calories or increase physical activity--or both. Improved Services/Protection: l. The labels of all packaged foods should contain useful calorie and nutrient information to enable consumers to select diets that promote and protect good health. Similar information should be displayed where nonpackaged foods are obtained or purchased. 2. Sodium levels in processed food should be reduced by 20% from present levels. 3. The proportion of employees and school cafeteria managers who are aware of, and actively promote USDA/DHHS dietary guidelines should be greater than 50%. 4. All states should include nutrition education as part of required comprehensive school health education at elementary and secondary levels. 5. Virtually all routine health contacts with health professionals should include some element of nutrition education and nutrition counseling. Improved Surveillance/Evaluation System: 1. A comprehensive National nutrition status monitoring system should have the capability for detecting nutritional problems in special population groups, as well as for obtaining baseline data for decisions on National nutrition policies. Note: Promoting Health/Preventing Disease: Objectives for the Nation, U.S. Department ofFHealth and Human Services, Public Health Service, Fall l980, 75-76. APPENDIX B APPENDIX B Abstract No. We: Overall Score: SCREENING/CLASSIFICATION INSTRUMENT Authors: Title: m: Purpose of Program : Target Population: Educational Approach Used: Sub-Scores (___) Literature Review: Number of References Cited: (___) Purpose of Evaluation: Formative Summative Comments: (___) Evaluation Questions: Yes No Hypotheses: Yes No ( ) Variables: Independent: ( ) (___) Measure- Dependent: EEDLS Categories of Dependent thod(s) of Instrument(s) Reported Instrument variables variables Assessment Pretest Relia- Validity bility ( ) ( l J ( l ( 1 ( )j 1 ) Appropriateness of Instruments:( ) Quality of Instruments:( ) (___) Evaluation _ Specific Design Design Limitations Discussed: £5493: Pre-experimental: _Yes _No 4 ESubscores Quasi-experimental: .___ Yes .___ No True-experimental: __Yes _No Appropriateness of Design:(___) Quality of Design:(___) (___) Sampling: Discussed in article: Yes No Specifics: Subscores Sample size: __ _ ( ) Sample description: ____ ( ) Population description____ ____ ( ) Sampling Method: _ _ ( ) (’5447 Response Rate: ( ) Statistic“ Appropriateness of Seamus thod=1_), Quality of Sampling Method:( _) '__' .Apglygjgg Inferential Statistics: _Yes .___ No Descriptive Statis.:____Yes _No Test(s) Performed: Level(s) of Significance: Not Spepif)ied Limitation(s) of method(s) of analysis discussed: _Yes _No ( Appropriateness of Sataistical Analysis: (_ ) Quality of Statistical Analysis: ( _) Overall Reported Result: (___) Effects: Short Term Long’Term Variables Direction Type Interval Variables Direction Type Interva Effects Measurement Effects Measurement Not Applicable 93 APPENDIX C APPENDIX C Bibliography of Potential Data Set and Disposition Relative to Inclusion in Study and Reasons for Non-Inclusion Potentng_Subject Listing (Articles) Abbey, 0. E. and Godge, H. F. Nutrition Ideas in the supermarket. Journal of Nutrition Education, 8(3):129-130, July-Sept. 1976. Armstrong, J. Nutrition awareness in the supermarket. Nutrition News 35:2 (6) 1972. Barton, A. F.I.T. for action in nutrition. Nutrition News 36:3 (1) 1973. Bell, M. Reaching Mexican-American homemakers. Nutrition News 32:4 (14) 1969. Benzley, J. Selling nutrition in Salt Lake City. Journal_pf Home Econo- mics, 64 (6): 28-29, 1972. Boylon, B. Volunteers and education = better health. Nutrition News 35:11 (2) 1972. Bradsher, M.S., Extension education and food programs = better living. Extension Service Review, 37 (7): 6-7, 1966. Brent, C. TV commercials can teach nu- trition. Journal of Home Economics, (21-23) March 1974. Bufton, K.R., "Ask Kathy": an experiment in nutrition education. Extension Service Review, 40 (6): 12-13, 1969. Burtis, A. market. Nutrition education in the super- 94 Nutrition News, 33:4 (16) 1970. Disposition & Reason In study N/A Program description only In study N/A Subjects children N/A Program description only In study In study In study In study 10 11 95 Carruth, B.R. and Force, S.B., Cartoon approach to nutrition education. Journgl of Nutrition Education, 3: 57-59. 1971. Chamberlain, V.M., Oops! We'd better go shapping. What's New in Home Econo- mics, 33 (8): 83-85, 1969. Cobb, J.L. Traveling nutrition clinic. Nutrition News, 33:3 (13) 1970. Cook, F. Nutrition education via people to people. Journal of Nutrition Education, 1 (2):9-11, 1969. Cordaro, J.B., A curriculum for the nutri- tion programmer. American Journal of Clinical Nutritigp, 24:1352-3, November 1971. Darby, W.J., The renaissance of nutrition education. Nutrition Review, 35 (2): 33-8, Feb. 1977. Dodds, J., A new audience for nutrition education. Journal of Nutrition Edu- cation, 1(2):23-24, 1969. Echols, I.J., Comparative group approaches. Journal of American Dietetic Assoc., 59:460465, 1971. Egan, M.C., Working together in community nutrition. Journal of American Dietetic Assoc., 45:355-58, 1965. Evans, R.I. and Hall, Y., Social Psychology in changing eating behavior. Journal of American Dietetic Association, 72: 378-83, 1978. Fitzgibbons, J.J. and Garcia, P.A., TV, PSA's, nutrition and the elderly, Journal of Nutrition Education, 9(3): 114-118, July-Sept. 1977. Gaman, E. A WIC pilot program with nutri- tion education. Journal of Nutrition Education, 8(4):157-8, Oct-Dec 1976. Gassie, E.W. and Jones, J.H. Jr., Sustained behavior change. Journal of Nutri- tion Education, 4(1):19-22, Winter 1972. N/A Subjects high school students N/A Subjects high school students In study In study N/A Professional education N/A Professional education N/A Subjects hospitalized teenagers N/A Theoretical N/A Professional education N/A Theoretical In study N/A Medically based program In study 96 Glanz, K., Strategies for nutrition coun- N/A seling. Journal of American Dietetic Professional Association, 74(4):431-7, April 1979. education Goldberg, S.J., et. al., Use of health edu- N/A cation and attempted dietary changes Subjects to modify atherosclerotic risk factors: children a controlled trial. American Journal of Clinical Nutrition, 33(6):1272-8, June 1980? 12 Green, L.W., Wang, V.L. and prross, P.H. In study A three year, longitudianl study of the impact of Nutrition Aides on the knowledge, attitudes and practices of rural homemakers. American Journal of Public Health, 67(7): , July 1974. Gross, L.P. Can we influence behavior to N/A promote good nutrition? Bulletin of Essay New York Academy of Medicine, 47:613- 23, June 1971. 13 Groves, M.P. Nutrition fest - more than In study just fun. Extension Service Review, 43(3):12-13, 1972. 14 Gustafson, C. and Steiner, W. Looking for In study the answers, Journal of Nutrition Education, 1(2)?12-13, 1969. Harden, M. and Lamb, M.W., Bioassay - tool N/A for conceptual learning. Journal of Subject Nutritional Education, 1(4):13-15, University students 1970. Animal research Hay, C.P., Reaching urban women. Extension N/A Service Review, 32:59-61, 1961. General Extension Ed. 15 Herrick, K.L. Developing and evaluating In study audio-visual media for dietary educa- tion. Journal of American Dietetic Association, 73(6):660-2, Dec. 1978. 15 Henderson, S.E. Facts-not fads for the In study listenin public. Nutrition News, 36:3 (11? 1973. Hill, M.M. Helping people to help them- N/A selves - Extension aide program. Program Nutrition Prpgram News, March - planning discussion April 1970. 17 19 97 Hill, M.M. Nutrition education - an inte- gral part of consumer education, Nutrition Program News, May - June 1970. Hill, M.M. Tools for nutrition education - some examples. Nutrition Prpgram News, Nov. 1970 - Feb. 1971. Hill, M.M. and Page, L. The food guide - a tool for teaching nutrition, Nutrition Community1News, May-June 1963. Hinkle, M.M. Exhibit advances community nu- trition education. Journal of Ameri- can Dietetic Association, 49:512-13, Dec. 1966. Holmes, Z.A. et. al. Computer-simulated lab experiments in food science. 11 Evalua- tion of the model in use. Journal of American Dietetic Association, 76(5): 1474-6, May 1980. Hughes, M. Nutrition education, Can it work? Health Values, 2(3):119-123, May/June 1978. Huiatt, A. and Hockin, B.L. Nutrition programs for senior citizens. Journal of Home Economics, 63:683-4, 1971. Hyland, K.M. Nutrition broadcasting. Journal of Hgman Nutrition, 34(1):52-3, Feb 1980. 20 Jansen, G.R., Harper, J.M., Kendall, P., 21 Houts, M. and Frey, A.L. Menu evalu- ation - a nutrient approach for consu- mers. Journal of Nutrition Education 8(4) Oct-Dec 1977, 162-165. Nutrition education - what is Nutrition Review, 23:353-56, Johnson, O.C. the goal? 1965. Kelly, K.L. Evaluation of a group nutri- tion education approach to effective weight loss and control. American Journal of Public Health, 69(8):81 - 4; Aug. 1979. Knutson, A. and Newton M.E. Behavioral factors in nutrition education. Journal'of American Dietetic Assoc., 37:225-5, Sept. 1960. N/A Discussion of topic N/A Subjects children 1 N/A Description In study N/A Professional education college N/A Discussion only In study Not available at bindery In study N/A Professional education In study N/A discussion 23 98 Kornblueh, M. and Parke, H.C. Survey of the use of written recipes. Journal pj_American Dietetic Association, 47:113-15, Aug. 1965. Kornblueh, M. The cafeteria game. Nurs. Outlook, 15(2):47, 1967. Kragt, D. Volunteers in nutrition education, Journal of Nutrition Education, 2:110- 111. 1971. Larson, L.B., Massoth, D.M. and Chase, P.H. A Potpouri of nutrition education methods. Journal of Nutrition Education, 6(1):21-23, 1974. Leventhal, H. Changing attitudes and habits to reduce risk factors in chronic di- sease. American Journal of Cardiology, 31:571, 1973. Linch, P.F. New labels help sell nutrition. Journal of Home Economics, 64(9):28-29, 1972. Macoby, N, Farguar, J.W., Wood, P.D., and Alexander, J. Reducing the risk of cardio vascular disease: effects of a community based campaign on know- lege and behavior. Journal of Com- munity Health, 3(2):100-14, Winter 1977. Mandriota, R. et. al. Nutrition interven- tion strategies in the Multiple Risk Factor Intervention Trial (MRFIT) Journal of American Dietetic Assoc., 77(2):138-40. Aug. 1980. MacRenolds, J.P. Can teaching good nutri- tion be bad? Journal of Nutrition Educaton, 2:13-15, 1970. Marvosh, M. A 'fair' way to teach nutri- tion. Journal of Nutrition Education, 2:48-49, 1970. Marqusee, R.H. Bioenergetics: an ecological approach to nutrition education. Jour- nal of Nutrition Education, 2:147-148, 1971. McDaniel, J.M. Utilizing the nursing process model to teach nutrition and diet therapy. Journal of American Dietetic Association, 74(5):568-71, May 1979. N/A Survey only N/A Subjects children N/A ,Subjects children In study N/A Discussion only In study In study N/A Subjects patients N/A Discussion only N/A Professional education N/A Subjects children N/A Professional education 24 25 26 27 28 99 McDonald, S.C. and Owen, H.B. Nutrition education workshop. Journal of Nutri- tion Educatiop, 2:68-69, 1970. Medvid, E. Television in nutrition edu- cation. Journal of Home Economics, 58(3):167-70, March 1966. Mills, E.R. Applying learning theory in teaching. Journal of Nutrition Edu- cation, 4:106-107, 1972. Moore, M.L. When families must eat more for less. Nurs. Outlook, 14(4): 66-69, 1966. Mortnedt, M.M. EFNED - homemaker similar- iteis. Journal Extension (4-6) March/ April 1975} Murphy, M.J., Smicklas - Wright, H; Heasley, D.K. and Hamilton, L.W. Impact of EFNEP on nutrition related practices. Journal of American Dietetic Assoc. 76:570-4, June 1980. Napier, C. Better nutrition - economically. Extension Service Review, 39(3-4):3-5, 1968. Niehoff, A. Changing food habits. Journal of Nutrition Education, 10-11, 1969. Niemeyer, K.A. Nutrition education is behav- ioral change. Journal of Nutrition Education, 3:32-33, 1971. Oliver, M. 'Program Aides' for work with low income families. Part IV. Pilot study in Alabama, Journal of American Dietetic Association, 50(483-486) 1967. Oomstead, A. How to teach food buymanship. What's New ingome Economics, 32(3): 24-26, 54, 1968. Polcovits, J. Nutrition for older Ameri- cans. Journal of Americgg Dietetic Assogjgtion, 58:17-21, 1971. Penner, H.W. Have you tried teaching foods via TV? What's New in Home Economics, 35:33-34, Nov/Dec, 1971. N/A Professional education In study N/A Subjects college students N/A discussion only In study In study In study N/A Discussion only N/A Subjects patients In study N/A Subjects students N/A Discussion only N/A Subjects College students 29 3O 31 32 33 34 35 36 100 Perry, V.T. Food fun, fairs, facts. Hospital 46(19):92-95, Oct 1, 1972. Piper, G. Planning new community nutri- tion services. Journal of American Qjetetic Association, 44:461-4, June 1964} Podell, R.N., Gary, L.R., Keller, K. and Mulvihill, M. The public seminar as a nutrition education approach. Journal of American Dietetic Assoc., 67:460-3, 1975. Poolton, M.A. Predicting applicability of nutrition education. Journal of Nutrition Education, 4:110-113, 1972. Pichard, K. and Hall, M.R. Attitudes of aides and clients in the expanded nu- trition program. Journal of Home Economics, 63:545-48, 1971. Prouix, J.M. Nutrition outreach for migrant workers. Ramsey, C. and Cloyd, M. An evaluation of EFNEP's side effects. Journal Exten- sion (11-20) May/June 1975. Remmell, P.S., et. al. A dietary program to lower serum cholesterol, Journal of American Dietetic Association, 54:13- 19, 1969. Ritchie, J.A. Learning better nutrition. A second study of approaches and techni- ques. FAO Food Nutrition Service, 19-20:1-264. Schild, D.T. A converted bus takes ENEP to the people. Journal of Nutrition Education, 1(3)?22-23, 1970. Shefchik, M. housin projects. 36:1(3? 1973. A nutrition program in Nutrition News, Shipman, J.A. and McCannon, N.R., Urban- ites must be approached through re- cognized information sources. Journal of Home Economics, 56:744-47, 1964. Short, S.H. et. al., Development and utili- zation of a self-instruction labora- tory. Journal of Home Economics, 61: 40-44, 1969. Nutrition News, 35:2(7) 1972. In study N/A Discussion only In study N/A . Discussion only In study In study In study In study N/A Initial study In study In study N/A Discussion only N/A Subjects college students 37 38 39 40 101 Shortridge, R.C. Learner success or failure. Journal of Nutrition Education, 8(1): 18-20, Jan-March 1976. Sipple, H.L. Combating nutrition informa- tion through coordinated program. American Journal of Public Health, 54: 823-826, 1964. Sliepcevich, E.M. and Croswell, W.H. A con- ceptual approach to health education: implications for nutrition education. American Journal of Public Health, 58: 684-692, 1968. Spindler, E.B. Home economics aide reach poor families. What's New in Home Economics, 31(1):31-33, 1967. Spindler, E.B. , Jacobson, M.E. and Russell, C.B. Action programs to improve nu- tition. Jourggl of Home Economics, 611 635-639, 1969. Spitze, H.T. Teaching nutrition to the neediest. Journal of the Coopera- tive Extension, 7(2):95-103, 1969. Spitze, H.T. Games that teach. Journal of Home Economics, 64(4):8-12, 1972. Spitze, H.T. Innovative techniques for teaching nutrition. Journal of Nutrition Education, 2:156-159, 1971. Stare, F.J., Meyers, M.L., and McCann, M.B. Nutrition education via the public press. Journal of American Dietetic Association, 39:124-5, August 1961. Stern, M.P., Farquhar, J.Q., Maccoby, W. and Russell, S.H. Results of a two- year health education campaign on dietary behavior - the Stanford three community study. Circulation, 54(5): 826-33, Nov. 1976. Sullivan, H.J., et. al. Development of a nutrition education program for home- makers. Journal of Nutrition Educa- tion, 8(3):118-121, July-Sept. 1976. Taylor, H.R. Mass Media and learning. Exten- tion Service Review, 37(6):6-7, 1966. N/A Discussion only N/A Discussion only ‘ N/A Discussion only In study In study N/A Discussion only N/A Subjects students N/A Description of teaching methods In study N/A Same data as # In study N/A General Home Ec. Education 102 Todhunter, E.N. Approaches to nutrition education. Journal of Nutrition Education, 1(1):8-10, 1969. Tonon, M. Models for educational inter- ventions in malnourished popula- tions. American Journal of Clinical Nutrition, 31(12):2279-2283, Dec. 1978. 41 Ugelow, E.I. Mobilizing the potential of home economics for low income families. Journal of Home Economics, 57:648-650, 1965. Vargas, J.S. Teaching as changing behavior. Journal of American Dietetic Assoc., 58:512-515, Jan. 1971. Waggener, E. Action on Serena Street. Nutrition News, 33(3):12, 1970. 42 Wagner, M.G., Huyck, M.C. and Hinkle, M.M. Evaluation of the Dial-a-Dietician program II. Impact of the program on the community. Journal of Ameri- can Dietetic Association, 47(5)385- 390, November 1965. Wagner, M.G. Answers to order. Nursing Outlook, 12(10):45-47, 1964. Wagner, M.G. Dial-A-Dietician: A commu- nity nutrition education program. American Journal of Clinical Nutri- tion, 18:60-67, 1966. 43 Wang, V.L. Changing nutritional behavior by aides in two programs. Journal of Nutrition Education, 9(3):107-113, July-Sept. 1977. 44 Wang, Y. L. and Ephross, P. H. EFNEP Evalu- ated. Journal of Nutrition Ed. Spring:148-152. 1971. 45 Zanelli, V. Better nutrition for a Portland community. Nutrition News, 32(2):6 1969 46 Zifferblatt, S.M., Wilbur, C.S., and Pinsky, J.L. Changing cafeteria eating habits. Journal of American Dietetic Association, 76:15-20, Jan. 1980. N/A theoretical N/A international In study N/A Discussion only sN/A Discussion only In study N/A Repetitive of above study N/A Repetitive of above study In study In study In study In study 103 Zirdle, V. Eat right - you're on candid N/A camera. Etension Service Review Subject 43(6):3, June 1972. teenagers APPENDIX D 104 now». n 52.2.3 account mucus icocHs «soucH .oucoHvao gssHvue m.>h voucaocc< euHs 38H Hues; a osHmaHo. o» m :oHuauHcaesou an: amuvHao woo». oco=3A=a -com uHHnam cane: .coHH coHucuaco :oHH HeNmHH .030 a on can >h. gov mumoaaoz o>HquE:m mmu <\z a nose: Hz conH>mHop iouavo neH -chac mchm «coco muHuHuoesou =.masaum coo» co» mumoacoc mHHmH> use: vac» a too; a 49.3... 88 S a 33883 H.328 9:2. muHHHEuH be H a. oHvoc .monHu masoum ouH>com veep e oocoguv mvcouoc ouo vac .onHanu woo» a too» a coHuch moon» gouacou a Nonnumow cm: a mouHo .vo .mocasuoca uauocoe How: is: cocoon. HoomHH we on: ummuacucH -zmcou HascoucH o>HNNsE:m mug .chaz msoHca> Hz .mommnHu meHHHeau soon» a» cogmvacn » coco u HHHEom cHugu coco—Hsu saw: H» ouwwHou Boga vomaao mcoxasaso: was» mc—zuoEOm cauwcms< ucoHuch ewecaoH us: has» Hum» amen movHe HmommoHuH :oHu :aono: -3: a macaw HmomHH n nag» vmcuHuou «No. a men Ho>c=m o>HuueLom muu amzum Hz Hz -uacumcH usage ueeucH er mcngaub. HHom can» -cH uHcoHou can—ence; cocoa in. a mHuue «Loam c. maoHuchac .mmue naoco ocpcanoca a 3.3»... coo» 58 DEE-So . ..8 9.33:... N chsu cHagu axoox c .mucaguocm museum mo .ochcoHa u acoHHaHuuLn -uoam cussmcou Hogans gasses .mu< ocoumcH cHagu H . unococozo HNanH -ao mLmE=m=ou= uoHHuHHomca o>Huussam -coaam .cuaz neonam .mcao» N .mu¢ oHvo: meannesm econ coHchuaz . acecumsc< guooa .va a» guooa cH cam coxo» muao oHnooa o:— 1coa 0: use .aHaau -uca; No .02. >< a -uuouuua we HonH. >< cog: uaovca; mcHanon Huozm HH :oHuuocuucH mmoco>Huuoe «agave H ago» o» uoaaoum oanoq we .oz- Hoxcme Hegemcua ocean H 1 two o>HuoHoc a uHaooa we u ummgmH: - coHua>cwmoc o>Huusssm Hz .cuzz -cuasm canoe H .suooa um mcoaaozm coon «woman a» aona< mHHamoz coHuaaHo>m :oHuoaHN>u camcoam :oHHuacumcH mcHuaom Eggnog; new: coHuoHaaom sucmocu can» .oz mo vogue: No onxh ma gouacnn< «coca» mo omoacaa co=u=< Huaum guanoH HocoHuouaum 105 ceH>egon mcHuoo ce Neowwo oHaoecem -aae o: a seam—zocx meucaem veew .eceea :csu muoHe cHogu we auHHeee Heewcuec on» :e uuewwe e: «an HHauoc veew .cg vN vacuum: >h mucus o>eceeH e» ope m. ANHm ecozzxce -euceecca eeH> meoNHHHu -eoe con—e one HNNNHH oH vouceeoc choeHu -coucH amen a one e>HHuasem -cm>H:= <\z a ease: -cem uHHoee ceHcem mcH>HH icaeuce eu...: oHucac a 38: e Z remixes .8 5:752. .8233: mueew . meeuchae venue—om emmgucee ueew we aces .omogucee «see we muceeoc muHangm .muee cage—ng :eHuchec a HuHeessee eeew Hexeuacoeem a eeew luau; .mceHuucum saw: meHH eeew e» noueHec we em: a cewuuuH AuHueaEeo we on: a meeaHe mcH lease: ocwm: -HENw mseucH HHuoocHu He: -Hee ace: .mHHHa :eHuueHHee .mHHHa -ueae HuHc mmcHueae eaecm zeH .zeoec meoHaece :e oeH o veew cuHHusm as. ueew msoanmsez we movHu -aseeu .ceHuoecumcH e» use; we -xceza a cewuau HmomHv -uceeuc meemeaee: muceeec HchewcH o>Humscew mmu emzwu a mwee: Hz one e» ace meoHuENEez -euo :eHuchez xeeu so>Hu -uenae Heavens :eHuua -eHeeam...a=H e>Hu -coucH Hocemcee -Homceeu ceHH ceHuoeHuHucoe -aucomeceec «Heenece a mHeHwoeH -checs a Ease nique eeew omeococH Hmsz m< a ucheHu mchaeg .mecesuecn -ece usage veew a 838:: 3:3... 88 3:2 -75.... .3 53 E .323 a... we 5:32;: use; area" veew :H we an: muceeez we .ueuo -Haowu a amHa mceHunueH Huegm ocev anu oH> ace—u meHHHsow ozone mace HcNoHH umuceucH omaocucH unseen; HuELewcH e>HuNscew a mmo -eceuo use: Hecaceu Hz ooucomoce such usage. :eH van—mac acewcH naeu «cox—w mouH>com .ceuueHmze: we on: we use» .mxHaH zuweaaaeu aceuecoee -xo a macuaeeu .mceeu .mce ocean an aceEume> Hocemcee Hchew noses: Hosea: menu» cH -geuHx anon :H mceeeegm ceHunacewcH N .c. HoHucecww oHH:3 -cH .xea :eHumom neoeem umHe -cueem cew aceHuacumces ueew ceHuchec Ha HouuHH usage: a uocoeHmceu imam secw "ceuueH o>Huu3cew ee-eu -eceeo use: ee-eu ucHeeco -me .xmoe .eu ee-eu -cecem oeH>ece “Hucem mouH>com .eweH casemceu ceHquCewcH no: echcaHe New :eHuchec a usecH ceHuaacew moucaemoc we «Nassau Huegm HH uoeHuoc zuH: mceeeegm ceHNuacewcH -cH Hewom: .emceem on: a Huaaeouw uuHeeceuo use: -coeam mcHuoHHea ueew cone: :eHHHcaac He HmaaHv o in; uHHnee u>HuHmee cuseumau HaacewcH o>HNa5cew mug ceHmcouxu zuHu coccH ”see: c .zaeea .eweH useecH 3eH -cecem NeH>eee ceawem muHemoa ceHueaHa>u ceHuoaHa>u cemeeem :eHuuacumcH ecHuuom anemone new: ceHuaHaeeN Issues; Leo» .ez we vogue: we «ex» we suuecee< poecaw we omeecee cegua< xenon sumcoH HoceHueuauu Q x3592 1(16 EH. acu wwe «H necoece cow ..uwHaae acu we meaac cuHaa; ace uaaa eu a9.) ueew a>Hmcawwe cc mucus -co ceHuchec . cewuaum ce uuee mcHucee eHocc a -auceeccc ac.) vceem xHch neH .mcwmeacucH cam- .uemm< vac ow» m.acacu -cam uchae uHHcae -HwHucaHUW. HNNNHH mH anceemac cacaumHH HNELewcH a>wucscew uHuauaHo caHuHuaHo acacxac< mcucee o eweaz Hecacaa aew>ece ew cemcaeca: aauuHe cmzwu eeHa>ac useo .em masque veew eu caeHac .mveew cewuwcuez maHHHscw we an: a acem we an: cH muweeac aucama mcaxaaaaec aeeucH mcerc ueew HmomHH eH amaacucH cauceeaa HalcewcH a>Hucscew -caucH eacwach ease: Hz mcewuacumcesao er mzewuwcuez cemwcumeu mucee -HuHucae an ceHuucmeuem- eacu am: eu acaauceHe> sec a maHucamc caHNEaeec a muwcaaseu uaea mauH>cam macaw. maewa omzwm maHesam a -a aces :ecx- cacue a soceece maHocauo mHoceHmmawece caucau aHom neew aHeeac cuHoac cauuac mH veew HaucasaHeeea muceeac cacue NH maHocamu auwc .mceHuccumceEac aaeocH cew veew teem HNNNHH we an: cH mamoacecH HuggewcH a>Huacsem a mmu mceHcc> uaeseu nae H mcHHcHech zeH uoa eu ccaaH- ma>eco sac—3 H8 -HcHI we ac. a» can :.cecu cw mcaxaaaeec use» .a: .ca cc me \x muwmw> aeec ecH: acaccoaH mume> use: we -cwucee .uecuceu we a: auwuecce a «maca>Huuawwa a» umH cauwu macmH ceHuacumceeae cac umeu cH mcccu HvaHH :cHsHu ucaaa>ececH \: cewucueca -vHch cqu -ac ochchwn «meccem NH «Nagasaaec ce aeHa maa>c=m macwa ace eu ace «cauaeaeec -Hua we ucwee a ace: acu we ueecw acu.... umau umee a ace a>HuacE=m mmu emzwm mace: mama» m HcecH>HccH aeeucH 3e: acu mH uccza .caacw acacceaH an aeHuuace a muchac caxHu mcewuccumceeae cac ceH>acac a coHs.=aH N aeeaH N: coHuauaua -uHng guH: .aauaonca HNHNHH izecx cH maaccce uxaH>caucH macw< ace eu ace mcaxcaaeec ceHuch macea a HH a>HuHmee acem umau umee a ace a>Huaccew mmu euzwu mace: . mxaaz m HaacH>HecH aseecH zeH -ac a>ecesH anmam muHemax ceHuoeHa>m ceHuaeHo>u cemceem cewuuecumcH ucwuuam assuage can: cewuaHeeec seaweed Lea» .ez we cecuaz we aux» we cuaecee< uamcup we ameecec cecu=< xveum cuecaH Hccewuooeem Q .3332 7 nu ii maHuH>Huee a xuw—waowu -em .ceHuch mceHuacumcec 1:: we Enamece eHcm -ac a maHesam uaucchceieu a mucus muceeHeHu -cemceem ueew .maeHeac cw ucaea>He> vac Houeuuacc- vane ac mucae Hecaec HuHc mceHmmcemwe mcaNHuHu -cH= ac cewu HHNoHH mH «c0339. «lb—won caou pgows wigs—Eon. 1.3.2; amt—3:090 9:0: 13:55 9:3» n Hus-Low:— Lowcam 4.5:: «2:95 .333: maHuH>Huue ancz cuce: meeecm no ecHuaa a chaeHa cew aceu ceHmmaumHv- mcereu wanna; -uaH uec . uaucawce uchcaHe- . HHchHeec «mauece meecuae mxaH>caucH :cas- -euav ceHuecee mmaaeem umefi .euaHe umau umee a aauuHceeu- mcaNHuHu -aae a acHasc Heueu cHacu we cauao age a aceHuo> caucau .mcaHHw- ceHcam ecw ueew .ceHu icocu. uHu no ca>ece -camce a mucae caucau Huwc maceuuaH- ->HH xHuca -chec ueecc HNNQHH NH .5 332 ocuucu -18 $585 alumnae“. quchc—eu ucheHuchez -838 3.893 333:! .3332 .va 838...... 3.22. .ua ocuccwu -ceu anaceeeca- ceHuch 1:: a mueew we echcoumcaec: m—ceo a mmacacoza we ceHuauchac ameacucH- acu aucuHccH eu mucaneeu Lassa muuow cuHx ceHu a>cam uueex mucas -ceu cauHuHHem .eemm< cewuuecum uchee -aficewcwmws a HoonH NH taco vauwuwpemces 1:: HascewcH a>Hqucew uHuauaHo -cHuwHaw Scams: mcweace uwc—cxu Hacacau veew uaaeeuu achH: maeeHccoau mcH cau -cHocu ceH>ecac -cau cewu a N a H HN uawe HacaumaH locumceeao mamas—u -ecu :eH .ucw a cuceam geese a H HN vaucaeucm er mac. -az Ham muaHHeec a we maHeHu -vcww ce cameo vaNH ima> ceeHm HmcHea cev ceHuuacum -cwce acu cuwx -ceuceac aaceece acHaccewumaaa a ucaa: cewmmam can -cH wHam HH uHHcee auccvceuea cw HoNoHH oH a mueauceu catz umau umee a ace a>Huaccew H.uez ceHuwuawo Hz coax H muecuaa c Hecacau muawc cauH< uoHaca: muHamax cewuoch>u cewuaeHo>N cemceem cewueecumcH mcHuuam saceece can: ceHucHeeeN necmecc Lea» .ez we cecua: we aeaw we cacewee< uaeccw we ameecac cecu=< Heeum cumcaH Hocewuaueuu a:282 ..chmuaeuem HHHaHu -aema ac eu va>ecc awcas mace acu ucacaHeeem eu ceHu -uecumcw a>HmcaucH ...amcamwc 108 ccuaumc> ewccee we mcaucau owuac wuHac 3m?- Lpflzu @0300.» OH Auwcg a cowuuawum mmaov. no em ceH>ocac cwacu -eeu a men -cw a>wmcaucw -He caHeema) aecucu eu sec cccaH mumau a eczeHee ces cew>ocacu -ewccou cow HNNmHH eu rec cccaH eu HauHcHHa a mumau caeaemxac macceece mceueow uch cacccxaH< cheae eHac cue... umee N a ace us mHacewm ewcac .>w cewuuzuac Hmwc we amcaHzecx a tee: MN asacoece cewueeeac cew>acac a aucaH Auwm -mawece maHuHcas oweae HuHex- omimm muccewewucae .cueecaw lam—c ewcac mmcE...= izecx we Ha>c=m a>wuaeeem -ca>Hc= cuHaaz -eee m mcca» N aHewuHaz muHee< amaacucw e» .xneuuu: ....meeca -ece aucaumwm -ma ceew cw .HHecca camaacu -cw a .ca cewu -chac genoccu ..mmaueem anew cuwx maHucce .eee uaaccu uac mcecuas cacue coca ocwumcu a mcaxcsaeec acu we mcu HvaHH mumaeuem ceHuceHo>a mucee vecc .cm muaxca: mcewuocumceeac mcaxcez ccaw uaum HoceHuch amacu a>Hueancem.... ac. -ac a acewua> ceHuchaz aew< -caeam a cewuuecumcw .a< cauHxa: -cc acu acw a cuemmax NN than auccece ace: ucamce HascewcH aawuccaem ucccmw: cewuchcz a aee: mcasscm N ace eu ace aceocw :eH .>ecesw.... .ceacaH Haea Hcou -ceuam as ea aa>cem umau umee Hacewm cuHaac cauaH .ewcH .va cuHcac a a .maccmaac -mawece mueeeccc -ac a Hecu owe. coHuchac . eaocouac-wHam cHHaac HH.c a «peaceaHm .uaae aUHHoa -cou unaca: uawc we muwwacac umee .c» H .ueaa auowce aceueaH -mo Hccewuce cance we .ca cewuwcu HowaHH HN uauceeac .mmeH .u: a.umec .acc a>ruac=3m auHHee .ecee<3 Hauwemez mxaaz NH .maceuuaH eeecu mcauwwwo 1:: aew>ecc xHHax Heecm ca Heecem -xcezv aaH>ac cow; cqu Haaumxw cewua .ua cacccHe cewu amecu a va -eHe>a Haas Heecam ceHc a cccu -chec a acwaw Heecem ceHc av cueece mmaH \3 chHmmee uec ueew a we am: cuwz amecu -ea ucawcu .meeece veew cucu a «secuece Ac cacw>Hc .3: a acwm: ceaau eu ueauceu acwcccewumaee umws mcaucau mecca accu aeme we eHHcee ca cewuwcu HNonH ON caucuwHecee aces < cacauchHEcc-wHam a>Hucccew muu -eceua use: HuHmca>Hca -xce: N mceHueHcc> m Hacacau -cc acH>ecc camcae muHemaN cewueaHa>u cewuoch>N cemcecm cewuuecumcH mcwuuam cacmecc can: cewuauceec aecmecc coo» .ez we vecua: we aezw we cucecee< uamcaw we ameecac cecu=< xueum cuecaH Hacewuaucem Q chcaee< 109 mucaea>ecesw ucaa -auccas ace: Aces mueew cauacee we an: cauuac mcwxce ceHueueea ceHu -wcuac xmcaxca .muwcac ecwuca -acec acne» HHHcaw ea>ocacw «Ne echuaau a mHHme ceHuacecace mcewuccum meaca Hacec mcwcuaac cew neew va>ecceH nee -ceeae .mcewm cw mcaxaa chHcauus a manuaace ecwxea muceeac macw< mcHem imam eeecm a -aeec mceea meecuas umau HNomHH NN ceew ua>eceew ace gee—a aew< a>HucEcew muu ceHuchez ease: -ce mcaax m Haecw>wecH .aEeecH er a eeHa>aos ca>wHo mawuwu -833 33...; cewu we cewucceeace -ucceace a ucHxac a mcwxsc eeew ueew weeHuchcc cewuocumceeae .aecaercx a mueew Huwvesseu muceeac maewc a ceHueccumcH mcaucsasec ceHuchec HwomHH NN we am: ua>eceeH HNEcewcH a>Huccs=m mmu ceHuchcz mace: Hz ececm HHcem aeeucw 3eH a>eceeH caweaz «cause .mauwuuace caucH cewu -asec cuaac eu vac .ceHuchec we -uecumcw ace eccc camcu cew>ccac a aucchewcae va>ece macHecceHumaae maeHc mcaaa N eu ace vaNH -co>ucmwe= auto—sec; cewu caoHH oN .ew mucazeu acacu: cewuc>camno a>wuascew mug cuzwm ease: . es a -HaecH>HecH aeeucH 3eH -wcuec a>ece=H secs: maceuwuuc wHam .mee . a>oc a mcaxusaeec eu cewuoewuwucee Howe ac uHaecu -em a aaeucH cH caH mcHe emzww -Hewm ace ec: mauwa mavHu cmzwm a mcaxusasec eu cuzwm uwccuac HaeHu HH mcaxmeasec cuneHm sec HmNaHH mN eu uaac sc—wceu Haaccm a>wua5cea mug <\: <\: <\z <\z aaeucw er acwccauac e» uca>ucez mecca -ece ceHuchec camw>aHau ucecm mcwccaHe we aecaueao Haas va>ecesw eaace -uu caxaeaeec aeoaHzecx ceHu -ece >w cewu mcaxaeaeec acwecauae ecu -chcc cH amoacu mumau umee a ace >w a vascewcH ceHu cauw>cw mcwccoHe Haas HoooHH cN -e. ucaunHceHm vacuum—cwcaa-wHaw ascuaccac wee «\z acacxxcc Hz .ch3: ucocm wee a>ocacH ow- ecseaz muHamax cewuaaH~>u ceHueeua>m cemceem cewueccumcH mcHuuam cacaecc can: ceHuaHeeea socmeca caa» .ez we cecua: we aexw we cuaecce< uaecaw we ameecae cecuc< xveum guacaH Hacowuaeaua AH xwvcaee< 110 cewucacewcw Hocewuwcuec caeece Hzcaeseeuu eu a>Hauac ec: cas HHau accum camaceeeca tau-i.e.: acuHcac we -ac eeew .cc vN ucaa: acwHamceeu maszv cae HccaumaHece e: Hecaoex ececu eu choueaeec mawceum HouHcHHu uaHo umw acauaHe aea aHecwe -cam cazeH eu .xamau cm a. aecace Hcauawox a mzaw>caucH a>Huac=3m H.uaz -cewuwcuez eHcHHu mcucee a Hanew>wucH xcuHaaz ca ceHuchez .Haceac sceH>ocac caucH aocece ec caucuHc -ac Huwceceeu -cH muHamac .HHcca>e manna—e a mcew>acac ....uau.Hu=.umH=a meccuuam aaoce N coHu uaHaHac-uooc. HmNaHH aca: w .maocaeeam maewa eeecm -uccumcw ace eu mcaxceaeec muuawwa acwm axe—u ecu mm -ceu vac—mac N we: exaw>caucH a>Hucccew mug cwzau a mace: coax H ace .>HecH aceecw er xwwucacw ew HamEmc ncchece mca cocuchac -xce: cuHcac mucuewewucae a>ac eu eccew AuHcacseu N maze: ce we ucaea>He>cH “cages «aw—Heew cew ..ua cuHeac a cc> aHHcee mcwuacumceeac vase: .caxce: Eacmece .cu muwcac mucee .umwcewuwcu coeac mcewuaucamace Ecaw ucccmw: cuaacuco cc HNNmHH Nn ueew ua>ecesH -ac HNEcewcH a>wum5cew use in: a .EQaw -ueo Hz mcewmmam eeecu aeeucw :eH auw>ece ow. xweecc mach cc. ecceece eecw mcaxasace: mucaHHa cuzcu HHNmHH caunacac mcaxaa a macwc we accuwu HHc: ace Hm -acag a acwc cues ocHaceoHumaao asHuaccec nae <\z «\z (\z <\z Nazca -u. cwHucaeH ucoceHcc amaamwe uHceccu eu mec aoecac eu cew>ccac macwcc a mascuHu aecawccc uaua>wues a -cewumaee cacau mcaxaaem muchcEcc a -uc aucaHzecx cascewcH Huaz \3 -chHeec-wHam AcaHHx=< umaem mcaxaaemv cue uHHcae ceHuchac HmNmHH on cececeec Hawam: umec a ace a>Huacc=m Heuwemex ucaexm Heuweme: mcewmmam n tweam uwHaee Hccacam a>ececH ow HHacec ecHem coHeHuaHe go He -a-HaHe Ha HmanHeacm a macaw ceew NH Huwcecceu eu .chcewmmaw ceHu HHwocaH we mucoaaNHa H\z asHuaccoc HaHHamoz caHuHuaHe mecca -oca cocuccu -chac uaoc. .:c a cuHaac :ccaucee ewH mucce mHaceHmmawece Aces \3 ceHuec -cae cameecc ceHucacewcw -wowucce eu caHHae ccHuHuch a use -caucw .macHHm uHche we auaewHu HNNmHH oN aces cew mumaeeaz acwoccewumaee-wHam a>wuaccew Heuweme: cuHaac mcewca> chweme: ace Ha .muwcwcxu Ha Hacacaw a aucacu. accac muHemac cewuaeHa>u cewuaaua>u cemcecm cewuaacumcH ocwuuam sccmecc can: ceHueHeeec seceecc caa» .ez we vecua: we aexw we cucecee< uaocaw we ameecec cecua< xvaum cuecaH Hocewuaeeem AH xwucaee< 111 Buca .0.- 95:38.2. a oucc Ha eoHuuameuam cew>acac .acw a .zecc .chec ameacucw eu acaucw eeew ecH maewc HaceHmma .acec a macuc ->ecesw a aeeaercc HHce -weceacne a \3 came ac cue cewuwcuec acwmaace -ac ueew cc VN Hamcaeu uchewuchcc mceec mcaccsaeec uccu .eecc .ea HomeH co -cw cw a>Huuawwu macwcccewumaco a>Hucscew chao a aew< Hz Nim meecmcce: aseecw rec .chec a umah ce>HHH=m «caecc canuHe -mzac on Na ech amaca>a acu cu HHeoHH -ccu mcwac accmece HaceHu mcccHeo Hmcacaac canoe ceHuchec we ccauuz on ewcw cew mumaeeac ncauuaH Huwm -uacumcw Hccam HV ca ceHuwcucc -mxacH ewH mucaeceu uwman a mcaxa: cew mcauuaH Ace: uauHu—Hemc: a>HucEcew -ca>wc: <\z wHam mcwemce caumuHccAm -cae Hacacao ca>HHae ew .aceum HmHHHcm .ua asec cacue .mcweaacamcec .acae vHch cew .va emHav .ea mcwc mcacucacec ceHuuecum mcacucaaec ice—e Home a a maewc cew maewe uHe menace -cH ace eu aceecw er ceHuNEcew HwooHH mm maucccu a>HuHmec maeH aeH< a>Hucccew mug -eceua ace: a ease: mcwemce ace .>wucH cewucecea rec -cH cewuwcucz cachHem mcHasc veew Haewuuccc gnaw—o cecuece ensue veew we an: acu cw ucHucewuHuccc cuaeccu muaHu .ca ueeca maceuwuua cHacu a>ece HmumHH ua>ecesw .maeHc ceHuuecumcH .3— eu ceee HHammcm a we mcwxecu ueae meceuac m.ucau mauw< ececm HHasm mcaccEacec acu auouava cemceuoa Nm -cee-wHam ua>ecesH -mwmm< sacaecc a>Hucccew muu ceHuchcz ease: Hz ce Hoecw>chH aeeecw sec cHac eu.... .caHecHem .mauwuuace ceHueaemcee neew masec ucae ceHuuacaucH ua>eceew a amvaH meceuac m.uc~u veeHa>au ececm HHNEm acacasaaec amuawc aeccm HnNaHH on -zecc ceew camcacucH -mHmm< sccoecc a>Huuccew mmu auzwu echeez Hz ce Haccw>wecH aseucw sec -c: eu.... cwcuwacm ..ocHsHH HHHsow a>ecc .5 3 mceew ucuHc acu mcw mxcHe HuHc: ea -meecu a mcaH mac acu -mwc muw a mac =Haccemcae aHHcecH -Hee cHacu mcw cauwmw> aHeeae ocHumH> aHeeae cacwocu mac mcHH caax H 1 muceecac maHHHacw -cuuacums ce HoNaHH mm ooo.oH .ec o cH we w HuccewcH a>Huascew mmu HHHNHeaem. -Ha>acw caec H >< .xcHemHo aseucw er ca cewuwcuez eHchm muHcmac cewuoaHa>u cewuozHa>m cemceem cewuaacumcH ucwuuam Eocmecc can: ceHucHacec eecuecc caaw .ez we vecua: we aezw we cueeccc< uaocew we ameeccm cecu=< Heeum cuecaH HocoHHauaca D .3383 112 ecH -uaa .Heu er ua» maewuwcuec .>ae ow =cec=H mchee eeew camcacuac Hoe HmaNHce cheHau case— wcemcae cae mawce aucuwumcH \3 camac ueaHam eu mcac -ch we w aeoca>c ceeHN a «Heaev asau -eumce Hchau .uca Hacu aHch mamocucce veew uccaz uwcau =uce=ocu -awmev aeacaee HonHH ac cccaH cue aHeeac: we cewua>camcc a>Huoc=3m H.uaz <\z -awcu mcaax m cew coca: maaerQEN sz -ca 3..... uuchcawwHN Huge... m2: 8.: . 23335. a caunacac a>cc Hacu mcewua> ceueecumcw Ha Haaw muceewowucae -camce eeecu uceeecm mcewuccumcee .ea mcw muow—u 092.5qu puEObcm hop-emu uwwaocoum 583:0 nut .oczoou mcmxmsgos 13% 3.2.0..» Amg: we ucwew>ecc Huwcenu: Ha>cem xceuawo a>Hucecew -caucH ace: HaeeH mcwemce .>< . meeecu aseucw :eH a ceHuchcz HHHacaN >w a eHuoc- mamaaHac a mcauuaHmzac- mow—Hacw cwacu HHeuac muchcxa- =.mauwuuoce a mcacccacec acu veew .cc vN muacccs .mceeu- Hacewuwcuac cew,acmucw Hacewu maowa Ac -caeem .mcewu mcacae HHHsew a>ece -wcuec we mcwcocme: machcceHumaee mcwecHacH -ccumceeac- vasec aeeucH new a amcccu chuceumccm a caac cacaumwcwsea maecHe .mecH :eH Hacec a eu mcacue HHNmHH mac acacu .eaceece -wHam .mucaHHu macH< .mue eeecu -uaae eeecm- cacce cecac vasec aoca mmeccem a cc acu we uHemac a m<. \z mxaw>caucH a>Huaccew mmu cmzwu a maze: mcweece muwmw> acezi eu ecu: -cchH eu...= ace: NNEN a .2... Hchac veew cew mmaca>Hu meucum HcceHuwcu cc vN .mauwu -eawwauacwc -cc mcw>ececw cH lance Hocewuch macH< mceex cuwz acmesea a a>wuoawwa ma ueece -cc ocwccaeceu awquE=m mmu cmzwm mace: mcaa» N ace eu ace camcuco>ccmwo -chcc a>eceeH ecu: cewumacewcw Hecewu -chec cew auceemac chccHa> a chHmmae ceHuuameuam aceccaHau HmomHH -uc HHHceac a ma cam: .mHch aH> mcewumace caHuHuaHu chcHx mucaHHe Ac vazaH> we ucauceu mumH acece a eu mcaxmca a Howe oH> a come: Ne mw auH>cam ac». a Hocaceacw a>wumccam aHewuch -ceHuchcz acaczxc< Hz ace eu ace uHHcec Hccacaa .ewcH cewuwcuaz .cacmc: macmHu umee mumaeeac veew auco aunwo cewuaewu 3eH we cewuacae mcewuwcuec cane—ecw -umwmm< .umm< uHH uHucac a cewu -ace a ecHeeecm chas .zHamwz aces muceeac uHHccc maewo Hacewm -ccc .ueao -acumcesac a ..ceHe Haas cew HmamHH He caeeecm mcacusaee: HmccewcH a>Huaccew we .ueao -maweceneccc cw caceuwx mcaaz e mammaHe caecu aeeucw sec ca ceHuchcz erae: muHemax ceHuach>u ceHuacHa>m cemceem cewuencumcH mcHuuam caceecc cam: ceHueHceec seceece caa» .ez we cecua: we aexw we cuaecee< uamcaw we ameecec cecue< Aueum cumcaH Hacewuaeeem HQ .3383 NI CES DSS FSN CDC 0E0 113 No Information Cooperative Extension Service Dept. of Social Service Food & Nutrition Service Center for Disease Control Office of Economic Opportunity APPENDIX E Lit. eview 114 Kpnas lad along sf '* '134 10 g iuasau] Sampling '11w11 5U1ldw9s aieu asuodsaw spouiaw a1dwes °4dtuosap 'dod 'dtuosap aldwes azts a1dwes Statistical Analysis ‘3510 SUGllellwlT '615 10 13A31 $1915 434“! J3410 g 40 ‘z ‘1 '1915 'sap uauio H, °p°s pue x Appendix E Measurement '510 SUOlleliwll KiLPLLPA 41111491133 isaiaud iuawnuisu1 Evaluation Design Instrument '810 “Ollellwll 2 I’Tjooo 2 ’§// 2,000 0 r;/ 5,000 2 2 stsauiodKH '0 uotienleAg asodund cu anal tseno aJd u—c v—l Cl C S. O CD 0.)" in“ A SA A :r-s ma A A a 83 me Hg 38 23 38 :2 H2 c8 JeaA/uouinv .ooaom Em .-ea rec» we» was ces nos cm .0 die—c e-l Q’s—I Lv—c 5.1—1 :v-i 3H OH OH < CDV (V mv mv mv mv mv UV UV 'ON Kpnis H N m <- m a N co m 115 °nis lad 34033 g :53 5: <1- es H S m c.» '}——BJ .40 # H o m o N iua saud N\N N\¢ N\¢ o\° o\° o\ NV. o\ N\ noun 'dwes O 319.; asuodsau :3 °u43w a1dwes H N 'sap 'dod o o 'sap aldweg N H N H .H O O O O O O 3 N \8 8 mg AH? 9.2 :2 3215 a1dwes N N N N N\r-1 N '3510 ‘llml‘l N o e—iLO HLD e—t 'fiis .10 181181 “40.0. “\99 O. '1915 3w;- ‘tno e"): H“ 3 JO ‘2 ‘1 NH "191$ '59:) mo N "' "' apes R N H '3510 own 0 KllpllPA '11991138 4531314 '3510 '11!le ‘3 'OdKH O N N 'le’d o N N asodund N N anuy m tsenc N N 3“ N H H H H N c . O +9 C C .0 to 05 a) O 0 Dog. (I) m x 'l- U" Q) A 0 A U!" l.l—A LA Uh mm (hr-s Dis-N 't-UlN C d’ OJN rem (UM w-O v—LD OJN use use 3.3 as as es :3 es E8 JPBA/JOHWV 't- 1-1 cool-a SHu—c—i Lo-o 3H div—i QJv-l PH 00-4 41-05" way (9 (UV (5v (9" IV IV Iv IV '01,] A‘pnis $3 121 9.1 53 S 3 5.3 S 95’. 116 -n15 Jed 'aJoa 6*) g g: ,I'-f'\ 'Ur‘ (Di-s >,r-\ LA HH (UN >305 ens-=1- .QIUN 'PLD >m .cc moo H N m N v— N m N O N > o «H [N mac n- ED "523 5.93 32 $.52 8J2 823 82 ‘59. 5:93. JeaA/Joqnnv Iv "av xv —l cvv z mv 22v 2v zv 2v 0N Kpnus a 8 a 22‘. 8 a “(a a R °n1s 43d 34033 ';34 ;o # nuasaqd‘ 117 obf’ 18 'ILWL1 'deS 3194 asuodsaa 'q13w atdwes 'sap °dod 'sap aldwes 'OSLO '1Lw11 '§;s 1o [3A31 '1915 '434u1 43q1o 3 JO ‘2 ‘1 '1918 '83P uauno °p°s g x *DSLG '1;wL1 KltpllPA ‘llq9tl98 asaand °n415u1 321$ alduuesN\Ln N\ N\H 2 2 q-o—o r—l c3\ .10 2 ’Tfi,ooo 1 'DSLG ’1Lw11 °odKH 'lPAB SSOdJnd 3n41 ;seno 34d JeaA/uoqnnv ’ON fipnns; Oliver 28 (1967) Perry (1972) Podel] (1975) 29 3O Prichard & Hall (1971) 31 Prouix (1972) 32 Ramsey & Cloyd 33 (1975) Remmel, et. a} 34 (1969) Schild (1970) 35 °nqs 43d 3403 “434 JO # 51m °1Lw;1 'dwes 3134 asuodsau “ulaw aldeS 'sap °d0¢ 'sap aldweg 3213 aldwes '35!G 'lleW '615 JO [3A31 "191$ '434u1 43q10 j 40 ‘z ‘1 ‘53P Jauzo '9fs g x “3910 '1LWLTI KllpllPA 'lLQPELBH 153134d '1315 °n4qsu1 <- 2 ’121.2 2 2 118 17 A? ’3200 2 19 2 ’11 20 913 20 .05 15 N\H .10 'OSLO 'ILNLT 'odKH 'lPAB asod4nd 3n4l gseno 34d 4934/4oq1nv ’ON Kpnls Shefcik (1973) 36 1 SpindIer, et. a] (1969) 37 Spindler (1967) Stare (1961) Su11ivan (1976) 38 39 40 Ugelow (1965) 41 Wagner 42 et. 31 (1965) Wang (1977) 43 1 Wang & Ephross (1971) 44 'n1s 43d 34035 “434 J0 # 1uasa4d ON 22 12 '1Lw11 'dWPS 3134 3suodsaa 'uzaw atdwes 's3p 'dod 'sap aldwes 321$ aldmes 'DSLG ‘1LWLl '515 10 [3A31 °1e1s '431u1 43u1o 3 40 ‘z ‘1 '1215 'sap 43q1Q 'p's 9 x '35!G 'aLw;1 fiszLLEA 'llq91l38 153134d °n41su1 'DSLG '1lwll °odKH '1943 350d4nd 3n41 1sen0 34d 4e35/40q1nv Zanelli (1969) 'ON Kpnzfl 45 Zifferblatt (1980) 46 Ivv 119 CODES blanks - not present present but limited or inadequate or unclear 1: present and adequate and clear 2: excellent 3: APPENDIX F Classification Manual for Studies of Heaith Promotion Programs and Evaiuation Kathy Akpom, M.P.H. August 1981 120 121 Table of Contents Page_ Introduction .................................. I... 1 Classification of Studies ......................... 3 Identifying Information ................... .. 3 Purpose and Strategies of the program ....... 5 Evaluation .................................. 13 Results ............................... . ..... 23 General Information .............. . .......... 25 APPENDIX F CLASSIFICATION MANUAL FOR STUDIES OF HEALTH PROMOTION PROGRAMS AND EVALUATION CHAPTER I Introduction This manual is the result of the research of a dissertation which sought to apply data synthesis techniques to a subset of evaluation studies of health promotion programs. The synthesis techniques were developed in the fields of education and educational psychology and were not applicable for use with the subset of studies (nutrition education) upon which they were tested. Through review of literature on health promotion programs, evaluation-synthesis methods and secondary evaluation techniques, as well as discussion with health professionals, it seems that the need exists for a methodology for classifying according to multiple categories, the content, methodology. and quality of studies of health promotion/ risk reduction so that this data can be easily retrieved and used in developing health promotion programs based upon existing evalua- tion research findings. The classification manual is an attempt to answer this need. It offers program planners, administrators, and evaluators a flexible and standardized methodology for organizing and retrieving information on programs via a variety of multiple categories. That is, if a program planner wishes to review literature to plan a smoking cessation program, this could be done in a number of ways: l. by review of literature in the thematic area (smoking cessation); 2. by review of various program strategies that have been used with your intended target audience, from a variety of areas; or 3. by review of evaluation designs that have been used from a variety of thematic and/or program strategies. Each type of review could be quickly and easily accessed using the manual. 122 123 2 The manual is not an annotated bibliography; it does not contain summaries of studies from various thematic areas. Instead, it pro- vides a framework--a system--for coding the content, methodology and quality of program reports (journal articles, final reports, etc.). The classification enables quick and efficient retrieval of informa- tion. It is a tool to enable practioners to extract, code, and make available for organized and easy access, information from studies that we all read routinely in professional journals. No statistical ability is required. The manual can be used with a computer retrieval system or a hand-sort method using index cards, and therefore, can be accommodated by any size budget. Like any retrieval system, it will only be able to "give out" data from that which has been fed in. The more studies classified and stored, the more useful the multiple category retrieval system will be. Because the classification manual defines and uses standard terminology, a large "data bank" can be established through a cooperative effort of several programs sharing coded studies that they have classified as a result of on-going liter- ature review. 124 3 CHAPTER 2 Classification of Studies As stated in the Patient Classification Manual], "A fundamental requirement of any classification is a linguistic base that is well defined, clearly understood and systematically used." In this chapter, definitions will be provided for terminology to be used in the classification of studies. There are five major sections into which the classification has been divided and the importance of each in the classification of health promotion studies is briefly discussed. Terms used in the classification are defined by section and specific classification items are presented as well as suggested numeric codes for computer or hand-sort use. A. Identifying Information2 Items in this section are: Program or project Classifier Study number Thematic area of study th-d l. Program or project This item identifies the program or project conducting the classification. Categories Suggested Codes Individual programs or ........ Identification numbers projects assigned to individual programs or projects It is important to be able to identify the program or project if classification is needed. It is anticipated that an efficient way for programs to build up a large "data bank" of classified studies will be through sharing (exchanging) coded studies. To enable clarification, if needed, each program participating in the exchange will have a unique identification included in each coded study. 2. Classifier3. This item identifies the person completing the study classification. 125 4 Categories Codes Individual classifiers ........ Identification numbers assigned to individual classifiers. It is important to be able to identify the individual within a program or project who classified an individual study in case questions arise concerning classification. 3. Study number4. This item identifies the individual study by a number unique to the study. Categories Codes Individual studies ............ Unique numbers assigned to each study Each study that is classified by each program or project will be assigned a number unique to that study for the purposes of: (a) distinguishing one coded study from another, and (b) linking the coded information so that the original article or report can be consulted for more indepth review. All studies will be referenced according to the APA method. 4. Thematic area. This item refers to the specific health/risk behaviors that is the concern of a study. Categories Codes Preventive health services: High blood pressure control ............. 0 Family planning ........................ l Pregnancy and infant health ............. 2 Immunization ........................... 3 Sexually transmitted diseases ........... 4 Health Protection: Toxic agent control ..................... 5 Occupational Safety and health .......... 6 Accident prevention and injury control.. 7 Fluroidation and dental health .......... 8 Surveillance and control of infectious diseases ............................. 9 126 5 Categories (continued) Codes (continued) Health promotion: Smoking and health ..................... lO Misuse of alcohol and drugs ............ ll Nutrition .............................. l2 Physical fitness and exercise .......... 13 Control of stress and violent behavior ............................. l4 The thematic categories are those identified in the U.S. Department of Health and Human Services, USPHS document, Promoting Health/Preventing Disease: Objectives for the Nation, USGPO Fall l980. B. Purpose and Strategies of the Program Items in this section are: 1. Purpose of the program 2. Program strategies 3. Program design 4. Length of program 5. Instructors/providers 6. Target population 7. Setting 8. Sponsor(s) These items are basic program descriptors that define the scope and program processes--necessary information for replicating the program. The information also provides data to determine, in Section III (1) if there is congruence between the evaluation of the program and its purpose and strategies. l. Purpose of the program The purpose of a program is the aim towards which interven— tion strategies are targeted. Program purpose can be implicit (not stated) or explicitly stated. (a) Categories Codes Explicit purpose ...................... 0 Implicit purpose ...................... l Unable to determine purpose ........... 9 (b) Categories Codes Needs assessment or survey ............ 0 Program to stimulate interest ......... 1 Program to increase knowledge ......... 2 ‘fihw _nr- . _ - 127 6 Categories (continued) Codes (continued) Program to introduce a new skill or reinforce a skill ............... 3 Program to change altitudes, values and beliefs ................. 4 Program to extinguish negative behavior ........................... 5 Program to introduce new positive - behaVior/or reinforce an existing positive behavior ......... 6 2. Program strategies This item refers to the general health promotion approaches used by the program to achieve their goals. Categories Codes (Multiple coding possible) Host directed: Instruction .......................... 0 Education ............................ l Persuasion ........................... 2 Behavior Modification ................ 3 Proselytizing ........................ 4 Screening ............................ 5 Advertising .......................... 6 Agent directed: Marketing ............................ 7 Product modification ................. 8 Engineering .......................... 9 Substitutes .......................... 10 Regulation .......................... ll Legislation .......................... 12 Environment directed: Physical influence ................... l3 Social-cultural influence ............ l4 Economic influence ................... 15 Media influence ...................... 16 Not applicable ......................... 98 No information ......................... 99 The categories of strategies used for classification are those identified by Lauzon in his paper "An Epidemiological Approach to Health Promotion," Canadian Journal of Public Health Volume 68, July/August 1977} The following definitions are quotes from Lauzon's paper. Host-oriented approaches: "Instruction Teaching of very specific information or skills. Education Persuasion Behavior Modification Proselytizing Screening Advertising Counseling 128 7 A logical and rational approach to information in which cognitive and affective objectives predominate. Intensive messages intended to stimulate compliance to some target behavior. Often rely on fear-arousing appeals. Altering of behavior by modeling and and selective reinforcement. Process of recruiting others to one's own beliefs. Evaluation used to identify health risk factors. Such evaluation may stimulate awareness and personal relevance for some particular health threat. Any paid form of non-personal presentation and promotion of products, services, or ideas by an identified sponsor. Personal guidance of an individual using various techniques of the personal interview." Agent-oriented approaches: ”Marketing Product modification Engineering Substitutes Regulatory controls Discovering the wants of target audiences and the creating goods and services to satisfy them. Changes in existing products having a positive or negative effect on health behavior. Improvements in the design of structures, machines, products, systems, and processes. Alternative choice of some product or behavior. Authoritative rules governing certain details or procedures associated with availability of goods or services." 129 8 Environment-oriented approaches: "Physical Effect or change in the physical influences space; equipment or facilities inhibited or employed during work or leisure. Socio-cultural Effect or change in current or influence traditional social customs or Economic Effect or change in the economic influence factors which may facilitate or hinder the availability of some particu- lar opportunity, product, service, or standard of living. Media influence Effect or change in the mass media." 3. Program design This item refers to the specific educational/intervention activities undertaken by the program. Categories Codes (multiple codes acceptable) (a) General sessions 0 (b) Small groups Topical discussion groups .......... 1 Laboratory groups .................. 2 Special interest groups ............ 3 Problem solving groups ............. 4 Planning groups .................... 5 Instructional groups ............... 6 Inquiry groups ..................... 7 Evaluation groups .................. 8 Skill practice groups .............. 9 Consultative groups ................ 10 Operational groups ................. ll Learning teaching teams ............ 12 Dyads .............................. 13 Triads ............................. 14 Buzz groups ........................ 15 (c) Individual consultation .............. 16 (d) Reading .............................. 17 (e) Recreation or meditation ............. 18 (f) Preparatory activity ................. 19 (9) Other ................................ 20 Not applicable ....................... 98 No information ....................... 99 130 9 The descriptive terms and their definitions for this item are taken from Knowles' The Modern Practice of Adult Education Andragogy Versus Pedagogy. Association Press, New York, 1970, pp. 289-290, and represents fairly well accepted educational terms and definitions. Definitions for the category terms are as follows: (a) General sesssions Meetings of all participants as a whole, with a variety of patterns of platform presentations and audience participation. (b) Small groups of various sizes and for a variety of purposes, including topical discussion: Groups Laboratory groups Special interest groups Problem-solving groups Planning groups Instructional groups Inquiry groups Groups organized for the purpose of reacting to, testing the meaning of, or sharing ideas about informational inputs from readings or speakers on given topics. Groups organized for the purpose of analyzing group behavior, experimenting with new behavior, and sharing feedback regarding the effects of various behaviors. Groups organized according to categories of interest of participants for the purpose of sharing experiences and exploring common concerns. Groups organized to develop solutions to procedural or substantive problems of concern to the total assembly. Groups organized to develop plans for activities within the design for back-home application. Groups organized to receive instruc- tion through the service of resource experts in specialized areas of knowledge, understanding, or skill (or behavior). Groups organized to search out information and report their findings to the total assembly. (C) (d) (f) Skill practice groups Consultative groups Dyads Triads Buzz groups Individual consultation Reading Recreation or meditation Preparatory activity 4. Length of orogram 10 Groups organized for the purpose of practicing specified categories of skills. Groups organized for the purpose of giving consultative help to one another. Two-person groUps organized to share experiences, coach each other ...or help each other in any other way. Three-person groups organized for mutually helpful purposes. Randomly organized groups that meet in a general assembly to pool problems, ideas, or reac- tions.... Counseling or direct study in which the services of resource persons are made available to individual participants for personalized help. The scheduling of special times (between meetings) for reading handouts (or self-instruction not accompanied by group-learning sessions). Periods of time set aside for socialization... or creative solitude. Things that learners are invited to do before a learning activity starts, such as reading, self- analysis, data collection, etc. This item refers to length along three dimensions: (a) Length of the intervention program (i.e., a 12-week CPR course) (b) Interval of interventions (i.e., twice a week) (c) Amount of time per intervention session (i.e., 30 minutes) 132 11 (a) Length of intervention program Categories Codes 1 time only ..................... 0 1 week or less .................. 1 >1 week3 years ........................ 10 No information .................. 99 (b) Interval of interventions Categories Codes once/week ....................... 0 >once/week ...................... 1 twice a month ................... 2 once a month .................... 3 >once a month ................... 4 quarterly ....................... 5 >quarter1yzsemi-annually ........ 6 other ........................... 7 No information .................. 99 (c) Length per intervention session Categories Codes ?30 minutes ..................... 0 >30? 60 minutes ................. 1 >60 minutes ..................... 2 No information .................. 99 Not applicable .................. 98 5. Instructor/Provider This item identifies the type of professional person(s) conducting or directing the intervention strategy. Categories Codes Health educator ................. 0 Physician ....................... 1 Nurse ........................... 2 133 12 Categories (continued) Code (continued) Nurse practitioner .............. 3 Physician assistant ............. 4 Occupational therapist .......... 5 Physical therapist .............. 6 Social worker ................... 7 Paraprofessional ................ 8 Other ........................... 9 Not applicable .................. 98 No information .................. 99 6. Target population This item identifies the group of individuals to whom the intervention strategies or needs assessment was directed. Categories Codes (Multiple coding per case) Children ........................ O Adolescents ..................... 1 Adults .......................... 2 Elderly ......................... 3 General public .................. 4 Blacks .......................... 5 Whites .......................... 6 Hispanics ....................... 7 Other ethnic groups ............. 8 Low income ...................... 9 Middle income ................... 10 High income ..................... 11 Urban ........................... 12 Rural ........................... 13 American ........................ 14 Canadian ........................ 15 Third world ..................... 16 Other ........................... 17 No information .................. 99 7. Setting This item refers to the place where the interventions take place. Categories Codes Community Center ................ 0 Health facility ................. 1 Individual home ................. 2 134 13 Categories (continued) Codes (continued) Shopping center, supermarket, or other p1ace-of-business.... 3 Television ...................... 4 Radio ........................... 5 Newspaper ....................... 6 Magazine ........................ 7 Other ........................... 8 Not applicable .................. 98 No information .................. 99 8. Sponsor(s) This item refers to the agency(ies) or group(s) responsible for offering the intervention. Categories Codes (Multiple coding acceptable) Hospital ........................ O Ambulatory care program (excluding HMO's) ............. l HMO ............................. 2 Voluntary health organization (such as the American Cancer Society) ............... 3 Social services agency .......... 4 Community/consumer organization .................. 5 Cooperative extension service... 6 University or college ........... 7 Other ........................... 8 No information .................. 99 C. Evaluation Items in this section are: 1. Evaluation design 2. Instruments 3. Sampling 4. Data analysis This section describes aspects necessary to replicate an evaluation design as well as items that assess the adequacy or quality of the evaluation. 135 14 1. Evaluation design This category of items refers to specification of the evaluation plan and consists of the following items: (a) Purpose Evaluation questions Hypothesis Intended outcomes Variables Evaluation design Limitations of design Purpose (1) Purpose-specification This item refers to the broad category of purpose of the study. (2) Categories Codes Needs assessment ............. 0 Assess change in knowledge... 1 Assess change in skill ....... 2 Assess change in attitudes, values,and beliefs ......... 3 Assess change in behavior.... 4 Assess community impact ...... 5 Assess cost benefit .......... 6 Assess administrative performance ................ 7 Assess quality in process of instruction/care ........ 8 Other ........................ 9 No information ............... 99 Type of evaluation This item identifies the reason the evaluation was conducted. Categories Codes Formative .................... O Summative .................... 1 No information ............... 99 Formative evaluation is conducted for program development and improvement. Summative evaluation measures program effectiveness. 136 (3) Congruence This item refers to whether or not the purpose of the evaluation is congruent with the purpose of the program. Categories Codes Incongruent .................. O Uncertain .................... l Congruent .................... 2 Not applicable purpose(s) not known .................. 98 An evaluation purpose is congruent if it assesses formative or summative evaluation relative to the intended outcomes of the program. (b) Evaluation questions This item identifies the presence and clarity of specific evaluation questions. Categories Codes Evaluation questions not stated ................. 0 Evaluation questions stated, but clarification needed ..................... 1 Evaluation questions clearly stated ............. 2 (c) Hypothesis This item identifies the presence and clarity of hypothesis. Categories Codes Hypothesis not stated ........ O Hypothesis stated, but clarification needed ....... l Hypothesis clearly stated.... 2 (d) Outcomes This item identifies whether or not the intended out- comes of the program (and its interventions) are clearly specified. 137 16 Categories Codes Outcomes not stated .......... 0 Outcomes stated, but clarification needed ....... 1 Outcomes clearly stated ...... 2 (e) Variables This item identifies the category of variables under study. Categories Codes Structure .................... 0 Process ...................... 1 Outcome ...................... 2 Impact ....................... 3 Context ...................... 4 Uncertain .................... 5 (f) Evaluation design (1) Description and appropriateness This item identifies if the evaluation design was fully described and whether it seems appropriate for the study. (2) Categories Codes Evaluation design not described .................. 0 Evaluation design described, but not adequately ......... 1 Evaluation design adequately described, but seems inappropriate .............. 2 Evaluation design adequately described, and uncertain about appropriateness ...... 3 Evaluation design adequately described and seems appropriate for study ...... 4 Design This item refers to the specific evaluation design of the study. 138 17 Categories Codes Preexperimental designs: One shot case study .......... lO Posttest only - 1 group ...... ll Quasiexperimental designs: One group pretest-post design ..................... 20 Non-randomized control group pretest-posttest ..... 21 One group time series ........ 22 Non-randomized control group posttest only ........ 23 Other quasiexperimental designs .................... 24 True experimental designs: Randomized pre-posttest design ..................... 30 Randomized solomon four group design ............... 31 Randomized control group ..... 32 Other true experimental design ..................... 33 No information ............... 99 Most designs included above were those identified in Issac, S. & Michael, N.B. Handbook in Research and Evaluation for Education and the Behavioral Sciences, San Francisco, Robert K. Knapp. Publishers, 1974, 36-49, as being the most commonly used designs. (9) Limitations of design This item refers to whether or not any limitations in measurement of the particular design of a study were dis- cussed. Categories Codes Limitations not discussed.... 0 Limitations discussed, but require clarification.. 1 Limitations discussed adequately ................. 2 Not applicable, no design discussed .................. 99 2. Instruments This category of items refers to specification about the quality and type of instruments used in data collection and contains the following items: (a) 139 18 (a) Instrument (b) Reliability-Pretest (c) Data collection procedures (d) Validity-Pretest (e) Limitations (f) Appropriateness Instrument This item identifies the type of data collection instrument used in the study. (b) Categories Codes No instrument used-- anecdotal reports only ..... O Survey-~persona1 interview without instrument ......... l Survey--personal interview using instrument ........... 2 Self-administered questionnaire .............. 3 Observation instrument ....... 4 Other ........................ 5 No information ............... 99 ReliabilityG-Pretest This item refers to the replicability of data collection methods. (C) Categories Codes No report of instrument reliability provided ........ O Instrument reliability reported, methods not described ................... l Instrument reliability reported, and methods adequately described ........ 2 Data collection procedures7 This item concerns whether data collection procedures are clearly enough described to be followed by others. Categories Codes Data collection procedures not reported ............... O 140 19 Categories (continued) Codes (continued Data collection procedures not adequately reported.... 1 Data collection procedures adequately reported ........ 2 Not applicable, anectdotal... 99 (d) Validity-Pretest This item refers to the accuracy of the measurement instrument. Categories Codes No report of instrument reliability provided ....... O Instrument reliability reported, methods not described .................. l Instrument reliability reported and methods adequately described ....... 2 (e) Limitations This item refers to whether or not any limitations in measurement using the particular instrument were discussed in the study. Categories Codes Limitations not discussed ..... 0 Limitations discussed ......... l 8 (f) Appropriations This item refers to whether or not the instruments appear appropriate to the evaluation questions and hypothesis. Categories Codes Instruments seem inappropriate .............. O Undecided .................... 1 Instruments seem appropriate ................ 2 141 20 3. Sampling This category of items refers to specification about the quality and type of methods used in obtaining a study population, and contains the following items: (a) Sample size (b) Sample description (c) Population description (d) Sampling method-description (e) Appropriateness of method (a) Sample size Categories Codes Sample size as reported in study ................... same No information ............... 999 (b) Sample description Categories Codes Not discussed ................ O Discussed, but not adequately ................. 1 Adequately discussed ......... 2 (c) Population description Categories Codes Population not described ..... 0 Population described, but not adequately ......... 1 Population adequately described .................. 2 (d) Sampling method-description Categories Codes Sampling method not described .................. O Described, needs clarification .............. l Adequately described ......... 2 142 21 (e) Appropriateness of sampling method Categories Codes Appears inappropriate ........ O Undecided .................... 1 Appears appropriate .......... 2 4. Data analysis Items in this category are: Descriptive statistics-discussion Descriptive statistics-application Inferential-discussion Inferential-tests Inferential-application Significance level Limitations (a) Descriptive statistics-discussion Categories Codes Not discussed ................ O Performed, inadequately discussed .................. l Performed, and adequately discussed .................. 2 Not applicable--none performed .................. 98 (b) Descriptive statistics-application Categories Codes Appears inappropriate ........ O Undecided .................... 1 Appears appropriate .......... 2 Insufficient information ..... 3 Not applicable ............... 98 (C) Inferential-discussion Categories Codes Appears inappropriate ........ O Undecided .................... 1 Appears appropriate .......... 2 143 22 Categories (continued) Codes (continued Insufficient information ..... 3 Not applicable ............... 98 (d) Inferential-tests Categories Codes (Multiple Coding) x2 ........................... o t-test or z-test ............. l Anova ........................ 2 Ancova ....................... 3 Multiple regression .......... 4 Facts Analysis ............... 5 Other univariate statistic... 6 Other multivariate statistic .................. 7 Not applicable ............... 98 No information ............... 99 (e) Inferential-application Categories Codes Appears inappropriate ........ O Undecided .................... 1 Appears appropriate .......... 2 Inadequate information ....... Not applicable .............. 98 (f) Significance level Categories Codes .Ol ........................ O .05 ........................ 1 Other ........................ 2 Not applicable ............... 98 No information ............... 99 (9) Limitations of statistical methods Categories Codes Not discussed ................ O Inadequately or incorrectly discussed .................. 1 144 23 Categories (continued Codes (continued) Adequately discussed ......... 2 Not applicable-~not used ..... 98 D. Results and Conclusions Items included in this section are: Results-presentation Direction of results Results-interval Conclusion-presentation Conclusions-substantiated Generalizations mm-th-d l. Results-presentation This item refers to how clearly the results of the study were presented. Categories Codes Not stated ................... O Stated, needs clarification.. 1 Stated clearly ............... 2 8 2. Direction of Results This item identifies whether the reported results were positive or negative and if these reports are substantiated by evidence presented in data analysis. Categories Codes Negative, but not sub- stantiated by the evidence presented ......... O Negative, and substantiated by the evidence presented .................. 2 Positive, but not sub- stantiated by the evidence presented ......... 1 Positive, and substantiated by the evidence presented .................. 3 Not applicable ............... 98 No information .............. 99 145 24 3. Results-interval Categories Codes Short term ................... 0 Long term .................... 1 Not applicable ............... 98 No information ............... 99 4. Conclusionsepresentation This item were presented. refers to how clearly the results of the study Categories Codes Not stated ................... O Stated, needs clarification.. l Stated clearly ............... 2 5. Conclusions-substantiated 6. Generalizations Categories Codes Not substantiated by evidence presented ......... O Undecided .................... l Substantiated ................ 2 Not applicable--no conclusions ................ 98 No information ............... 99 10 Categories Codes Inappropriate-~not confined to population from which sample was drawn ........... O Undecided .................... 1 Appropriate .................. 2 Not applicable ............... 98 No information ............... 99 E. 146 25 General Information Items included in this section are: 1. Literature review 2. Clarity of report 3. Tone of report 4. Meta-evaluation score 1. Literature review These items concern the appropriateness and extent of literature review presented (included) in the study. (a) Literature review-quantity Categories Codes Number of footnotes (references cited) in study (b) Literature review-appropriateness and adequacy Categories Codes Literature review inadequately or inappro- priately conducted ......... Still question relation- ship of previous research to the problem under study, adequate and appropriate ................ 1 Not applicable--not literature review .......... 98 . ' 12 2. Clarity of report Categories Codes Not clearly written .......... 0 Clearly written .............. l 13 3. Tone of report This item refers to whether the report displays an unbiased, impartial scientific attitude. 147 26 Categories Codes Appears unbiased and impartial .................. 2 Undecided .................... 1 Appears biased and partial... O 4. Meta-evaluation14 score This item refers to the quality of the article or report as identified by considering the ratio of high to low codes for selected items applicable to a study (98 and 99's not included). (The foils were organized, where appropriate, with the least desirable alternatives being assigned the lowest codes and the most desirable, the highest codes, excluding 98 and 99). Example: a score of 3/1 or .33 would indicate that for every low score, the article received three high scores. Directions for calcu- lating the score is provided as an estimate for comparative purposes among articles. APPENDIX G Kathy Akpom, M.P.H. Evaluation Consultant 966 Longfellow Drive 0 East Lansing. Michigan 48823 0 (517) 351-5 2114 August 27, 1981 Dear Health Promotion Specialist: Thank you for agreeing to take the time from your busy schedule to review the attached paper. It is a classification manual for reports of health promotion programs, and is a result of my dissertation research. The manual is intended for use by people such as you who are developing and running health promotion/risk reduction programs and who want to apply to their programs "the best of what is known" about promoting positive health behavior. Since very few health promotion programs can afford to conduct extensive research, some of our colleagues have identified the need for some kind of a method for sorting out the vast amount of program experience available from a variety of fields and types of projects that can be useful in planning and evaluating health promotion programs. The attached classification manual is an attempt to answer that need. It offers program planners, administrators, and evaluators a standardized, simple, and inexpensive methodology for organizing and retrieving information on relevant programs via a variety of multiple categories. That is, when you are planning a program, you may wish to review literature concerning the thematic area of your program (such as smoking); the promotion strategies you plan to use (such as group counseling methods, from a variety of thematic areas); or evaluation designs that have been used from a variety of thematic areas and/or promotional strategies. The attached manual is not an annotated bibliography; it does not contain summaries of studies from various thematic areas. Instead, it provides a framework--a system--for coding the content, methodology and quality of program reports (journal articles, final reports, etc.). The classification enables quick, efficient retrieval of information program planning and evaluation. The manual is a tool to enable practioners to extract, code, and make available for organized and easy access, information from studies that we all read routinely in professional journals. No statisti- cal ability is required. The manual can be used with a computer retrieval 148 149 8/27/81 Page Two system, or a hand-sort method using index cards and, therefore, can be accommodated by any size budget (a computer coding form and a manual index card-coding form will be included in the final form of the manual). Like any retrieval system, it will only be able to "give out" data from that which you've "put in." The more studies classified and stored, the more useful the multiple category retrieval system will be. Because the classification manual defines and uses standard terminology, a large "data bank" can be established through a c00perative effort of several programs sharing coded studies they have classified as a result of literature reviews. Since the classification manual is still in the pilot stage, I am requesting your assistance. Please read it over and consider how useful the classification system could be and what changes you would suggest to improve it. To facilitate your reSponses, a short questionnaire is attached. Please complete the questionnaire as well as any other comments you would like to make. Because this manual is a result of my dissertation research, I will be including it and a summary of comments about it, in an appendix of the dissertation, and to meet my orals deadline, I need to have your comments back by September 10, 1981. Thank you for your help. Attached is a self-addressed, stamped envelope for your convenience. Sincerely, /} l /4.' _. $46.4.st C, (v if '3' "’1 / A Kathy Akpom, M.P.H. KA/aj enclosure P.S. Please do not reproduce the manual in the draft form. If you would like a copy of the final version, please indicate that on your questionnaire along with your name and address, and I will be happy to send you a copy after my orals. 150 Expert Opinion Questionnaire for the Health Promotion Programs' Classification Manual Are there any suggestions you have for making the directions to the manual more easily understood? yes .no If yes, please specify. Are there any items that should be added, changed or deleted from the manual? yes no If so, please specify the modifications you suggest and rationale for each, by manual section. Section Modifications Rationale A. Identifying Information B. Purposes and Strategies C. I. Evaluation Design II. Instruments III. Sampling IV. Data Analysis 3. 151 D. Results E. General Information Are there any categories of information that should be added to those covered in the manual? yes no Please specify. (a) Do you think that many of your colleagues feel the need for a classification system such as this? yes no (b) If your answer to question 4 (a) is yes, then what types of positions do these colleagues have and in what type of programs do they work? Positions Programs (examples) health educators health department program evaluator non-profit agency (c) Do you feel the above identified persons are more likely to use the classification system by computer or handvsort method? computer method hand-sort method Do you personally feel your program planning and evaluation efforts would benefit from use of the manual? yes no (a) If the manual were available do you think you would ever use it? yes no (b) If yes, by which method? computer hand-sort 152 In order to generalize your comments, I would appreciate your answers to the following questions: (a) What is your professional position? (b) Which activity best describes your primary responsibility? program planning administration program evaluation community health education other (please specify) (c) In what type of agency do you work? (d) Please identify your highest degree. B.A. ” M.P.H. or M.H.8.A. Masters Degree other than above Ph.D. Other (please specify) (e) How many years have you been in your profession (not your current position)? years (f) Please identify the l or 2 journals upon which you rely most often for information related to planning and evaluating your programs. Please make any other comments fimnthe reverse side of this page) that you feel would make the classification a more useful tool for planning, administering, and evaluating health promotion/risk reduction programs. Thank you for your cooperation in completing this questionnaire BIBLIOGRAPHY BIBLIOGRAPHY Ardell, D. B. and A. B. Newman: Health promotion: Strategies for plan- ning. Health Values: Achievinngigh-Level Wellness 1(3):100- 107, May/June 1977. Bertram, Dennis A. and Peggy Brooks-Bertram: The evaluation of continu- ing medical education: A literature review. Health Education Monographs 5(4):330-362, Winter 1977. Block, D. Health promotion: Prospects for the future. The Second Katherine Boucot Sturgis Lectureship in Preventive Medicine. Pre- sented Oct. 19, 1980 Detroit Michigan, American College of Pre- ventive Medicine. Bracht, Glenn H., and Gene V. Glass. "The external validity of experi- ments," American Educational Research Journal 5 (November 1968): 437-474. Bracht, G. H._ Evaluation of the evaluation proposal. Unpublished manuscript, 1974, as reported in Sanders, James R. and Nafziger, Dean H. A baSlS for determining the adequacy of Evaluation Designs. Paper #6 in Occasional Paper Series, Evaluation Center, College of Education, Western Michigan University, Kalamazoo, MI Bray, G. A. Nutrition in the Humphrey tradition. Journal of American Dietetic Association 75:116-121. Aug. 1979. Briggs, G. M. The need for nutrition. Journal of Nutrition Education, 1(1):7, 1969. Brtian, Gerald M. Experimental and contextual models of program evalu- ation. Journal of Evaluation and Program Planning (1): 229-234, 1978. Campbell, Donald T. and Julian C. Stanley: Erperimental and Quasi-Ex- perimental Designs for Research. Rand McNally College Publishing Co., Chicago, 1966, p. 84. Cohen, D. K. "The value of social experiments." In Rivlin, A. M. and Timpane P. M. (eds.), Planned Variation in Education. Washington D. C.: Brookings Institution, 1975, 147-176. As quoted in Datta, L.E.: Does it work when it has been tried? and Half full or half empty? Evaluation Studies Review Annual, 12:302, 1977. 153 154 Donabedian, Aredis. Evaluating the quality of medical care, Milbank Memorial Fund Qparterly. XLIV(3):166-206, July 1966 Part 2. Dunn, H. L. What high-level wellness means. Health ValueszAchieving High-level wellness, 1(1):9-16, Jan/Feb 1977. Dwore, R. B. and M. W. Krueter. Update: Reinforcing the case for Health Promotion. Family and Community Health, 2(4):103-119, February 1980. Eysenck, H.J. An exercise in mega-silliness, American Psychologist, 33(5):517, May 1978. Reprinted in Evaluation Studies Review Annual, Vol. 3:697, 1978. Flaherty, E. W. and J. A. Morell. Evaluation: Manifestation of a new field. Journal of Evaluation and Program Planning (1):1-10, 1978. Freeborn, D. and M. Greenlick. Evaluation of the performance of ambula- tory care systems: Research requirements and opportunities. Medical Care, 11:68-75, Supplement, March-April, 1973. Glass, G. V. Integrating findings: The meta-analysis of research. Review of Research in Education, 5:351-379, 1977. Goldberg, S.J., H. D. Allen, G. Freedman, K. Meredith, M. Tymrack, and A. Y. Owen. Use of health education and attempted dietary change to modify atherosclerotic risk factors: A controlled trial. The American Journal of Clinical Nutrition 33:1272-1278, June 1980. Grants for Research on Health Promotion and Disease Prevention, U.S. Department of Health and Human Services. Public Health Services. Office of Health Research Statistics and Technology, National Center for Health Services Research, 1980. Green, L. W. Evaluation and Measurement: Some dilemmas for Health Education. American Journal of Public Health, 67(2):155-161, Feb. 1977. Green, L. W. How to evaluate health promotion. Hospitals, 53(19): 106-8, October 1, 1979. Green, L. W. and I. Figa-Talamanca. The pre-hospital era of health edu- cation evaluation. Health Education Monographs 2(1):55-7l, Spring 1971. Healthy People: The Surgeon General's Report on Health Promotion and Disease Prevention, Washington, O.C., G.P.O., 1979, DHEW 79-5507 1 A. Knowles, M. The modern_practice of adult education andragogy_versus pedagogy. Association Press, New York, 1970. 155 Lauzon, R. An epidemiological approach to health promotion. Canadian Journal of Public Health, August 1977. Reprinted by Bureau of Health Education Focal Points, U.S. DHEW, Public Health Service, Center for Disease Control, Atlanta, Georgia, May 1979. Light, R. J. and P. V. Smith. Accumulating evidence: Procedure for resolving contraditions among different research studies. Harvard Education Review, 41(4):429-471, Nov. 1971. Longest, J. Designing evaluative research. Journal Extension, XIII: 48-55, March/April 1975. Machiavelli as quoted in Block, 0. Health promotion: Prospects for the future. The Second Katherine Boucot Sturgis Lectureship in Pre- ventive Medicine. Presented October 19, 1980, Detroit Michigan, American College of Preventive Medicine. McGill, A. M. Proceedings of the national conference on health promotion in occupational settings. U.S. Department of Health, Education and Welfare, Public Health Service, Office of the Assistant Secretary for Health, 1979. National Study of Secondary School Evaluation. Evaluative criteria. Washington, O.C.: Author, 1969, as reported in Sanders, James R. and Nafziger, Dean H. op. cit. Otterness, E. G. Bibliography in evaluation in health education. Group Health Plan, Inc. St. Paul SOPHE Presentation, September 12, 1980. Owne, A.Y. Community nutrition in preventive health care services: A critical review othhe literature. Health Planning Biographic Series #7, U.S. DHEW Public Health Service, Health Resources Admin- istration, May 1978. Owen, A. L. and G. M. Owen. Training public health nutritionists: Com- petencies for complacency or future concerns. American Journal of Public Health, 69(11):1096, Nov. 1969. Pillemer, D. B. and R. J. Light. Synthesizing outcomes: How to use research evidence from many studies. Harvard Education Review, 50(2):176-195, May 1980. Promoting Health/PreventipgrDisease Objectives for the Nation, U.S. Department of Health and Human Services, Public Health Service, Fall 1980. Quelch, J. A. The role of nutrition information in national nutrition policy. Nutrition Review, 35(11) November 1977. Rosenthal, R. Combining results of independent studies. Psychological Bulletin, 85:185-193, 1978. 156 Scriven, M. An introduction to meta-evaluation. Educational Product Report, 1969,_2, 36-38, as reported in Sanders, James R. and Nafziger, Dean H. op. cit. Smith, N. L., and Murray, S. J. Evaluation review checklist. Unpublish- ed manuscript, 1974, as reported in Sanders, James R. and Nafziger, Dean H., op. cit. Stake, R. E. A check list for rating an evaluation report. Unpublished manuscript, 1970, as reported in Sanders, James R. and Nafziger, Dean H., op. cit. Stufflebeam, D. L. "The Use of Experimental Design in Educational Evaluation," Journal of Educational Measurement. Vol. 8, No. 4, Winter 1971. Stufflebeam,D. L. Educational evaluation and decision-making in Worthen Blaine and J. Sanders (eds.): Educational Evaluation: Theory & Practice. Charles A. Jones, Publisher, 1973, p. 128-142. Stufflebeam, D. L. An administrative checklist for reviewingrevaluation plans. Unpublished manuscript, 1974. Turner, R. L. "Appendix G: Criteria for Comprehensive Evaluations and the Appraisal of Evaluation Success in Experimental School Contexts," in Guba, Egon 6.: Clark, David L.; McClellan, Mary C.: Sanders, James R.; and Turner, Richard L. The Design of Level III Evaluation for the Experimental Schools Program. A report presented to the U.S. Office of Education. Project No. R020862; Grant No. OEG 0-72-1867, September 30, 1972. Wandt, W. Form for the evaluation of an article, reprinted in Issac S, and Michael W. B. Handbook in Research and Evaluation for Education End the Behavioral Science, 5. Francisco Robert K. Knapp Puo, 36- 9, 1974. Wang, V. L. Changing nutritional behavior by aides in two programs. Journal of Nutrition Education, 9(3):107-113, July-Sept. 1977. Wilson, C. and S. Knox. Methods and kinds of nutrition education (1961- 1972): A selected annotated bibliography. Journal of Nutrition Education, Vol. 5, No. 1, Supplement 2, Jan-March 1973. Winikoff, 8. Nutrition and food policy: The approaches of Norway and the United States. American Journal of Public Health, 67(6):552-7, June 1977. Worthen, B. and J. Sanders. (eds.). Educational Evaluation: Theoryrand Practice. Charles A. Jones, Publisher, 1973. 157 Worthen, B. R. A look at the mosaic of educational evaluation and accountability. Research, Evaluation, and Development Paper Series. Portland, Oregon: Northwest Regional Educational Labor- atory, as reported in Sanders, James R. and Nafziger, Dean H., op. cit. GENERAL REFERENCES Anderson, Scarnia B. and Samuel Ball. The Profession and Practice of Program Evaluation. Jossey-Bass Publishers, Sanfrancisco, Wash- ington, Londan, 1978. Bass, Mary Ann, Lucille Wakefield, and Kathryn Kolasa. Community Nutri- tion and Individual Food Behavior. Burgess Publishing Company, Minneapolis, Minnesota, 1979. Borg, Walter R. and Meredith D. Gall. Educational Research: An Intro- duction. David McKay Company, Inc., New York, 1974. Campbell, Donald T. and Julian C. Stanley. Experimental and Qpasi-Ex- perimental Designs for Research. Rand McNally College Publishing Company, Chicago, 1966. Frankle, Reva T. and Anita Yanochik Owen. Nutrition in the Community. The Art of Delivering Services. The C. V. Mosby Company, St. Louis, 1978. Glass, Gene V. and Julian C. Stanley. Statistical Methods in Education and Psychology. Englewood Cliffs, N.J.:Prentice Hall, Inc., 1970. Mitchell, Helen S., Henderika J. Rynbergen, Linnea Anderson and Marjorie V. Debble. Nutrition in Health and Disease. J.B. Lippincott Company, Philadelphia, 1972. Wiersma, William. Research Methods in Education. F. E. Peacock Pub- lishers, Itasca, Illinois, 1976. 158 11111111111111“