OVERDUE FINES: 25¢ P'r day per item RETURNING LIBRARY MATERIALS: Place in book return to remove charge from circulation recorct :2 /.-:»‘- I DEVELOPMENT AND APPLICATION OF AN EVALUATION PLAN FOCUSING ON TEACHER IMPLEMENTATION OF INNOVATIVE PROGRAMS BY Sandra Kay Meacham A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Education 1980 ABSTRACT DEVELOPMENT AND APPLICATION OF AN EVALUATION PLAN FOCUSING ON TEACHER IMPLEMENTATION OF INNOVATIVE PROGRAMS BY Sandra Kay Meacham Often times the success or failure of innovative programs in education is judged solely by student achieve- ment with little or no consideration for the extent to which teachers implement the program. Gaining an understanding of implementation can facilitate interpretation of student out- comes, as well as identify aspects of implementation which need to be modified to improve future implementation of that program. The purpose of this investigation was to develop an evaluation plan which focuses on teacher implementation. The plan includes procedures for: measuring teacher imple- mentation; measuring student outcome; determining the rela- tionship between student outcome and teacher implementation; and, identifying factors related to teacher implementation. The evaluation plan was applied to a career/leisure program being implemented by five teachers in five schools in Michigan. The intent of this application was to determine the feasibility and practicality of the evaluation plan. Sandra Kay Meacham Nine teacher implementation variables for the career/leisure program were identified. The variables were measured by a teacher interview and by self report methods. The interview instrument contained twelve questions. The questions included yes and no response and open-ended response formats. In addition, each teacher was observed in order to cross-validate information collected by the interview and self report methods. Scores obtained for each variable were transformed to standard z-scores. The overall teacher implementation score was the average of the standard scores. Student outcome was measured by a 60 item cognitive skills test administered on a pre-post basis to both a treatment group (N==212) and a control group (N==94). Independent t-tests on the pre and post test scores were calculated to determine whether there was a significant difference between the treatment and control groups at the beginning and conclusion of the program. The relationship between student outcome and teacher implementation was determined by a Pearson product-moment correlation analysis. A total of 1? factors were investigated. To deter- mine the extent to which each factor was present across the five teachers, an interview technique was used. Teachers, principals, and the project director were interviewed. The interview instruments contained both scaled response items Sandra Kay Meacham and Open-ended items. Each teacher was given a rank score for each factor. A Pearson product-moment correlation coefficient was calculated comparing the rank scores for each factor to teacher implementation scores. Implementation of the career/leisure program varied from one teacher to another. The student outcome measure did not relate to the overall teacher implementation score, but did relate to one implementation variable, time spent on implementation (r==+.60). Six factors related to teacher implementation: perceived value of teacher inservice; communicability of the program; complexity of the program; principal support of program; perceived value of the pro- gram; and, perceptions regarding Career Education. The evaluation plan did provide a measure of teacher implementation and did identify factors relating to imple- mentation. The data collection methods which were used provided a wealth of information in a relatively short period of time. Problems inherent in the plan were identified. Recommendations for improving the plan are provided. DEDICATION : To my AonA, Shan and Monday: Byline ii ACKNOWLEDGMENTS A special thanks to Dr. Phillip Reuschlein for his counsel and pragmatic advice throughout the develOpment of this dissertation. Also, a special thanks to Dr. Eric Gordon who provided invaluable assistance in the designing and implementation of this study. This dissertation is actually a result of several people who were willing to devote their time and to provide constructive advice. Especially helpful were members of my doctoral committee, Drs. Larry Carmichael, Robert Ebel, Richard Featherstone, Eric Gordon, and Phillip Reuschlein. I thank my family and friends for their words of encouragement throughout the completion of my doctoral program. I especially thank my sons, Shan and Brendan, who willingly accepted me devoting several nights and weekends in order to complete my program. Their love and understanding is greatly appreciated. iii TABLE OF CONTENTS LIST OF TABLES O O O O O O O O O O O O O O O 0 LIST OF FIGURES . . . . . . . . . . . . . . . . Chapter I. II. III. THE PROBLEM O O C O O O O O O O C O O 0 Overview of the Implementation Process Need for the Study . . . . . . . . . Purpose of the Study . . . . . . . . Study Plan . . . . . . . . . . . . . Overview of the Dissertation . . . . Definition of Terms . . . . . . . . . REVIEW OF RELATED LITERATURE . . . . . Measuring Teacher Output . . . . . . Overview of the Studies . . . . . Procedures Used to Define Teacher Output . . . . . . . . . . . . Measurement and Data Analysis Procedures . . . . . . . . . . Implication of Major Findings . . Measuring Factors Relating to Teacher Output . . . . . . . . . . . . . . Project Directive Factors . . . . Incorporation Factors . . . . . . Summary . . . . . . . . . . . . . . . METHOD OF STUDY 0 O O O C O O O O O O 0 Introduction . . . . . . . . . . . . An Overview of the Career/Leisure Project Activities . . . . . . . . Description of the Career/Leisure Program . . . . . . . . . . . . Description of th Implementation Setting . . . . . . . . . . . . A Description of the Evaluation Design iv Page vii ix 10 11 13 14 16 17 17 19 26 31 32 33 40 44 46 46 47 50 55 56 Chapter Page Phase I: Evaluation of Teacher Output . . . . . . . . . . . . . . . 58 Phase II: Evaluation of Student Outcomes . . . . . . . . . . . . . . 62 Phase III: Identification of Factors Relating to Teacher Output . . . . . 65 IV. ANALYSES AND RESULTS . . . . . . . . . . . . 71 Results Obtained for Phase I: Evaluation of Teacher Output . . . . . . . . . . . . 72 Summary of Teacher Output Measures . . 83 Results Obtained for Phase II: Evaluation of Student Outcomes . . . . . . . . . . . 84 Student Outcome Results . . . . . . . . 84 Determining the Relationship Between Teacher Output and Student Outcome Measures . . . . . . . . . . . . . . 87 Summary of Student Outcome Analyses . . 89 Results Obtained for Phase III: Identi- fication of Factors Relating to Teacher output 0 O I O O O O O O O O O I O O O O 90 Part I: Results Obtained for Project Directive Factors . . . . . . . . . . 90 Analysis of the Relationship Between Project Directive Factors and Teacher Output . . . . . . . . . . . 97 Part II: Results Obtained for Incorporation Factors . . . . . . . . 98 Analysis of the Relationship Between Incorporation Factors and Teacher output 0 O O O O I O I O I I O I O 0 ll 1 A Discussion of the Appropriateness of the Evaluation Plan . . . . . . . . . . . 113 V. SUMMARY AND RECOMMENDATIONS . . . . . . . . . 117 Summary of Phase I: Evaluation of Teacher Output . . . . . . . . . . . . . 119 Summary of Phase II: Evaluation of Student Outcome . . . . . . . . . . . . . 124 Summary of Evaluation--Phase III: Identification of Factors Relating to Teacher Output . . . . . . . . . . . 125 Conclusions Regarding the LEAP Curriculum Evaluation . . . . . . . . . . 131 Conclusions Regarding Teacher Output . 131 Chapter Appendix Conclusions Regarding Student Outcome . . . . . . . . . . . . . . Conclusions Regarding Factors Relating to Teacher Output . . . . Conclusions Regarding the Evaluation Plan . . . . . . . . . . . . . . . . . Conclusions Regarding Teacher Output Conclusions Regarding Student Outcome Conclusions Regarding Factors Relating to Teacher Output . . . . Recommendations Regarding Future Implementation of the LEAP Curriculum . Recommendations Regarding the Evaluation Plan . . . . . . . . . . . . . . . . . Implications for Future Studies on Implementation . . . . . . . . . . . . A. A DESCRIPTION OF THE LEAP CURRICULUM . . . B. INSTRUMENTS FOR MEASURING TEACHER OUTPUT . C. STUDENT COGNITIVE SKILLS TEST . . . . . . . D. INSTRUMENTS USED TO MEASURE FACTORS . . . . REFERENCES vi Page 132 133 134 134 135 135 136 138 140 143 145 159 156 165 Table l. 2. 10. 11. 12. LIST OF TABLES A Summary of Studies on Teacher Output . . An Overview of Major Findings/Theories for Project Directive Factors Relating to Teacher Output . . . . . . . An Overview of Major Findings/Theories for Incorporation Factors Relating to Teacher Output . . . . . . . . . . . Distribution of Treatment and Control Students 0 I O O O O I O O 0 Teacher Output Score for the Extent to Which the Leisure Concepts Were Implemented Teacher Output Score for the Extent to Which the Leisure Concepts Were Integrated . Teacher Output Score Relating to the Number of Learner Activities Implemented per Target Area . . . . . . . . Teacher Output Score for the Extent to Which the Teachers Involved the Community in the LEAP Program . . . . Teacher Output Score for the Which Students Were Involved and/or Decision Making . . . Teacher Output Score for the Which Students Were Involved Awareness Activities . . . . Teacher Output Score for the Which Students Were Provided Extent to in Planning Extent to in Self- Extent to an Opportunity to Investigate Leisure Resources . . . . . Teacher Output Score for the Implementing . . . . . . . . vii Time Spent Page 20 35 42 64 75 77 78 79 81 82 83 84 Table 13. 14. 15. 16. 17. 18. 19. 20. 21. Summary of Teacher Output Scores . . . . Results Obtained for Control/Treatment Pre Test Analysis . . . . . . . . . . . . . Results Obtained from Control/Treatment Post Test Analysis . . . . . . . . . . . The Average Student Outcome Gain Score for Each Teacher . . . . . . . . . . . . Results Obtained for Correlation Analysis of Teacher Output Variables and Student Gain Scores 0 O O I O O O O O C C I O 0 Teacher Ranked Scores on Project Directive Factors and Correlation with Teacher Output Teacher Responses to Statements Regarding the Value of the LEAP Program . . . . . Teachers' Responses to Statements Relating to Perceptions of Career Education . . . Teacher Ranked Scores on Incorporation Factors and Correlation to Teacher Output viii Page 85 86 87 88 89 98 103 110 112 Figure 1. LIST OF FIGURES Page Components of the Implementation Process and Their Inter-Relationship . . . . . . . . 4 General Procedures for Evaluating an Innovative Program . . . . . . . . . . . . . 57 ix CHAPTER I THE PROBLEM Educational institutions across the nation are continually experimenting with innovative programs in an attempt to meet the demands made by societal change. Accompanying the implementation of most of these innova- tive programs is the charge to measure their impact. The focus of the impact has conventionally been student outcomes with little consideration being given to the actual use of the innovation, or what the innovation consists of in practice. Implementation of innovative programs has often been treated as a "black box" into which new programs are entered and from which outcomes such as student achievement emerge. To totally understand the impact made by an inno- vation, it is necessary to conceptualize, operationalize, and measure directly the implementation process (Charles & Jones, 1973; Fullan & Pomfret, 1977; Hall & Louckes, 1977). By studying implementation, educational researchers and practitioners would know what had actually changed; they would be able to understand better why so many educational changes fail to become established, and they would be more able to interpret learning outcomes (Fullan & Pomfret, 1977; Hall & Louckes, 1977). The central focus of this study is on evaluating the implementation of innovative programs. Overview of the Implementation Process Implementation is not synonymous with adoption. According to Fullan and Pomfret (1977), adoption of an innovation refers to decisions regarding the use of, as well as plans for, implementing the program. Implementation does not assume that an inno- vation is defined in advance by developers and then disseminated to several schools. It merely says that regardless of who develops an innova- tion, when it is developed, or how it is developed, some implementation will have occurred at that point when certain new characteristics are actually in use in a social system. (Fullan & Pomfret, 1977, p. 336) For quite some time, a major focus of educational studies has been on the change process and the adoption of innovative programs (Havelock, 1971; Rogers, 1962; Rogers & Shoemaker, 1971; Zaltman et al., 1977). Recently, several studies have focused more on the innovation itself. More specifically, these studies have analyzed how the innovation is changed during implementation (Berman & Pauly, 1975; Hall & Louckes, 1976). According to Hall and Louckes (1978), there has been an increase in awareness that "change is a process and not an event and that in the course of this process a multitude of variables interact to change the innovation" (Hall & Louckes, 1978, p. 4). In an effort to explain the process of ad0pting and implementing an innovative program, four interrelated conceptual components have been identified by the author of this study. A diagram and an explanation of the four components and their relationships is presented in Figure 1. Project directives. The process for adopting an inno— vation has been the focus of many research studies on change (e.g., Crowther, 1972; Havelock, 1971; Rogers & Shoemaker, 1971; Zaltman et al., 1977). As a result, there have been numerous theories offered which describe recommended pro- cedures for changing individuals, organizations, or systems (Zaltman et al., 1977). The notion of deliberate and con- scious change implies that at some point strategies for effective change are selected or created. Project direc- tives refer to the strategies used to plan for implementa- tion of the innovation (e.g., Was there a need established for the innovation? Who was involved in the selection and planning process?) Incorporation. From a study conducted by the Rand 'Corporation (Berman & McLaughlin, 1975), which included an extensive analysis of policy and system practice relative to the implementation of federal change agent projects, emerged a concept significant to understanding imple- mentation as a change variable. This concept, termed as "mutual adaptation," was defined by the authors as .mflanOwumHomluoucH Mamas can wmoooum coaumucoEonEH on» mo mucmcomeoo .H onsowm _.L.|I ell omcmso cowucucmSmHQEH mmmoonm mmoooum mswccmHm H0H>c£mm III-II Mo 6958 illll 05.3302 .525 ,I, coaumuswEmHmEH “mosouuso usmosum "usmuso Moaomma “cowumuomuoucH "mo>fiuoouwo uoonoum H Juli a .mflanOAUMHomluousH mamas can mmoooum coaumucoEmHmEH on» no mucmcomaou .H ousmwm . Ilrj wooden coaumucmsmHQEH mmmooum mmmooum mcflccmHm uoflfinom IIIII! mo menace illlll mcwumog was: all cowumunwEmeEH “mweoouso unmpsum "usmuso nuances "coaumuomuoocH "mw>wuomuwa uomnonm 4 jl t. "an organizational process in which an innovative plan is developed and modified in light of the realities of the instructional setting, and in which the organization changes to meet the requirements of the innovative project" (Berman and McLaughlin, 1975, p. 31). Prior to implementation, the selected teachers or users of the innovation will go through a process whereby they will incorporate or adopt the inno- vation into their existing organizational environment. The teachers who are to implement the program possess character- istics (such as knowledge, skills, and values) which may vary from one teacher to another. In the same sense, school environments differ from one school to another. For example, the support given by a principal or fellow teacher for a specific change may vary from one school to another. Teacher output: The Development Center for Teacher Education at the University of Texas, in 1975, conducted a study (Hall & Louckes, 1978) which established that eight Levels of Use of innovations exist. This research was based on the assumption that a variation in use by each individual exists, and the research supported the hypothesis that individuals generally move from "non-use" through a "routine Level of Use" in a sequential pattern. It cannot be assumed that each teacher will implement the innovation according to the project objectives or instruct in the manner prescribed by a particular innovation. Teacher output for this study represents a measure of the teachers' behavior while implementing the innovation. Student outcomes. This term refers to the impact made on the students by the innovation as it was implemented. Included in student outcomes would be changes in knowledge concerning the innovation, changes in affective behavior, and/or change in psychomotor abilities. The diagram presented in Figure 1 yields three ques- tions of concern to this investigation on implementation. What is the relationship (or does an operative feedback loop exist) between: 1. Teacher output and project directive factors? 2. Teacher output and incorporation factors? 3. Teacher output and student outcomes? Need for the Study William Cooley (1978) expressed a concern that too often research endeavors in education view the treatment as being distinct and discrete, "worthy of consideration as levels in a typical analysis of variance contrast" (Cooley, 1978, p. 1). He saw a need to view the edu- cational treatment as being multi-dimensional. "The treatment dimensions reflect the various ways in which a program can be implemented by different teachers, as well as other important ways in which the teachers and programs vary" (Cooley, 1978, p. 1). Evaluation of innovative programs must include observation and documentation of the various ways in which a program is being implemented. It cannot be assumed that all teachers are implementing a program in a homogeneous fashion. Furthermore, the study of implementation is crucial to every innovative program in order to more accurately interpret student outcomes. If student outcomes are the sole source of information determining the success or failure of that program, and are not interpreted in light of the implementation, it is possible to be measuring "non events" (Charters & Jones, 1973). In studying implementation, not only must teacher output be measured, but factors which account for differing degrees of teacher output also must be identified. Knowl- edge of these factors could improve teacher output in one or all of three ways: (1) the project director could use directives which would most likely facilitate teacher out- put; (2) the project director could select those schools and teachers where implementation of an innovative program is most apt to be successful; and, (3) the project director could attempt to change characteristics of the adopting unit prior to implementation in order to assure success. The literature reveals very few studies in which factors that affect teacher output have been identified, and several of these studies have been criticized for not adequately measuring teacher output (Gross et al., 1971; Fullan & Pomfret, 1977). Gross et a1. (1971) found the literature relating to implementation of innovations to be deficient in several important aspects. First, there has been little concern for testing theories or generating testable hypotheses about factors influencing degree of implementation. Secondly, data used to isolate conditions having an impact on implementation are typically obtained only from the perspective of those who initiate them; they generally ignore the point of view of organization members who must make the behavioral change specified by the innovators. And, thirdly, the method used to assess the degree of implemen- tation of an innovation is open to serious ques- tions; careful measures would require collecting and analyzing data based on systematic observa- tion and not using data about "effects" as indices of successful implementation. (Gross et al., 1971, pp. 35-36) One reason why implementation has not been included in the evaluation of innovative programs could be that a review of existing evaluation models revealed inadequate methodologies for obtaining and analyzing information on implementation. Although several authors discussed the need for curricular evaluation to provide information for making decisions regarding future implementation of the program being evaluated (Cronbach, 1963; Hastings, 1966; Provus, 1971; Stake, 1969; Stake, 1967, Stufflebeam, 1968), none of the authors related directly to the measurement of teacher output. Stake (1967) and Stufflebeam (1968) expressed the need for observation and documentation of variables existing prior to, and during, implementation which might relate to student outcomes. Neither author, however, provides clearly defined procedures for measuring teacher output or the aforementioned variables. Further- more, no guidelines are provided for determining the extent to which selected variables affect teacher output and the extent to which teacher output affects student outcomes. Hall and Louckes (1978) developed a procedural model for measuring teacher output by identifying configurations of an innovative program. This model provides the evaluator with means for observing teacher output of an innovative program over time and for documenting its changes. But, it does not relate teacher output to student outcomes; nor, does it contextually relate teacher output to project directives. Another drawback to this model is that it requires a great deal of time and a large number of teachers in order to identify the configurations. This time and money is most often not available to an evaluator who is charged with evaluating a program at a local district level. Furthermore, many evaluations of innovative programs are conducted during a pilot test of the innovation where the number of implementing teachers is intentionally kept to a minimum. 10 Purpose of the Study The purpose of this study is to develop an evaluation design that focuses on teacher output and factors that affect teacher output of innovative programs and to deter- mine the feasibility and appropriateness of the evaluation design. The evaluation design will focus on the various components of the implementation process. Specifically, the evaluation will be designed to determine the extent to which teacher output relates to student outcomes; and, to indicate the relationship between teacher output and project directive and incorporation factors. The evaluation design will include three phases. The objectives of each phase are as follows: Phase Objectives 1. An Evaluation of 1.1 To obtain a measure for each teacher Teacher Output on the extent to which they implemented. 1.2 To provide a description of program components as they were implemented. 1.3 To identify problems teachers encountered during implementation. 2. An Evaluation of 2.1 To determine whether or not behavior Student Outcome changes as a result of the program. 2.2 To ascertain the relationship between the student outcome and teacher output measures. 3. The Identification 3.1 To provide a description of project of Factors that directive and incorporate activities Relate to Teacher surrounding implementation. Output 3.2 To indicate the relationship between teacher output and project directive factors. 3.3 To determine the relationship between teacher output and incorporation factors. 11 Study Plan To determine the feasibility and appropriateness of this evaluation plan which was designed to test the interrelationship of the components of the implementation process, the plan was used to evaluate a career/leisure program. The career/leisure program to be evaluated was implemented by five teachers in a rural intermediate school district in Michigan through a Career Education project funded by the Michigan Department of Education. The decision to evaluate a career/leisure program was a pragmatic one. First of all, the director of this career/leisure program requested that an evaluation be conducted which would provide information for making decisions regarding future implementation of this program. Second, the implementation conditions of the program were typical of many evaluations of innovative programs in that the program was being pilot tested using a small number of teachers (N==5). Evaluation of an innovation under these circumstances served as a good indication of the feasibility and practicality of conducting such an evaluation. To evaluate teacher output, implementation variables were identified. To measure the variables, three data collection methods were employed: interview, self report, and classroom observation. The data collected for each 12 teacher output variable was analyzed resulting in a numerical teacher output score for each teacher. All teacher output variable scores were converted to standard 2 scores. The overall teacher output score was an average of the variable scores. The student outcome measure used to evaluate the career/leisure program was a cognitive skills test. The test was administered to a treatment and control group of students on a pre-post basis. To determine if there was a significant difference between the two groups at the onset and conclusion of implementation, independent t-tests were calculated. To ascertain the extent to which the student outcome measure related to the teacher output measure, a Pearson product moment correlation coefficient was calcu- lated using the average student gain score obtained for each teacher and the overall teacher output score. The project directive and incorporation factors which might affect teacher output were identified based on a review of related literature and a review of the career/ leisure program. The factors identified were measured by extensive interviewing. The teachers were rank ordered for each factor based on the descriptive data provided by 'the interview. To determine the relationship between each factor and teacher output, the overall teacher output score imas converted to a ranked score. A Pearson product moment 13 correlation coefficient was calculated on the ranked scores. Application of this evaluation design provided information regarding the extent to which it adequately measured implementation and identified factors affecting implementation. Application of this evaluation design to this career/leisure program also yielded valuable informa- tion regarding implementation and factors affecting imple- mentation of this program. This information serves as a knowledge base for future research endeavors in career education. Overview of the Dissertation The previous discussion has pointed out the need for evaluation of innovative programs in order to incorporate means for analyzing teacher output and determining its relationship to student outcomes, project directive factors, and incorporation factors. Chapter II provides a review of previous research relating to teacher output and factors affecting teacher output. Chapter III presents a description of the evaluation plan including data collection and analyses procedures used to measure teacher output, student outcomes, project direc- tives, and incorporation factors. The evaluation plan is described as it was applied to the evaluation of a career/ leisure program. 13 correlation coefficient was calculated on the ranked scores. Application of this evaluation design provided information regarding the extent to which it adequately measured implementation and identified factors affecting implementation. Application of this evaluation design to this career/leisure program also yielded valuable informa— tion regarding implementation and factors affecting imple- mentation of this program. This information serves as a knowledge base for future research endeavors in career education. Overview of the Dissertation The previous discussion has pointed out the need for evaluation of innovative programs in order to incorporate means for analyzing teacher output and determining its relationship to student outcomes, project directive factors, and incorporation factors. Chapter II provides a review of previous research relating to teacher output and factors affecting teacher output. Chapter III presents a description of the evaluation plan including data collection and analyses procedures used to measure teacher output, student outcomes, project direc- tives, and incorporation factors. The evaluation plan is described as it was applied to the evaluation of a career/ leisure program. 13 correlation coefficient was calculated on the ranked scores. Application of this evaluation design provided information regarding the extent to which it adequately measured implementation and identified factors affecting implementation. Application of this evaluation design to this career/leisure program also yielded valuable informa- tion regarding implementation and factors affecting imple- mentation of this program. This information serves as a knowledge base for future research endeavors in career education. Overview of the Dissertation The previous discussion has pointed out the need for evaluation of innovative programs in order to incorporate means for analyzing teacher output and determining its relationship to student outcomes, project directive factors, and incorporation factors. Chapter II provides a review of previous research relating to teacher output and factors affecting teacher output. Chapter III presents a description of the evaluation plan including data collection and analyses procedures used to measure teacher output, student outcomes, project direc- tives, and incorporation factors. The evaluation plan is described as it was applied to the evaluation of a career/ leisure program. 13 correlation coefficient was calculated on the ranked scores. Application of this evaluation design provided information regarding the extent to which it adequately measured implementation and identified factors affecting implementation. Application of this evaluation design to this career/leisure program also yielded valuable informa- tion regarding implementation and factors affecting imple- mentation of this program. This information serves as a knowledge base for future research endeavors in career education. Overview of the Dissertation The previous discussion has pointed out the need for evaluation of innovative programs in order to incorporate means for analyzing teacher output and determining its relationship to student outcomes, project directive factors, and incorporation factors. Chapter II provides a review of previous research relating to teacher output and factors affecting teacher output. Chapter III presents a description of the evaluation plan including data collection and analyses procedures used to measure teacher output, student outcomes, project direc- tives, and incorporation factors. The evaluation plan is described as it was applied to the evaluation of a career/ leisure program. 13 correlation coefficient was calculated on the ranked scores. Application of this evaluation design provided information regarding the extent to which it adequately measured implementation and identified factors affecting implementation. Application of this evaluation design to this career/leisure program also yielded valuable informa- tion regarding implementation and factors affecting imple- mentation of this program. This information serves as a knowledge base for future research endeavors in career education. Overview of the Dissertation The previous discussion has pointed out the need for evaluation of innovative programs in order to incorporate means for analyzing teacher output and determining its relationship to student outcomes, project directive factors, and incorporation factors. Chapter II provides a review of previous research relating to teacher output and factors affecting teacher output. Chapter III presents a description of the evaluation plan including data collection and analyses procedures used to measure teacher output, student outcomes, project direc- tives, and incorporation factors. The evaluation plan is described as it was applied to the evaluation of a career/ leisure program. 14 The results yielded by applying the evaluation plan to the evaluation of the career/leisure program are addressed in Chapter IV. The findings are presented in tabular form and discussed in terms of teacher output measures, student outcomes, and factors relating to teacher output. Chapter V is directed to integrating the results of this study, drawing conclusions, and discussing implications of the findings. This chapter focuses on the extent to which the evaluation plan appropriately evaluates teacher output and student outcome and determines the relationship between teacher output and student outcome, project direc- tive factors, and incorporation factors. Chapter V also centers on the implications of the evaluation results obtained relative to the career/leisure program being evaluated. Definition of Terms To facilitate the understanding of this study, the following terms are defined: 1. Innovation: Rogers and Shoemaker (1971) concisely and precisely defined an innovation as being an idea, practice, or object perceived as new by an individual. It matters little, so far as human behavior is concerned, whether or not an idea is "objectively" new as measured by the lapse of time since its first discovery. It is the perceived or 14 The results yielded by applying the evaluation plan to the evaluation of the career/leisure proqram are addressed in Chapter IV. The findings are presented in tabular form and discussed in terms of teacher output measures, student outcomes, and factors relating to teacher output. Chapter V is directed to integrating the results of this study, drawing conclusions, and discussing implications of the findings. This chapter focuses on the extent to which the evaluation plan appropriately evaluates teacher output and student outcome and determines the relationship between teacher output and student outcome, project direc- tive factors, and incorporation factors. Chapter V also centers on the implications of the evaluation results obtained relative to the career/leisure program being evaluated. Definition of Terms To facilitate the understanding of this study, the following terms are defined: 1. Innovation: Rogers and Shoemaker (1971) concisely and precisely defined an innovation as being an idea, practice, or object perceived as new by an individual. It matters little, so far as human behavior is concerned, whether or not an idea is "objectively" new as measured by the lapse of time since its first discovery. It is the perceived or 15 subjective newness of the idea for the individual that determines his reaction to it. If the idea seems new to the individual, it is an innovation. (Rogers & Shoemaker, 1971, p. 19) 2. Adoption: Adoption is referred to in this study as being a decision point regarding the use of the innova— tion. Adoption also includes planning for implementation of the innovation. 3. Evaluation plan. An evaluation plan includes a set of procedures for determining the extent to which a program is successful. The procedures and variables included in an evaluation design may differ one from another, depending on the purpose of that evaluation and philosophy supporting that design. CHAPTER II REVIEW OF RELATED LITERATURE Presented in Chapter I was an overview of implementation that depicted implementation as a change process involving the following components: project directives--implementation planning process; incorpo- ration--unit adopting process; teacher output-~degree of implementation; and student outcome--behavior change. The purpose of this study was to develop an evaluation design that focuses on implementation of an innovative pro- gram and to determine the appropriateness of the design by applying it to the evaluation of a career/leisure program. This design is to provide procedures for measuring teacher output and for identifying and measuring factors (project directives and incorporation) which might affect teacher output. To gain a better understanding of procedures to be incorporated into this evaluation design, several studies on implementation were reviewed. Chapter II contains two sections. The first section presents a discussion of pro— cedures used in, and outcomes of, studies measuring teacher output. The second section focuses on studies identifying factors (project directive and incorporation) affecting teacher output. 16 17 Measuring Teacher Output The intent of this section of the related literature review is to create a clear understanding of teacher output and how it can be measured. Included in this section of Chapter II is a review of five studies that have focused on implementation (teacher output) of innovative programs in education. Although this section is not inclusive of all studies conducted on implementation, the studies that have been included were selected based on the quality of the methodology used to measure teacher output, and the con— tribution they have made to implementation research. An overview of the five studies follows. Overview of the Studies The five studies on implementation research represent a variety of educational innovations. One study, conducted by the Rand Corporation (Berman & Pauly, 1975), involved 293 change agent projects and addressed teacher output in a series of federally funded programs. The purpose of this study was to answer the following research questions: 1. To what extent did differences among the federal change agent programs explain variation in teacher output? 2. What characteristics of an innovation affect teacher output? 18 3. Were differences in instructional settings related to variations in teacher output? A second study, conducted by Crowther (1972), focused on Social Science curriculum. The purpose of this study was to identify specific teacher variables which influence the extent to which teachers are implementing. This study involved 317 teachers who were implementing the Social Science curriculum. Gross et a1. (1971) looked at teacher output of an educational process ("Teacher as a Catalyst") over a year's time to determine where breakdowns in teacher output begin to occur, and to identify what barriers to teacher output exist. An in-depth case study approach involving 10 teachers in one school was used. The purpose of a fourth study, conducted by Hall and Louckes (1977), was to determine if teacher output of innovative programs changed over time. The hypothesis tested was that various Levels of Use of an innovative Program are related to the length of time a user has implemented the program. A total of 1,381 teachers representing three innovative programs were included in this study. The teachers were grouped according to the length of time they had been implementing the program. The fifth study was conducted by Lienhardt (1977). Like Hall and Louckes, Lienhardt also was interested in 19 determining if teacher output of innovative programs alters after three years of implementation. Seven follow-up schools and three area schools in Pittsburgh who were implementing the individualized Math and/or Reading program for the two preceding years were included in this study. A summary of the procedures used to define, measure, and analyze teacher output, and the major findings regarding teacher output are presented in Table 1. A discussion of this table follows. Procedures Used to Define Teacher Output Although each study was concerned with the extent to which an innovation was implemented as intended, how implementation was perceived prior to defining and devel- oping measures varied from one study to another. Fullan and Pomfret (1977) conducted an extensive review and analysis of implementation research and concluded that one of the most complex issues regarding the development of instruments to measure implementation relates to per- spectives on implementation and how it is defined. The authors further concluded that there have been basically two approaches taken to define implementation. One approach defines implementation as a product with a predetermined set of teacher behaviors. Success of implementation, in this ...-.-. u. . 5.. \A C o.-.luo-p.. .0. .Q'fh. Iu:§ .I-( 20 .susua was» an nouauocq ouo) ouuonOun unooo omnonu na~ uo money 4 «noduoucosonmsu louooum cw unawuowuo> on oeuonou coca ou oacauuoo Moccauoouuonu ca ooono unwed coo Iuouuao ouo: An. «cauuoucoiounia >H oauwa .xcsuo onu neon» uoouuo nauuo>occa no mo .cfl .oo> cw cocaaocw occauo>oncu uo ouaunauouuouono yon: Au. «canyon HHH onuqe .o:o«u«ocou Houcolcou. .aonaaono cadmnouoou huowuo> on» noouoo com: on InoloHnlu on» c« ceauouuo> :«oamxo Amped. uw>co unauouxo ou unouo, oumwunsa concave“ oqoaaono ouon unous aonu uonu novuo cu oioumoam ucovo oonono flouovou >oasom noduo>onna onu uno woo: .vououuouuo no) uofiu0u oonoaoou onh Hoganm ouot coo: monsooofl on» acoao noonououuuc vac unouxo a noiuom on» noon ouon) sebaceous .cOAuoucqu—nla unannouou occaumosv Moan onh .>Haou«u«uono vocauoo 0on3 09 .u. ”occauoosu nuuoooou .avsum annuals o o« noduounolonnln ou onomoou cu vogue ouo: muoonnuo onh no: ouoa occuuo>onnu onh mauso~H0u onu coo-ouvoo xuouo sank menu one wmmuoquMMNNNMImmm “Hon no omouo>o no vouanona ouoanono ouoo .ausuo ounu nu onuuoox .ucwumo coo Haou one ca venououcaaoo ouo) .avsuu on» :« nonsense ouo) oaoonoo nuuanouuum uco noon .aoom doovu ovnoasuuona ona .ooanoauo> cm 0» vouou,ooosaona occauo>occa uo huouuo> oouo oounu and ouoonoo nansouaou cw aloha ouOI o vuozou ona>OI vco oooa ouo: :oMuounoEOumHu uo nomou on» ooouoo noon on unadl >onu no>om .vooconu non .nuooh oounu loam oooa unooononu con olouooum oooa> .oooauuoua soouaoono ou ocuuoaou uonu novuo :4 Honodm ouox coo: uouuo .oloumOun o>uuo>onca on» uuosvw>aucu an» ac coauuuaoaaaaaa on» nonnaqua> an vocauucoo acosuuuuca aoa> monsoon: one .aaaauauuuomn uo couuaucoauamaq on unannouoo .spoac .QOAuon uooauoounu o nose nuovoa onh ”noduo>uouno omo zoa>uounn cocauou ouo: occuuo>onna ona on conuwooc oo: ausuo odna ovuonnoaq .oaa «0 ado>oqnmmu luwunoo on voEHOUuom no) omcauou on» .Ioumoum nun: nco .uoooouonu ow Iouooum o unol coo oldu noduoucosoanaa uo nuance on» onu mcwuconoumla coon non >on» onuvooz noanla aonu noun: on unouuo nootuon ouo>uoso “occuuoaouuoo o .oo: no .conoaanouoo old» uo numnoa on» ou unuoaoooo: cu olouu onu .:0auo>onna on» mnuoo oao>oa unououuao uo cocoam ouo: ouonuooa ouo: co«uo>o::« nuoo uo voaoouu coo vozou>uouod ouo: Iona oooa :4 oocouuomxo ouos cwou .ouaooou 3oq>uoucw coho» one uo ucduou o ocuuouuouoouono uo«>onon muonooou Hon.H uo Houou n .olqu uaooou>uvnu ouonooo» on .cbuuounosoamld vou:~oc« naoxaono ouoo .:o«uo>oc:« on» uonooou douocou .uonuoz uo>o ooonono ma oiouooum o>auo>on, arsed. uo ooouoou no one no ouo>oq on unduosou osouu Houocoo cocaoucoo .>Haoo«uuoomo conquov Ind no noduounoloanlu ud ondluouov nonoooa cocauo> nooounu co ouonoooa unoauuuonq zou>uoucq one ”mummmmmmmm yon ouo: o:o«uo>onn« onh 0» no) >09». ounu no ooonusm one uno duo: \wmmmmmmmwm,ouo3 nooauoco o>uumauoooo .uooOOum sunaaouoUs Ioouoooau on» no uuquowuouoouonu mcanauooov .>o:uo can» ca covsauca ouo: onu nu gonad oiouu conooo uuoxuq unqoano>uu NH onu ouonooou ooh .n04uounoloumla uouou o no .uuonuo non» unoflougaa ou vascuuuav one! ouo: cowuo> ouo) oonaoucOO cod: .:o«uo>uoono onu unquav vounonoum ooduw>au0o nocuoo~ .:o«uo>oc:« on» no zou>ou o :6 vooon o» ououuuon unuuoaxo auuucoou ou moo choc nooun on onaoon noduou uonoooa: o mud-a .anmuc nocna on» no ounonomloo oiom uno ocuuuoo ioouoooao on» onauowoc ou .oouuwunoou ouo: ouwuou ccoioanla coon: ocwiuouoo o» uooa o :«ouocuon .30” ano: out couuounoaounia olouu nouocou 03» cocoaucd unoaahuoca uuouoouonu hawoonon nonoooa uo>o scavo>onna on» no noduouoolounlu a oucosvoao on» no auuuoov “Houobo onh onB .:6«8o>uommv loomuumdo uoouan .oo:o«>ou no: :0«uo>onnw one ooouu 00 no) ans»: own» no unouna onb .ooouo .mmaccudm on» an co>Ho>co ouo: oucovsuo on» noun: cu unouxo on» uco .ESHsuau .auo>~oco uo van: on» no: luau on» unsouo vocnoam coo). uonuooa .xoonu noduoucofiouaaa ocooooa noun) 0» ucouxo on» no ouou o no coo: not cool one .Ioumoum onu no oousuoou Hanoi .ouonuoou on» .osouu on» no oncomoou zoos onu no on» uooauou cu vo>uouuon ouo: .avouo oanu :« coosaona ouo: an xoamlou on o» oouovuo cauuouauHou o cooSHocu uwouuono ouoo ouonuoou onu nownz ou unouxo ouonooou run we Houoa n .voo:ou:« noon no: ceauo>ocna one .oo>~ooeonu vouch ouonooou ona .voo: on» no cowon no: noduounol no unaucosonlq ouo ouonooou oouvsum 55~904uuso onu uo canon no: lanooauuou on» mo nououoou once on» nounia uo unouxo ona .oOAuoq nuant cu uoouxo on» oononauna nown) «owoow ououoauoona on volaouo mnauuoauou oaou« monooo unwouno>wu ma uuouuouonu nonos on» ocwsuouoo oounoauo> nonooou oauuuomu >uuuno04 Aahaa. ouonooou on» we omn auco onunuoucoo ouwo::04uooav t "mmmmmmlmmmm ou ooaou>ou no: coauo>onnu one o» no: aosao man» no ooomuou ona uonuDOuu unease uonoooa oouovououm uoduoo nonoooa avsum on» we 3oa>uo>o occuuoaoncu acupuooom nonuvcuh homo: maoaaon< ouoo one ucosousaoo: oedema cu poo: monocOUOum .uonusn usnuso nonuooh co mononum uo >uo£lsm n .A ounos 21 instance, is determined by the degree to which the predetermined teacher behaviors are observed. A second approach to defining implementation is based on the philosophy that it is inappropriate to limit measures of implementation to a predetermined set of teacher behaviors because educational innovations require changes in teaching styles which, in turn, require new role rela- tionships. Studies which have taken this perspective have treated implementation as a "process," with the success of implementation determined not by the ability of teachers to perform in one particular way, but by the "extent to which they are able to recognize the range of behavioral alternatives Open to them, ascertain which ones are appli- cable in a given setting and change accordingly" (Fullen & Pomfret, 1977, p. 363). Fullan and Pomfret criticized the majority of the "process" studies for their lack of specific measures. The basic problem with the "process" approach is that, theoretically, no Operational definition of implemen- tation is developed while attempting to avoid limiting teachers to a predetermined set of behaviors. This approach would not permit a "degree of implementation" to be calcu- lated or permit implementation to be related to student outcomes. Furthermore, this open-ended approach would deter from relating project directives and incorporation factors to teacher output because implementation would not be quantified. 22 Two important points made by Fullan and Pomfret are: (1) innovation needs to be defined in order for implementation to be measured, and (2) implementation is going to change through time, thereby making it necessary to continually redefine implementation as teachers adapt the innovation to meet their environmental constraints. Fullan and Pomfret advise that measurement of imple- mentation be treated as a 'snapshot' of, what users are actually doing with respect to the innovation at any one point in time. It is important to consider that the nature of innovations in use may transform over time and that we need measurements to detect these changes in further specification, redefinition, or development. (Fullan & Pomfret, 1977, p. 367) Of the five studies reviewed, only one utilized the "process" approach in defining implementation (Berman & Pauly, 1975). The authors of this study did not establish a specific definition of any innovation included in their investigation, thereby resulting in very global measures of implementation. It is not known, however, whether the choice not to clearly define any of the innovations and to use global measures was based on a strong "process" philosophy as expressed by Fullan and Pomfret, or whether the choice was basically a pragmatic one in order to use the same measure of implementation across all innovations. Hall and Louckes (1977) appear to be somewhere in- between the "product" and "process" approach in defining 23 implementation. Most assuredly, they are in agreement with Fullan and Pomfret that implementation is in a continual state of alteration. At the same time, they perceive a need to systematically define teacher behaviors which requires a clear characterization of behaviors as defined by the innovation. In a later paper presented at the 1978 Annual American Education Research Association conference, Hall and Louckes further discussed the issue of defining implementation as a variable in a change process. They developed a procedural model for establishing "configurations" of an innovation which they defined as the "operational patterns of the inno- vations that result from selections and use of different innovation component variations" (Hall & Louckes, 1978, p. 10). Implementation of an innovative program often involves teachers selecting only parts of the program to implement and possibly combining the parts selected in ways different from what the developer had intended. The procedures for identifying the "configurations" are as follows: 1. The curriculum developers and onsite facilitators are interviewed for essential innovation compo- nents. (When this is not possible or is imprac- tical, materials are reviewed instead.) 2. A varied, but small, sample of users is interviewed and observed to determine how the innovation varies in use. The set of components and variations (step 1) is expanded. 24 3. A set of interview questions and probes is developed around the set of hypothesized compo- nents resulting from the first two steps and a sample of users is interviewed about how they use the innovation. 4. A checklist is constructed which lists the components of the innovation as noted by both developers/facilitators and users, and the component variations implemented by users are arranged under each component. The checklist can be completed from interview information or directly by the user/non-user. 5. Variations in checklist patterns can be analyzed and the data reduced across respondents to provide a manageable number of dominant configurations. Users are then assigned to dominant configurations based on checklist responses. (Hall & Louckes, 1978, p. 16) In their discussion of this model, Hall and Louckes point out the need for deciding the extent to which imple- mentation of the innovation can be changed and still be considered to be the innovation. They suggest that "perhaps innovations should be described in terms of critical compo- nents, a core of components which must be present for the innovation to be in use, and 'related components,‘ periph- eral components that are of secondary importance and do not determine whether the innovation is defined to be in use" (Hall & Louckes, 1978, pp. 28-29). Theoretically, the "configurations" model provides a sound means for defining implementation as a change variable. Unfortunately, its application is not practical in instances where there is a small number of teachers implenemting the innovation because a large number of 4‘ w,——._— 25 teachers would be required to provide a valid description of an innovation's configuration. Furthermore, Hall and Louckes cautioned that if an innovation is at the initial stages of implementation, implementation may appear to match what was intended because of the teacher's lack of awareness of alternatives to implementation. In which case, it was suggested the configurations be determined over a long period of time. The three remaining studies included in this review have defined implementation as a product. In each instance, the innovation was carefully reviewed to establish a definition of the expected teacher output. Based on the studies reviewed and the theoretical constructs presented by Fullan and Pomfret (1977) and Hall and Louckes (1978), it is concluded that teacher output must be carefully defined by observing inherent characteristics of the innovation itself, as well as by interviewing teachers and/or project directors. It is necessary, however, that teacher output be defined in specific enough terms that meaningful measures can be developed. At the same time, caution must be taken to mpdify a definition based on an analysis of environmental changes that present a need to adapt the innovation. 4.1 no. 26 Measurement and Data Analysis Procedures Three techniques for measuring teacher output were used in the studies reviewed: 1. Observation; 2. Self Report; and 3. Interview. The technique(s) selected and the instruments developed reflected the investigator's perceptions of implementation and the purpose of the study. Following is a discussion of how each method was used in the studies reviewed. 1. Observation: Two studies used observation as a means of determining the extent to which teachers were implementing the program (Gross et al., 1971; Lienhardt, 1977). In both instances, an observation instrument was developed and used by more than one individual, therefore requiring inter-rater reliability on the observation scale to be determined. Both studies focused on innovations that required a change in teacher-student role behavior. Mea- suring teacher output in both instances seemed to neces- sitate that an observation technique be used. Gross et a1. (1971) used direct classroom observation to measure teacher output. The instrument used included two general items asking the rater to describe the classroom setting and the learner activities presented. The second part contained twelve five-point Likert scaled items 27 describing characteristics of the catalytic role model. Classroom observations were carried out in a random order. The instrument was completed after and not during observa- tion in an effort to cause less distraction from the teacher and student normal behaviors. An overall rating then was calculated for each teacher. Lienhardt (1977) used videotapes to determine specific teaching practices being implemented. Tapes of each teacher were made in the fall and again in the spring by educational specialists. Then the tapes were coded by an evaluation group at the Learning Research and Development Center on 30 variables which related to the characteristics of the innovations. The stability of the variables measured by the videotapes was estimated at .58. Inter-coder reliability for the videotapes was estimated at .95. The observation instruments used in both studies reflect great care and thought in their construction. The instruments were developed based on characteristics of the innovation. 2. Self Report. Only one study used self report as a method of determining fidelity of implementation. The self report method used by Crowther proved to be a valid means of collecting implementation data. Crowther devel- oped a questionnaire containing 12 Likert scaled items which were based on a thorough analysis of the Social 28 Studies curriculum being investigated. A unique process was used to validate this self report instrument. Following administration of the instrument to 317 teachers, Crowther observed the teachers and students of 13 implementing classrooms. In addition, the 13 teachers, a sample of students, and the principal were interviewed to verify the results obtained on the self report instrument. The results of this validation process showed that the self report instrument provided valid data for determining the extent to which the teachers were implementing the Social Studies curriculum as intended. 3. Interview: The interview was the most commonly used technique for measuring teacher output in the five studies reviewed. However, the level of specificity of the interview items varied greatly from one study to another. An interview technique was used in the Rand Study to measure teacher output across several different inno- vations. The investigators in this study perceived imple- mentation as a “process" whereby changes in both the innovation and the implementors is continually occurring. Consequently, the interview instrument contained global items non-specific to any one innovation. The questions asked for a rating (using a four point scale) of the extent to which the innovation had caused a change in the way 29 things were done in the classroom, and cmf the extent to which the project had been implemented as intended. Teachers were also asked to estimate the percentage of project goals achieved. The interview was the only method used by Hall and Louckes (1977) to collect data regarding the extent to which teachers were implementing an innovation as intended. The interview instrument they used was not specifically related to the content of any one innovation, but rather was spec- ified according to the hypothesized Levels of Use. The interviewers who collected the data were trained to seek information relative to the overall Level of Use in general, and not in identifying the Level of Use of any one innova— tion in particular. The interviews, which took about 20 minutes, were taped. The tapes were rated for the Teacher's Level of Use for the innovation being implemented. Inter- rater reliability estimates were calculated to be from .87 to .96. Hall and Louckes (1977) considered the interview to be a valid way to document what was happening in the classroom settings and also to provide a means for col- lecting additional information from teachers. Once teachers began discussing their activities and concerns, other aspects of implementation were unexpectedly identified. In addition to the observation technique, Lienhardt (1977) obtained data on classroom practices via a structured In 'ri 30 interview approach. An evaluation group from the Learning Research and Development Center interviewed evaluation specialists assigned to each school included in the study. The teachers were involved in the interview process only where and when appropriate. The interview responses were then rated by the evaluation staff resulting in data on 33 variables of implementation. The generalizability of the variables measured by this interview technique was estimated at between .76 and .90. The re-coder reliability estimates for the interview were estimated at .98. Unfortunately, the relationship between the observation scale and the interview scale was not discussed. The interview technique, as used in these three studies, appears to be a useful method for collecting the information needed to measure teacher output as long as the interview instrument is structured in such a way that it reflects the characteristics of the innovation. In summary, the three methods used to collect data on teacher output each have inherent weaknesses, as well as strengths. The observation technique requires trained personnel to adequately view what is actually occurring in the classroom. Furthermore, the observation, when used alone, is not likely to provide any "additional" information about implementation which might facilitate an understanding of what is actually happening in the classroom. 31 The self report method, as shown in Crowther's study, proved to be a valid means of measuring teacher output which did not consume a large amount of the teacher's time. How- ever, a self report method, combined with the observation method, might not provide any "additional" information. If the validity of the instrument is not checked, the information collected could lead to erroneous decisions regarding the extent to which an innovation is implemented as intended. The interview method can provide "additional" informa— tion. However, if this was the only method used to measure teacher output, this also could lead to erroneous decisions. Implication of Major Findings Upon reviewing the major findings in Table 1, it becomes evident that implementation is a variable in the change process, as every study revealed that the degree of implementation varied from one teacher (or school) to another. Therefore, implementation needs to be examined carefully in any study of innovative programs where the effect of the program on student outcomes is to be determined. While studying implementation of specific innovative programs, Crowther (1972) and Gross et a1. (1971) found that only parts of the innovation being studied were 32 implemented as intended. Implementation can be seen not only as "how much" and "which parts" but also as "what combinations." A comprehensive plan for evaluating an innovative program should focus on what parts specifically, and in what combinations of parts, a program is being implemented. The findings of the five studies also support the need to view implementation as a "snapshot" of what is happening at that one particular point in time. Imple- mentation will change as changes are made in both the innovation and the user in order to meet environmental constraints (Berman & McLaughlin, 1975). Implementation will also change as the teachers experience various stages of implementation (Hall & Louckes, 1977). Measuring Factors Relating to Teacher Output Presented in this section is an overview of the major findings and theories found in the literature regarding project directives and incorporation factors that affect teacher output in the implementation of innovative programs. Four studies identifying factors affecting implementa- tion have been included in this literature review: Berman and Pauly (1975; Crowther (1972); Gross et a1. (1971); and Nauman—Etienne (1974). Procedures used to measure the 'factors are similar to procedures used to measure 33 implementation. The self report technique was used in all four studies. In addition, Gross et a1. (1971) used class- room observation and interview techniques. Unfortunately, the procedures used to develop instruments, a description of the instruments used, and the data collection schedule were not adequately described in any of the four studies. Also included in the literature are several studies that identify factors affecting the rate of adoption of innovative programs. Although teacher output was not included in these studies, it seems plausible that some of the same factors found to relate to the rate of adoption also could affect the extent to which a program is imple- mented as intended. Therefore, this section of the lit- erature review makes reference to studies which have identified factors affecting teacher output specifically as well as to studies identifying factors affecting the rate of adoption of innovative programs. Following is a discussion of project directive and incorporation factors that seem to either promote or inhibit teacher output. Project Directive Factors Project directives refer to the strategies used to plan for the implementation of an innovation. Based on the review of literature relating to project directives or strategies, two different categories of strategies were identified: program selection strategies and implementation 34 planning strategies. The factors found that might affect teacher output for each of these two categories of project directive strategies follow: Category 1: Program Selection Strategies 0 Criteria for selecting the program; and . Teacher level of participation. Category 2: Implementation Planning Strategies . Number and frequency of planning meetings; . Teacher participation in implementation planning; . Inservice training time; Value of inservice training; and Resource support. Table 2 presents a summary of the major findings for each factor. A discussion of the major findings and theories follows. Criteria for selecting the program. Zaltman (1977), in an extensive discussion of change in the educational system, surmised that the majority of innovative programs are selected based on an identified need for the program to be implemented. Change for the sake of change is most likely to result in little, if any, effective change. On the other hand, a program selected based on an identified need for change is most likely to receive support from those who are to implement it, in the hopes of solving an identified problem. Naumann-Etienne (1974), in a study inked. aaaoa one custom 33: 85386 Amped. announce ecu uuouuonu .Hnaa. .Ha uo cacao .mhoac sauna on: custom .mhmac sauna can cusuon .hhmnv .uo uo alluaou nova“. osoncuan coo occon .mhaac sauna and cannon nomad. lsonnuun and onnon .msmac sauna can nuance .cOMuoucoEonesa cu uuowuuon ouo oaoauouoa ouoovonona can one» no noon .>u«ao=v on» no ucouuoasu no uoc o4 ova>uoona no auuunoav on» use» conquosuoaag an an .oonono nonuoou o» nouaaou an mansion» uuaum .oooUOua unaccoun vco ucoeno~o>oo on» :« Iouo>o acouao on» o>~o>cq 0» ouandou onu nu omcono Ou oucououoou no luanoum on» measduoocs coauo .couoouoou ow c0«uouco§u~&l« uauooooooo mo oocono on» .ouol occaouuov on» :4 ouonaowuuom uoc one saga ”can uuocoaou an .nowuoucoEOHQEA Asuonouosu no oucondluouoc unouuoeaa Ono omnuuooa unonvouu coo nousuoc uuoamso couscoom ocqcuouu oou>uoocq uo ooauo> osau ocucaouu oou>uoocm co«»oa«0quuoa uonuooa ounduool unaccoam no hunoovouu and non-:2 nowmououuo unannoua noduoucolougsu 35 Ahead. .Ho uo coauuon .ooma. lsonnuam and onnon insane sans; can cannon .oooac osoncuao and occon Achauv Cancuuulccolfloz .oooooum anancoue cco anoino~o>oc on» ca Ion-an acouau on» o>uo>cu 0» ounudou on» ma oocono cu oonououoou uo lounoun on» ana>uuoccs nouuo .voooouuoo ow noduoucoloansd unannouuso no oononu on» .ouol o:o«o«006 on» :4 ouon«u«uuom one cup Son» Hoou ouonooou an .:o«uo>o::« on» no uaooou o oo oounono uouoouo 0» con coon coauaucoon audaooa - no ocean oval ocw>uoo Io~n0um o ouunz .ucosuussou Hove“ oauuua nu vounooou ounce uo auuaunouuo>o on» 0» oonoaoou o no vooon noumoun o voodoo noun: ouoonoum noduoauu«uuoa uo Ho>o~ uonoook IouoOum onu ocauuouoo uou cauouauu ooauououuo coauooaoo louooum ooonououoc nuauoogpxuocnvcac nona: OhOuUOk gdgoudfl huooouoo acuoou unease nonuooa ou ocauonoc ououuom o>auoouua uoonoum new oouuoonh\omc«ccuh honor no 30u>uo>o an .a ounce 36 that addressed the question of how effective statewide legislated innovations can be for local user systems, identified variables that must be present so that measurable change in education will occur. In comparing strategies of change in four schools who were "successfully" implementing Open Education and in four schools who had tried but failed to successfully implement Open Education, Naumann-Etienne (1974) found that teachers who were successfully implementing Open Education were the ones who made use of a problem solving method. It was found that this method forced attention on an existing problem. Having identified a problem, a plan of action was established. Teacher level of participation. An extensive study that looked at factors affecting implementation was con- ducted by the Rand Corporation (Berman & Pauly, 1975). Unfortunately, the measures used to determine the extent of implementation were very global, and therefore caution must be used in interpreting the results obtained. None- theless, this study did serve as a worthwhile exploratory investigation of implementation and factors affecting implementation. One of several factors investigated in the Rand Study was the level of teacher participation in the decision making process. Projects initially getting underway require that many decisions be made on a day-to-day 'NO “V. 01 it 37 basis regarding intended Operational procedures. The results of this investigation showed that the extent to which teachers participated in implementation decisions was significantly (alpha level==.lO) related to the per- centage of goals achieved and tO the extent to which the teacher perceived implementation occurring. This signif- icance level was found across all four federally funded programs included in the study (Title III, Vocational Edu- cation, Title VII, and Right-to-Read). Interestingly, the significance level was found to be .01 for the Title III programs. It was projected that this significance level occurred because Title III projects were initiated at a local level and they were implemented to solve an identified local problem. In an article written by Kenneth D. Benne and Max Birnbaum (1960), the issue of teacher participation was discussed. Although not subjected to statistical analysis, it was Observed that, the effectiveness of a planned change is often directly related to the degree to which members at all levels of an institutional hierarchy take part in the fact-finding and the diagnosis of needed change and in the formulating and reality testing Of goals in programs of change. (Benne & Birnbaum, 1960, p. 292) Number and frequency ofgplanning meetings. Another factor investigated in the Rand Study relates to the number and frequency Of planning meetings. These results showed 38 that "regular and frequent meetings can facilitate communication and coordination and also enhance morale and teacher commitment to the project. This is especially so when teachers believe that the meetings contribute to the essential activities of the project" (Berman & Pauly, 1975, p. 61). Teacher participation in implementation planning. The level of teacher involvement in planning for imple- mentation of a program affects the extent to which the program is implemented as intended (Benne & Birnbaum, 1960); Berman & Pauly, 1975; Zaltman et al., 1977). Both Benne and Birnbaum (1960) and Zaltman et a1. (1977) agree that if those individuals who are affected by a change have not participated enough in the planning, the elements of the change are not likely to be internalized enough to maintain the change in the event those who have done the planning leave. Furthermore, the lack of participation in planning is likely to cause a resistance to change which, in turn, is likely to affect implementation of that change. Inservice training time. Although the Rand Study found that the amount of staff training in itself did not signif- icantly increase the teachers' perceived success in imple- nenting an innovation, an interaction was found between the frequency Of meetings held (see factor number 3) and staff training received. The conclusion drawn based on that 39 finding was that "pre-service training, including technical assistance by consultants and outsiders, may be less effec- tive than in-service and ongoing training linked to regular and frequent meetings Of the project staff" (Berman & Pauly, 1975, p. 61). Crowther (1972) found, in his study of factors affecting implementation Of a Social Studies curriculum, that those teachers who received the maximum training (not accounting for the number Of meetings attended and frequency Of the meetings) scored higher on degree of implementation than those who did not receive maximum training. The amount of inservice a teacher receives is likely to be related to the type of innovative program to be implemented. An educational innovation requiring drastic changes in teacher role behavior and teacher- student interaction patterns is likely to require a dif- ferent amount and kind Of inservice than an innovation that does not require role and interaction changes. Value of inservice training. A conjecture made by Berman and Pauly (1975), based on the results obtained from the Rand Study, was that the quantity of inservice a teacher received might not be nearly as important as the quality Of the inservice. The inservice must relate to methodologies Of implementation. WA \4_M4 14 40 Resources support. Berman and Pauly (1975) report that the implementation problem cited by more than half of the 684 participating teachers related to inadequate space, materials, and methods. In addition, lack of time and inadequate materials were identified as barriers to implementation by Gross et al. (1971), Charters and Pelle- grin (1973), and Crowther (1972). If an innovation is to be successful, there must be an adequate supply Of resources to facilitate its implementation. Incorporation Factors Incorporation refers to the process teachers go through to incorporate or adopt the innovation into their existing organizational environments. Incorporation factors that could affect teacher output as identified in the literature have been placed in one of two categories. The categories and corresponding factors are as follows: Category 1: Characteristics of the Innovation . Communicability Of the innovation; . Complexity of the innovation; and . Perceived value of the innovation. Category 2: Organizational Environment Characteristics 0 Support of the principal; . Morale Of the teachers; and c When the last innovative program was developed or implemented. 41 Table 3 presents a summary of the major findings or theories relating to each Of these factors. Following is a discussion Of each incorporation factor: Communicability of the innovation. Gross et al. (1971) found the majority of teachers included in their study were not able to identify the essential features Of the innova- tion they were using. Gross et a1. (1971) concluded that the lack of a clear definition of the innovation was a definite barrier to teacher implementation. Similarly, a review of four case studies of differential staffing indicated that innovations which were discussed with the teachers in ambiguous global terms resulted in unclear perceptions of behavioral expectations, confusion, and frustration, which in turn caused a low degree Of imple- mentation (Charters & Pellegrin, 1973). Complexity of the innovation. Based on past writings and research, Rogers and Shoemaker (1971) contend that the degree to which the innovation is perceived by its imple— mentors to be difficult to understand and use affects the degree to which the innovation is adopted by potential users. Crowther (1972) and Gross et a1. (1971) investigated "complexity" as a factor in their studies on implementation and found support for Rogers and Shoemaker's contention. Both Gross et a1. (1971) and Crowther (1972) found com- plexity to be one of the most important factors overall 42 Annoac .a« no coauauu imam”. sauna can nuance .uemac unausouo inked. xenon ecu cocoon .couocoan on can oofiuo>ooow oo no ocuuaoco on» .ooocq zoo ow couoououoq uoo ouo ouonuoouum «a .ooduouooEOaQEd «summon loco coo oooono uonuoou no nouoono on» coooouooa muonooou on» no odouoa no": .oo«uovoosquo« Houoooooao coo ouoono nonooou uo cocoono on» coooouoow domuooaum onu uo anemone o>wuo< .coucoooanoa uo coac~o>oc no) Binocum o>duo> nooou noon onu non: ononooou on» no cacao: Hoodoodun onu no unconcm acoaooua>oo floccuuoouoouuo gahmdv Moxoioonm coo whomox .thu. Monuhbuu Anson. uonoloonm coo ouomoz .nhaa. .ao no ooouO .uhaac nonusouo Imam”. :«uuaaom can uuouuunu .ooauooooaoagsd no ooooooo on» uuouuo ooo oowuo>oooa oo uo oo~o> co>uoouon onb .on 0» sauna" no ocuuouooio«OI« ova Houoooooao anon on» on: coauoucuaoaoao no suaaonaouu> on» uouaouo on» .omcunu onu ausoauuao one: one .oo«uouooooumsu no oouvoc on» unto” on» .ucoauuooaoxo on» oozed one .oouoouuu no sodas» unusuaooo con» .coouu>occa or» no nousuoou «owuooooo >uuuoocw 00 ounoou ouo ouonooou an noduo>oooq on» no o5~o> co>uoouom noduo>ooow on» no haw-.380 ocuuo>ooou on» no human-Odin oo«ua>0oo« on» no ouauoauouoouonu ooooououou ooquoonh\omoucowh homo: uncuooh oowuouonuooon >oououum uouoom unease nonuooa on moauoaoc ououuoh noduouomuooon new oouuoonk\ooo«co«h keno: uo zoa>uoso o< .n OHAIH 43 in distinguishing the extent to which teachers implemented the program as intended. Perceived value of the innovation. According to Rogers and Shoemaker (1971), the extent to which an innovation was perceived to be consistent with the existing values and needs Of the potential users is positively related to the rate of adoption. Crowther (1972) investigated the per- ceived value of the Social Studies curriculum on the part of the teachers and found the extent to which the teachers perceived the curricula to be Of value to them was related to the extent to which they implemented the curricula as intended. Support Of thepprincipal. The Rand Study (Berman & Pauly, 1975) revealed that insofar as the principals sup- ported an innovation, implementation was facilitated. The relationships between the principal's support and the degree of implementation was found to be statistically significant (alpha 1evel==.10). 'It was found that elementary principals were "gatekeepers" of change either facilitating or inhib- iting implementation. Crowther (1971) likewise found that the perceived influence Of the principals accounted for a significant difference in the degree Of implementation. Morale of the teachers. The Rand Study found that the higher the school's morale, the greater the project's percentage of goals achieved and the greater the degree of implementation. 44 When the last innovative program was developed or implemented. Based on a review Of writings and studies relating to change in school systems, Zaltman et a1. (1977) concluded that if principals were not interested in new ideas, the adoption of an innovation could be hindered. A principal who is receptive to change is consequently more supportive of teachers attempting to implement new programs than a principal who is not receptive to change. Summary As can be seen from the literature, there exists a need to consider the extent to which implementation occurs when evaluating the impact an innovative program has made on students. It became apparent that teacher implementation does vary from one teacher to another, and, although not directly determined in any Of the studies reviewed, it is conceivable that this variation in implementation could affect student outcomes. The literature suggested that a definition of imple- mentation should not be considered to be stationary. As teachers gain more experience in implementing an innovation, and as environmental conditions change, implementation is likely to be altered. For this reason, it can be surmised that an evaluation designed to measure implementation must incorporate means for obtaining a description of program 45 aspects that were and were not implemented, and for obtaining means for determining "why" certain aspects were not implemented as intended. This information could assist teachers, principals, and project directors in gaining a better understanding Of the implementation process at that particular point in time. Although research of factors affecting implementation is meager, there is every indication that such factors are identifiable. Incorporating methods to measure and analyze factors affecting implementation can provide useful informa- tion to decision makers. Since the factors included in this review, for the most part, can be controlled by a project director, principal, and/or teacher, knowledge Of factors adversely and positively affecting implementation can direct efforts to maintain or alter existing factors prior to any future attempts to implement the program. Hopefully, further research will generate more specific information regarding implementation of innovative programs and factors affecting implementation. Such information can provide a better understanding Of teacher behavior during implementation Of innovative programs. CHAPTER III METHOD OF STUDY Introduction Presented in Chapter I was a diagram describing implementation of an innovative program as a process involving four interrelated components: Teacher Output: The extent to which a teacher implements the program. Student Outcomes: Changes in the students' knowledge and/or affective behavior. Project Directive Factors: Variables relating to planning strategies employed to initiate implementation of a program. Incorporation Factors: Variables relating to the process of adopting a program into existing environmental conditions. The purpose of this study was to develop a plan for evaluating innovative programs which incorporates techniques for Obtaining a measure Of teacher output and for deter- mining the relationships between: teacher output and student outcomes; teacher output and project directive factors; and teacher output and incorporation factors. 46 47 To determine the appropriateness of the design, it was applied to the evaluation Of a career/leisure education program. The purpose of this application was to determine the extent to which the evaluation plan could provide adequate means to: 0 Measure teacher output; . Relate teacher output to student outcome; . Relate project directive factors to teacher output; and - Relate incorporation factors to teacher output. This chapter is divided into two major sections. The first section presents an overview Of the career/leisure project including a description of the career/leisure pro- gram and the implementation setting. The second section provides a description of the design used to evaluate the career/leisure program including procedures used to collect and analyze data. An Overview of the Career/Leisure Project Activities The career/leisure program was implemented through a career education project funded by the Michigan State University Department Of Education. In the fall of 1977, the Shiawassee Intermediate School District, composed of eight rural local school districts, was awarded career education project funds. To operate the project, a 48 committee composed of a project director and middle school teachers from the local school districts was established. The selection of each teacher was based on the school principal's recommendation and the teacher's interest in participating. The principals were contacted by the project director, and, if they agreed to participate, they each selected a teacher from their respective buildings to serve on the project committee. Of the eight local school districts in Shiawasee County, six agreed initially to become involved in the project. The committee was to complete the following tasks: . Identify career education program goals; - Assess the needs of the students relative to the program goals; - Select a career education product for implementation; and 0 Evaluate the career education program and the project outcomes. The intended duration Of this state funded project was one year, with implementation beginning six months after the project began. However, program selection was not completed until a year and a half later. During this time lapse, three project teachers chose not to continue and had to be replaced. One Of the three replacements elected to terminate and was replaced. Therefore, the 49 the career program goals, student needs assessment, and product selection were completed by project staff on hand at the time. Because of the high turnover in staff, not all implementing teachers were involved in the completion of all project tasks. Middle school students were selected to receive the career program instruction under the philosophy that middle school years represent an important time for students to explore their abilities, needs, and interests. The project members believed that part Of the students' learning should include becoming aware Of alternative family, citizen, leisure, and occupational roles. The goals for the career program were developed based on this philosophy. They are as follows: 1. The student will identify his/her unique characteristics. 2. The student will use decision making skills. 3. The student will identify the four life roles. 4. The teacher will provide exposure to alternatives within the four life roles. 5. The student will explore alternatives. 6. The student will describe interrelationships among the four life roles. The project teachers administered a needs assessment to determine how well their middle school students were able 50 to perform in the six goal areas. The assessment instrument used was developed in January 1977 by Dr. Julie Miller at the University Of Michigan. The results Obtained from the needs assessment indicated the students lacked awareness in the area of leisure, as it relates to career education. Based on these results, it was decided the focus of the career education program should be on leisure concepts. The student outcome to be attained was: Given appropriate materials, the students will explore as many leisure activ- ities as possible during the school year, and will state how the selected activities are related to his/her family, occupation, and citizen roles. The selection Of the curriculum materials was made by project teachers. The State Department provided assistance in locating career education prOgrams that related to leisure and helped establish a criteria checklist for material selection. (A more detailed description of the criteria checklist is presented in Chapter IV.) Description Of the Career/Leisure Program The career/leisure program selected for implementation was the Kangaroo Kit, a curriculum product of the Leisure Education Advancement Project (LEAP) published in 1977 by the National Recreation and Park Association in Arlington, Virginia. The program focuses on the use of leisure and 51 its relationships to an individual's life roles as a member of a family, society, and the work force. Procedures used to develop the LEAP materials included: (1) a nationwide study and an extensive review of the liter- ature to determine the need for a leisure education program; (2) a review of the LEAP curriculum by experts in both edu- cation and leisure education; and (3) a pilot test conducted with 34 teachers representing grades K-12 and a variety of subject areas in two public school systems in Indiana. The materials were revised following the pilot test. However, they were not field tested. The LEAP materials were intended to serve as a resource guide for teachers and curriculum development specialists. The materials contain information relating to the following student goals: The student will 1. use process of inquiry in order to analyze personal needs and reactions to leisure choices and experiences. 2. use the process of investigation to become aware of resources and facts about leisure. 3. use processes Of planning, decision making, and problem solving in determining leisure experiences that effectively meet personal needs. 4. engage in many and varied fields of leisure activities. 52 5. exhibit a willingness to attempt and sample new experiences in many and varied fields of leisure activities. 6. exhibit personal skill development in many fields Of leisure activities. Career education is considered tO be an integral part of the LEAP program due to the recommended approaches to learning the leisure concepts. These approaches provide the students with an Opportunity to analyze their personal needs, to investigate existing resources, and to become involved in the processes of planning, decision making, and problem solving. The LEAP curriculum has been organized into eight goal (or target) areas. For each target area, more specific Objectives (or focus points) have been identified. Although the focus points by themselves are not considered to be sequential, they have been grouped by ability levels with each subsequent ability level becoming increasingly more difficult and requiring knowledge and skills from the preceding ability level(s). The curriculum guide recommends that the ability levels be taught in sequence. The LEAP curriculum was designed to be infused into existing school subjects. Three classroom implementation strategies are suggested in the curriculum guide materials. 53 One strategy is to teach leisure as an incidental learning Opportunity. This strategy involves either using leisure to illustrate the content being studied or temporarily diverting from the subject area and covering leisure content in depth. A second strategy for implementation is to directly infuse the leisure content into the existing subject matter. This strategy involves either the teacher examining the existing subject matter content and attempting to inject leisure concepts; or, the students and the teacher together brainstorming for leisure activities that relate to the subject matter. A third implementation strategy involves altering the learning environment to create a positive teaching/learning environment in which a leisure attitude is an important aspect. Although not written as a specific strategy for classroom implementation, the curriculum materials suggest that teachers attempt to involve the local community. This suggestion was based on the philosophy that the school can provide a formal knowledge base for the curriculum, but it cannot be the only resource. Several suggestions for involving the community are presented throughout the curriculum materials. 54 From the program goals, the project teachers selected those which best matched the project's student goals. The teachers selected three target areas and several corresponding focus points for implementation. (The selected "target areas" and "focus points" are listed in Appendix A.) For each target area and corresponding focus point, the teachers identified more specific con- cepts they felt needed to be implemented. This was done to help clarify the focus points as they were written and to increase the teachers' consistency in interpreting the selected focus points. The strategy selected by the teach- ers for the implementation of the target areas and focus points was direct infusion of the LEAP materials into the ongoing curriculum. Prior to implementation, the teachers were asked to attend a half-day inservice session to increase their awareness of the LEAP curriculum. The inservice session, conducted by two professors of the Health, Physical Edu- cation, and Recreation Department of Michigan State University, included a discussion of leisure, the LEAP target areas, the format of the LEAP Curriculum Guide, and the implementation strategies. The teachers were provided with specific examples of how the LEAP concepts could be integrated into various curricula. 55 Description of the Implementation Setting The career/leisure program was integrated into four subject areas: Math, Reading, English, and Social Studies. Implementation of the career/leisure program began March 19, 1979, and ended May 11, 1979. The eight-week implementation schedule was considered by the teachers to be sufficient length Of time for them to implement, and for the students to learn, the selected target and focus points. Although seven teachers were initially included in this study, only five teachers completed the inplementation of the LEAP curriculum. One of the seven teachers took a temporary leave Of absence. The other teacher claimed the LEAP curriculum took too much time away from his basic subject area. The loss of two teachers is most unfortunate. However, the study is to serve as an exploration Of a method for evaluating teacher implementation and factors affecting implementation. Furthermore, the number of teachers remaining in this study does reflect the small number of teachers often involved in the initial imple- mentation of an innovative program. A total Of 211 students were provided instruction in the LEAP curriculum. Of the 211 students, 193 were in the seventh grade and 18 were in the fifth grade. The fifth grade students were unique to the implementation for three 56 reasons. First, they were fifth graders and not seventh graders. Second, they were enrolled in small classes (3-8 students per class) in a Title I Reading and Math Enrichment Center. Third, the classes met for only 20 minutes per day, whereas the seventh grade classes were in session for approximately 40-50 minutes. A Description of the Evaluation Design The design used to evaluate the career/leisure program includes three phases. The first phase focuses onteacher output and incorporates procedures used to Obtain a descrip- tion of, and a measure of, teacher output. The second phase adverts to procedures used to evaluate student outcomes and to determine the existing relationship between student Out- comes and teacher output. The third evaluation phase relates to techniques used to identify the relationship between teacher output and project directive factors and between teacher output and incorporation factors. Figure 1 illustrates general procedures to be followed to evaluate an innovative program. A description Of the evaluation techniques used for each phase, as they were applied to the evaluation of the career/leisure program, follows: Phase I Evaluation of Teacher Output I Define Teacher Output I Select Measurement Techniques I Develop Instruments] I Collect the Data J l , l Analyze the Data—] 57 Phase II — Evaluation of Student Outcomes Identify Expected Student Outcomes IdentifyI the Design for Determining Student Impact Select Measurement Techniques [Develop Instruments] 1 I Collect the Data I LW. .. NJ I Phase III — Identification of Factors Relating Teacher Output Select Factors for the Study Select Measurement Techniques I I Develop Instruments I [Collect the Data J I Analyze the Data I Determine the Relationship Between Teacher Output and Factors Determine the Realtionship Between Teacher Output and Student Outcome Measures I I | J Figure 1. General Procedures for Evaluating an Innovative Program. 58 Phase I: Evaluation of Teacher Output The purposes of this phase of the evaluation design are: . To measure the extent to which each teacher implements the program; . To provide a description of program components implemented; and 0 To identify problems encountered during implementation. Following is a description of: (1) techniques used to define teacher output; (2) data collection procedures and instrumentation; and, (3) data analyses procedures. Procedures used to define teacher output. This study takes two approaches to define teacher output. One approach is based on the need to obtain a measure of teacher output. In this approach, teacher output is defined as a product where anticipated teacher behaviors have been identified. The second approach views teacher output as a process whereby both the innovation and teacher implementation are continually changing. Viewing teacher output as a product provides a measure of the extent to which the teachers implemented the program. Viewing teacher output as a process provides a "snapshot" of implementation during the time the evaluation was taking place. 59 Teacher output for the career/leisure program was defined by reviewing the LEAP Curriculum Guide, by review- ing the content presented at the teacher inservice session, and by interviewing the teachers planning to implement the program. The interview was conducted in a group setting. The purpose of this interview session was to obtain from the teachers a description of how the program was to be implemented. The definition of teacher output included an identification of major components and subcomponents of the curriculum to be implemented, a description of anticipated teacher behavior, and information needed to determine the existence of each teacher behavior. Description of data collection procedures and instrumentation. Three methods were used to measure teacher output of the career/leisure program: interview, self report, and observation. The interview and self report techniques were used to measure all intended teacher behaviors. Observation was used to confirm the program was being integrated into the existing curriculum. A Teacher Implementation Interview instrument was developed to measure teacher output and to ascertain from the teachers which aspects of the implementation were working, which were not, and why. This interview instrument contained 12 quesitons, six of which required a yes or no response, as well as an explanation of their 60 response. The item format used for the remaining six items was open-ended. A self report form contained a list of the target areas and focus points to be implemented. The teacher was to document for each focus point, the data of implementation and the learner activities developed for implementation. The self report and interview instruments were used to cross validate responses obtained. (Appendix B contains a copy of each instrument used to collect teacher output data.) The self report forms were completed by the teachers throughout the implementation period. The interview and observations were scheduled on the same day for each teacher. Each teacher was contacted by phone to ensure the scheduling of convenient interview times. The time selected in each instance was the teacher's planning period. The classroom observation was scheduled either before or immediately after the interview depending on the teacher's schedule for implementation. Each Teacher Implementation Interview, which took approximately 30 minutes to complete, was taped. The responses were transcribed onto data sheets after the interview. The classroom observation lasted from 30-50 minutes depending on the teacher's plans for implementation on that day. Data analyses procedures. 61 A scoring technique was devised to quantify the degree to which the teachers were implementing as intended. This scoring process incorporated data collected from both the interview and self report instruments which were interpreted by the investigator. The variables for which each teacher received a score and the scoring technique used are as follows: Variable The extent to which the identified leisure con- cepts were presented. The extent to which the selected content was inte- grated into the teachers' curriculum area. The extent to which learner activities were developed for each target area. The extent to which the local community was involved in the LEAP implementation. The extent to which the stu- dents were involved in the planning and/or decision making processes. The extent to which oppor- tunities were provided the students, to become aware of their leisure activities. The extent to which the levels of the focus points were taught in sequence. Scoring Technique The number of concepts presented. Proportion of content area presented that was integrated. The average number of learner activities devel- oped for each target area. The number of learner activities implemented that involved the local community. The number of learner activities implemented. The number of learner activities implemented that provided an oppor- tunity to identify their leisure interests. The proportion of focus points taught in sequence. 62 Variable Scoring Technique 8. The extent to which oppor- The number of learner tunities were provided the activities implemented. students to become aware of resources and facts about leisure. 9. The amount of time devoted The number of hours devoted to implementation. to implementation. To calculate an overall teacher output score, the scores obtained for each variable were transformed into a standard 2 score. An average of the standard 2 scores represented the overall teacher output score. Data analyses for teacher output also included a review of the interview responses to obtain a description of implementation and to identify problems encountered by teachers as they attempted the implementation. The intent of this review process was to create a better understanding of the implementation of the career/leisure program. Phase II: Evaluation of Student Outcomes This phase of the evaluation was designed to accomplish the following objectives: 1. To determine the extent to which the student's behavior was altered as a result of the program. 2. To ascertain the relationship between the teacher output and student outcome measures. Following is a description of: (l) the design used to determine student outcome; (2) the instrument used to 63 measure student outcome; and, (3) procedures used to analyze the student outcome data. Design used to determine student outcome. The program was reviewed and teachers were interviewed to identify intended student goals and objectives. (Appendix A con- tains the anticipated student goals and objectives to be implemented and measured.) Based on a review of the student goals and objectives, it was decided that the student out- come to be measured was cognitive ability. Of primary interest was the extent to which the students' learned as a result of receiving the LEAP instruction. A pre-post, control treatment group design was used to determine the extent to which the student outcomes had been attained. One seventh grade class from each of four schools served as the control group for this study. The distribution of students in the treatment and control group for each school is presented in Table 4. Description of instrumentation. A cognitive skills test containing 60 multiple choice, true/false, and matching items was developed. The test items represented all program objectives, as approximately five items per objective were included in the test. The procedures used to develop the cognitive skills test included: (1) the teachers drafting from five to ten test items for each student objective; (2) the evaluator and 64 Table 4. Distribution of Treatment and Control Students Experimental Control Number Number Number Number School Subject Area Classes Students Classes Students A Math 1 24 1 25 B Math 3 71 1 13 C Social Studies 2 51 l 20 D English 2 47 1 19 E Reading and Math Enrichment __;§ .ggg __;1 .11. Total 11 212 7 94 the teachers reviewing the pool of items and selecting those which seemed to best measure each objective; and, (3) the evaluator reviewing and editing those items selected. A pilot test of the items was conducted using one class of seventh grade students. The purpose of this pilot test was to identify items and directions not clear to the stu- dents. The test items were administered by a project teacher who documented comments made by the students regarding specific items and directions. Alterations were made on the test based on this pilot process. (Appendix C contains a copy of the cognitive skills test.) The cognitive skills test was administered to the treatment and control classes on a pre—post basis. The 65 pre test was administered to both groups during the first week of implementation. The post test was administered during the last week of implementation. The project teachers were responsible for administering the tests to their respective treatment and control classes. Data analyses procedures. All tests were machine scored and computer analyzed. The total number of correct answers for each student was recorded. A Kuder Richardson Formula 20 was used to calculate the reliability of the test. This calculation was performed using the post test scores obtained from the treatment students. The results revealed a reliability coefficient of .79, which is con- sidered to be relatively high. Independent t-tests were used to determine the significance of the difference, between the treatment and control groups on the pre test scores and the post test scores. To analyze the relation- ship between teacher output and student outcome, a Pearson product moment correlation coefficient was calculated on the mean gain scores obtained for each teacher's treatment group and the teacher's overall teacher output score. Phase III: Identification of Factors Relating to Teacher Output The objectives to be accomplished by Phase III of the evaluation design are: 66 1. To provide a description of project directive and incorporation activities surrounding implementation. 2. To identify project directive factors relating to teacher output. 3. To identify incorporation factors relating to teacher output. Two sources were used to identify factors to be included in this study. One source was a review of liter- ature relating to project directives and incorporation The factors selected factors affecting implementation. based on this review are as follows: Project Directive Factors Criteria for selecting the program. Teacher level of participa- tion in the selection process. Number and frequency of planning meetings. Teacher participation in the planning process. Inservice training time. Value of inservice training. Resource support. Incorporation Factors Communicability of the innovation. Complexity of the innovation. Perceived value of the innovation. Support from the principal. Morale of the teachers. Innovativeness of the school. The second source used to identify factors was the conditions surrounding implementation. From this source, 67 the investigator identified one project directive factor (community involvement) and three incorporation factors (support of non-implementing teachers, teachers' perception of career education, and time spent on preparing lessons) to be included in this study. Although no research could be found to support these additional factors, they were considered to be factors likely to affect the degree of implementation. Not every project directive factor included in this investigation will be able to be included in the corre- lational analysis for identifying factors relating teacher output due to the fact that all teachers recevied the same treatment resulting in no variation. The project directive factors that will not be included in the correlational analysis are: criteria for selecting the program; number and frequency of planning meetings; inservice training time; and, resource support. These factors were included in this study in order that a description of these project charac- teristics could be established. Because previous research has found these factors were to be at least potentially related to implementation or adoption of an innovative program, the description of these factors provided by this study can serve as data base for future research endeavors. 68 Following is a description of: (l) procedures used to collect data for the factors, and (2) procedures used to analyze the data. Data collection procedures. An in—depth interview technique was used to determine the extent to which the factors were present. Interviews were conducted with the project director, implementing teachers, and the school principals. Three instruments were developed (one for each of the sources of information). The items contained in each instrument were developed through an identification of information needed to determine if a factor was present. This information was identified by reviewing past studies and by observing the implementation setting. The Teacher Interview form included a total of 39 items. Twenty-eight items required scaled responses, one required a yes/no response, and the remaining items required open-ended responses. The Project Director Interview instru- ment contained 16 items. Four of the 16 items required scaled responses, one item required a yes/no response, and the remaining items were open-ended. The Principal Inter- view form included three scaled items, three yes/no response items, and four open-ended items. For all closed-ended items, additional comments were invited from all persons interviewed. (Appendix D contains a copy of the instruments used to measure the factors.) 69 The Teacher Interview was conducted at the same time as the interview for verifying the teacher output measure. This interview, which also was taped, took approximately 30 minutes. The project director's interview was not taped and lasted about 40 minutes. This interview was scheduled at a time most convenient to the project director. The principal interviews, which lasted about 20 minutes each, were scheduled on the same day the teacher in that building was interviewed. Data analysis procedures. To obtain a description of the project directive and incorporation activities that surrounded implementation of the career/leisure program, the data collected was reviewed and summarized. To determine if each factor included in this study related to teacher output, the results obtained on related items of the interview forms were reviewed. (Appendix D contains a matrix, including the interview items used to analyze each factor.) In many instances, there was more than one interview item and more than one source of infor- mation used to analyze a factor. The outcome of this analysis process was a ranked score for each teacher on the extent to which the factor was present as compared to the other teachers. The ranking in some cases was a judg- ment call on the part of the investigator. In other cases, the ranked score resulted from a numerical calculation of responses obtained. 70 The ranked scores obtained for each factor were compared to the total teacher output score. To ascertain the extent to which each factor related to teacher output, the teacher output scores were also ranked. A Pearson product-moment correlation coefficient was calculated to determine the relationship of the ranked scores. CHAPTER IV ANALYSES AND RESULTS The purpose of this study was to develop an evaluation design that focuses on teacher implementation of an inno- vative program and to determine the appropriateness of the design by applying it to the evaluation of a career/leisure program. The evaluation design is comprised of three phases: Phase I, Evaluation of Teacher Output; Phase II, Evaluation of Student Outcomes; and Phase III, Identification of Factors Relating to Teacher Output. The primary focus of the evaluation plan is on the relationship between: teacher output and student outcome; teacher output and project directive factors; and teacher output and incor- poration factors. The accumulated data from each of these three evaluation phases as they were applied to the evalua- tion of a career/leisure program are analyzed, discussed, and interpreted in this chapter. Also discussed in the chapter are the findings regarding the appropriateness of the design developed to evaluate innovative programs. 71 72 Results Obtained for Phase I: Evaluation of Teacher Output The purpose of Phase I of the evaluation is twofold: (l) to measure the extent to which each teacher implemented the LEAP program, and (2) to describe how the various pro- gram components were implemented. Included in this section is a description of the results obtained for each of the five teachers who implemented the LEAP program on each of the nine scales used to measure teacher output. To provide a context for interpreting the obtained teacher output measure, a general description of the various approaches employed by the teachers to implement the LEAP curriculum follows. Teacher A taught seventh grade mathematics. During the scheduled implementation of LEAP, the main instruction focused on two units. One unit related to percents, deci- mals, and fractions. The second unit centered on geometry. Teacher A attempted to directly relate the leisure concepts to the two math units taught. The LEAP learner activities developed did not deviate greatly from the type of learner activities the teacher normally implemented. A priority with Teacher A was to develop learner activities for the career/leisure program that would simultaneously enhance learning in math. Teacher B taught a seventh grade geometry unit. This teacher's primary means for integrating the leisure concepts 73 into the math instruction was to involve the students in some form of leisure games. Teacher B had approached math as a leisure activity and had been incorporating math games into the program prior to implementing LEAP (although no formal teaching of the LEAP leisure concepts had taken place). Teacher C, who taught seventh grade American history, presented the majority of leisure concepts through a project that had been ongoing approximately two years prior to the implementation of the LEAP curric- ulum. This project involved the students in the creation and maintenance of a bicentennial historical museum. Over the past two years, students had collected and researched the items included hithe museum. In addition to the museum activities, the students investigated and reported on the history of various leisure activities. Teacher D taught seventh grade English. The units taught centered on written and verbal communication skills. The LEAP curriculum was related to the communication skills by using the career/leisure concepts as topics about which the students wrote and spoke. The student brought in one of their favorite leisure activities to share with their classmates. Teacher E taught fifth grade Title I math and reading. This teacher found it difficult to adequately integrate the 74 LEAP materials into the remedial proqrams because of the ability level of the students. In order for them to comprehend the leisure concepts, Teacher E found they had to be taught directly. Attempting to relate the leisure concepts to the math and reading concepts seemed to cause the students to be confused. Following are the outcomes obtained for each of the nine measures of teacher output: The extent to which the content was presented: No teacher was able to implement all of the 28 leisure con- cepts selected for implementation because of a reported lack of instructional time. Some days of LEAP instruction were lost because of a flu epidemic that resulted in a high rate of student absenteeism. Other days of instruction were lost when school closed because of heavy snow. All teachers felt pressured to complete the instruction they had planned for their primary curriculum area, before completing the LEAP curriculum content. Table 5 identifies the teachers and proportion of teachers implementing each selected leisure concept. This table also presents an implementation score obtained for each teacher on this teacher output measure. A review of Table 5 shows that all but eight leisure concepts were implemented by at least three of the five teachers. Because of the reported shortage of time, the 75 Table 5. Teacher Output Score for the Extent to Which the Leisure Concepts Here Implementeda Teacher Percentage of Teachers Leisure Concept A B C D 2 Implementing Define leisure . . . . . . . . . . . . . . . . . . . . . . . x x x x 100 Define leisure activities . . . . . . . . . . . . . . . . . x x x 100 Discuss why an increase in leisure time . . . . . . . . . . . x x x x 80 Discuss need for a variety of leisure experiences . . . . . x . . x . 4O Define leisure forms . . . . . . . . . . . . . . . . . . . . x x x x x 100 Define and discuss levels of leisure . . . . . . . . . . . . x x x x 100 Discuss need to develop positive leisure experiences and limit negative experiences . . . . . . . . . . . . . . . . x . x x . 60 Define terms in basic leisure elements and discuss when and how each element may be effective in satisfying needs 0 O O O O O O O O O O O O O O O O O O O I O I O I O x O x x x 80 Discuss idea of "maturation of ability“ . . . . . . . . . . . . . x . 20 Discuss the personal needs an individual has which may be satisfied by leisure activities . . . . . . . . . . x x x x 100 Discuss reasons for finding available leisure resources . . x x x x 100 Define leisure resources and give examples . . . . . . . . . x x . x x 80 Discuss how to gather and compile information effectively . x . . x . 40 Present how to construct a questionnaire . . . . . . . . . . x . x x . 60 Define ways to show appreciation . . . . . . . . . . . . . . x x x x . 80 Define demonstrate and demonstration . . . . . . . . . . . . x . . x . 40 Discuss ways people can be a leisure resource . . . . . . . x x x x x 100 Present how to write an effective letter . . . . . . . . . . . . . . . 0 Tell where to find leisure resource addresses . . . . . . . . . . . x 20 Discuss the functions of the various state and national bureaus . . . . . . . . . . . . . . . . . . . . . . . x . . 20 Discuss the legend, symbols, types of maps. BPQCialty maps, and where they can be found . . . . . . . . . . . . x x x . . 60 Discuss how to use resources to plan and develop leisure projects . . . . . . . . . . . . . . . . . . . . . x x x . . 60 Discuss interrelationships of leisure, family. citizenship, and occupation . . . . . . . . . . . . . . . x . x x x 80 Define basic needs and values . . . . . . . . . . . . . . . . x x x x 80 Define satisfaction . . . . . . . . . . . . . . . . . . . . . x x x x 80 Discuss relationship change in one area to change in mother I O I O O O O O O O I O O I O I O O O I O O O 0 e I x U x ‘0 Define and discuss personal needs . . . . . . . . . . . . . x x x x x 100 Define the following terms: service, product, aesthetic satisfaction, and sensory functions . . . . . . . . . . . x . . . . 20 Total number of concepts implemented 20 15 20 21 16 Teacher output scores 3 3 $ g g; + 7 + + u aOverall, 18.4 concepts were implemented across all five teachers. 76 teachers had to select from the list of 28 concepts those they were going to implement. This selection was made based on five factors: (1) the extent to which they felt they could integrate the concepts into their curriculum areas; (2) the projected time required to prepare for implementation of a concept; (3) the projected time required for implementation; (4) the extent to which they would feel comfortable implementing a concept; and (5) the extent to which a concept provided background information for other leisure concepts. This selection was made after the onset of implementation when it became obvious it was not feasible to provide instruction for all leisure concepts. The extent to which the LEAP content areas were inte- grated into the existing curriculum. Table 6 illustrates the extent to which each teacher integrated the leisure concepts they implemented. Given that all content areas were to be integrated into the existing curriculum, the obtained scores appear to be relatively low. Based on the teacher output scores and teacher interview responses, the subject area that appears to be least conducive to integration was the remedial math and reading class. The LEAP curriculum appears to be most conducive to integration in the social studies and English curricula. Regarding future implementation of the LEAP curricula, three teachers suggested that there be coordination between several 77 Table 6. Teacher Output Score for the Extent to Which the Leisure Concepts Were Integrateda Number of Number of Content Content Proportion Teacher Areas Areas of Concepts Output Teacher Subject Area Taught Integrated Integrated Score A Math 20 9 .45 +.50 B Math 15 5 .33 -.28 c Social Studies 20 10 .50 +.78 D English 21 lO .48 +.67 E Remedial Math and Reading 16 l .06 -l.67 a . . . . . The overall proportion of concepts integrated into ex1st1ng curricula equaled .38. teachers covering several different subject areas to facilitate integration of all leisure concepts. The extent to which learner activities were developed for each target area. Table 7 illustrates the number of learner activities implemented for each Target Area. Note that the total number of learner activities developed (75) does not equal the total number of concepts taught (92). This was the case for two reasons. One reason is that some concepts not related to any Target Area were selected for implementation to provide background information. The second reason is that some learner activities related to more than one leisure concept. The results obtained show that a substantially larger number of learner activities 78 Table 7. Teacher Output Score Relating to the Number of Learner Activities Implemented per Target Area Number of Learner Target Target Target Activities Teacher Area Area Area Implemented Output Teacher 1.0 4.0 8.0 per Target Area Score A 5 6 2 4.33 —l.00 B 8 6 3 5.67 +1.00 C 7 7 3 5.67 +1.00 D 6 7 2 5.00 0 E _§. _g_ _3_ 4.33 -1.00 Total 34 28 13 were implemented for Target Areas 1.0 and 4.0 than for Target Area 8.0. The teachers reported this was the case because they felt, for the most part, concepts in each Target Area provided background information for concepts presented in subsequent Target Areas. Therefore, the learner activities presented in Target Area 8.0 required a knowledge of concepts presented in learner activities developed for Target Areas 1.0 and 4.0. The extent to which the community was involved in the LEAP implementation. The number of learner activities developed and implemented by each teacher involving community members is presented in Table 8. 79 Table 8. Teacher Output Score for the Extent to Which the Teachers Involved the Community in the LEAP Program Number of Learner Activities Teacher Output Teacher Involving the Community Score A 2 -.62 B 4 +.92 C 3 -.15 D 4 +.92 E l -l.38 On the average, the teachers implemented 2.8 learner activities which involved community members in the LEAP curriculum. Two LEAP concepts were implemented which involved community members. One concept related to the use of resources to plan and deve10p leisure projects. Students were asked to help organize a career/leisure activity day where community members were invited to share information regarding their careers. Students also approached community members to visit their classroom to discuss how their careers could be considered a hobby. The second leisure concept that involved community members related to the development of a questionnaire. Students were asked to construct a questionnaire for the purpose of determining the various ways people use their 80 leisure time. The students were asked to interview their parents and other family members. In addition, a letter was sent home with the students to inform their parents of the purpose and the general pro- cedures for implementing the LEAP program. Two teachers also sent a supplemental letter further explaining how they were planning to integrate the LEAP concepts into the curriculum. The extent to which the students were involved in the planning and/or decision making process. Table 9 presents the number of activities each teacher implemented which involved students in the planning and/or decision making process. Although no teacher spent time on instructing how to plan and make decisions per se, each teacher did involve the students in at least one activity that required .them to plan and/or make decisions. The various learning activities included: (1) planning a career/leisure activity day; (2) planning a leisure resource file; (3) planning a trip to an area within a 50 mile radius of the student's home and selecting leisure activities they would do on this trip; (4) deciding which community members to invite into their classrooms to speak on their careers and hobbies; and, (5) discussing why people decide to do "creative" as opposed to "criminal" types of leisure activities. 81 Table 9. Teacher Output Score for the Extent to Which Students Were Involved in Planning and/or Decision Making Number of Learner Activities Relating to Planning and/or Teacher Output Teacher Decision Making Score A l -.46 B 4 +1.85 C l -.46 D l -.46 E l -.46 The extent to which opportunities were provided the students to become aware of their interest in leisure activities. The results obtained for this measure are presented in Table.10. On the average, the teachers implemented 2.4 learner activities relating to self- awareness. Activities implemented to help students increase their self-awareness include: (1) keep a daily record of the amount of leisure time they had and what type of activities they did during their leisure time; (2) rating their feelings and attitudes toward various leisure activities. Whether or not the levels of each Focus Points were taught in sequence. All teachers taught the Focus Points in sequence. Because there was no variability in scores, 82 Table 10. Teacher Output Score for the Extent to Which Students Were Involved in Self-Awareness Activities Number of Learner Activities Teacher Output Teacher Relating to Self-Awareness Score A 2 -.45 B 3 +.67 C l -l.58 D 3 +.67 E 3 +.67 this variable could not be converted to a standard score. Consequently, it was not included in the overall teacher output score. The extent to which the teachers provide an opportunity for the students to investigate resources and facts about leisure within the community. With only one exception, teachers were able to provide their students with at least one opportunity to investigate resources and facts about leisure. The teacher output scores obtained are presented in Table 11. Activities in which the students participated that provided an opportunity for them to investigate resources and facts about leisure included: (1) developing a file listing various leisure activities and where to find out more about each activity (including persons in the com- munity who could provide information); (2) investigating the 83 Table 11. Teacher Output Score for the Extent to Which Students Were Provided an Opportunity to Investigate Leisure Resources‘ Number of Learner Activities Teacher Output Teacher Related to Leisure Resources Score A 2 +1.41 B l 0 C 1 0 D 0 -l.4l E l 0 history of various leisure activities; and, (3) investi— gating and reporting how to do various leisure activities. Time spent on implementation. Table 12 presents an overview of the number of hours each teacher devoted to the LEAP implementation. The average number of hours per week devoted to LEAP implementation was .83 hours. The average number of weeks spent on implementation was 6.8. The average total number of hours devoted to implementation equals 5.67 hours. Summary of Teacher Output Measures The purpose of this phase of the evaluation was to establish a measure of the extent to which each teacher implemented the LEAP curriculum. Table 13 presents a composite of the standardized teacher output scores. 84 Table 12. Teacher Output Score for the Time Spent Implementing Average Number of Teacher Average Number of Number of Hours Spent Output Teacher Hours/Weeks Weeks Implementing Score A .67 6.5 4.36 -l.l7 B .83 7.0 5.81 +.l4 C 1.00 7.0 7.00 +1.16 D 1.00 6.5 6.50 +.72 E .67 7.0 4.69 -.87 Average .83 6.8 5.67 The overall teacher output score equals an average of the standardized scores obtained for the eight teacher output variables. Results Obtained for Phase II: Evaluation of Student Outcomes The purpose of this evaluation phase is twofold. One purpose is to determine the extent to which the students increased their awareness of leisure concepts as a result of participating in the LEAP program. A second purpose is to ascertain the extent to which the student outcome measure relates to the teacher output measure. Student Outcome Results The student outcome measure consisted of a 60 item, objective, cognitive skills, test administered on a pre- post basis to a treatment and a control group of students. 85 .uouoom sameaum> usmuso segues» ucmwo ecu mo ooeuo>e co :0 venues 05.: hm.u o no.+ ov.: mn.~l oo.Hn no.~n mm.u m o~.+ ~e.+ av.uu hm.+ ov.u ~m.+ o no.+ mm.+ o HN.+ o~.~+ o mm.uu ov.| mu.+ oo.~+ m>.+ mm.+ o em.+ v~.+ o so.+ mm.a+ mm.+ oo.H+ m~.n o~.~u m m~.u ea.uu H¢.H+ mv.| ov.u «0.: oo.~n Om.+ mm.+ < eeuoom ceaueucoEeamEH wouu50nux succeed unecouesznuuom ocuxe: aaacsfisoo cont venues poueumoucu poucoamamlu scrunch usmuso cu ucomm eusoz usausmwune>CH cu addendum codewoeo mcq>ao>cu \mo«uw>«u0< numoucou numeucoo success as usaueaem newuu>auo¢ use mcwcceam noduu>wuo¢ mcacueeq no cassava mo seaweed uo adeno>o neuua>auo¢ uo semis: on mcaueuem uo homes: honesz omeus>¢ accumuomoum acoduuomoum uo amass: moaua>uuu< uo amass: mouoom unease accuses no >uefifism .nH canes 86 To determine whether the LEAP curriculum had a significant effect on the students' cognitive skills, two independent t-tests were calculated to compare the mean pre and post test score obtained by the treatment group of students to the mean pre and post test score obtained by the control group of students. To determine if there was a significant difference between the two groups of students at the onset of the LEAP program, a t-test on the pre test scores obtained by the treatment and control group of students was cal- culated. The results of this analysis are presented in Table 14. This analysis revealed no significant difference between the two groups of students at the onset of implementation. Table 14. Results Obtained for Control/Treatment Pre Test Analysis Average Number of Pre Test Standard Significance Group Students Score Deviation t-Value Level Control 94 35.03 6.32 .74 Treatment 212 35.64 6.49 .46 87 The results obtained from the t-test analysis of the mean post test scores obtained for each group are illustrated in Table 15. Based on these results, it can be concluded there was a significant difference in knowledge between the treatment and control groups at the conclusion of the LEAP implmentation. Table 15. Results Obtained from Control/Treatment Post Test Analysis Average Number of Post Test Standard Significance Group Students Score Deviation t-Value Level Control 89 36.44 5.73 2.92 .004 Treatment 193 38.78 7.20 Determining the Relationship Between Teacher Output and Student Outcome Measures To determine the extent to which the student outcome related to the teacher output, a Pearson product-moment coefficient was calculated on the average student gain scores obtained for each of the teachers and the respective overall teacher output score. The student gain scores obtained for each teacher are presented in Table 16. The correlation coefficient calculated was equal to .02. It appears that the extent to which a teacher implements the program as intended does not relate to 88 Table 16. The Average Student Outcome Gain Score for Each Teacher Teacher Average Student Gain Score 2.00 2.60 4.08 3.54 3.41 WUOW> extent to which the students increase their knowledge of the leisure concepts included in the LEAP curriculum. The teacher output score is comprised of a sum of eight variables. In an attempt to identify aspects of teacher output that did relate to student outcome, subse- quent analyses were conducted to determine the relationship between student gain scores and each teacher output variable. A Pearson product-moment correlation coefficient was calculated to determine the relationship between each teacher output variable and student outcome. Table 1? pre- sents the results obtained from these analyses. A review of Table 17 indicates a moderate correlation between the stu- dent gain score and the time spent implementing the LEAP curriculum (r = +.60). It appears that the more time spent. on implementation, the more likely the students were to gain knowledge regarding the leisure concepts. 89 Table 17. Results Obtained for Correlation Analysis of Teacher Output Variables and Student Gain Scores Correlation Teacher Output Variable Coefficient Number of leisure concepts implemented . . . . . . . . . . +.17 Proportion of leisure concepts integrated . . . . . . . . +.03 Number of Learner Activities/Target Area . . . . . . . . . +.33 Number of activities involving the community . . . . . . . +.05 Number of activities related to planning and decision making 0 C O O O C O O O O O O O O O O O O O I O O O O O - O 29 Number of activities related to self-awareness . . . . . . -.23 Number of Learner Activities relating to investigating Leisure Resources . . . . . . . . . . . . . . . . . . . +.26 Time spent in implementing . . . . . . . . . . . . . . . . +.60 Summary of Student Outcome Analyses The results obtained from the student outcome analyses indicate that the LEAP curriculum did increase the students' knowledge of the leisure. Based on the Pearson product- moment correlation coefficient calculated to determine the extent to which the overall teacher output score related to the student outcome measure, it appears that student outcome was not related to the extent to which the teachers imple- mented the program. However, subsequent analyses did reveal that student outcome did correlate to one aspect of teacher 90 output. Student outcome was moderately related to the time spent on implementation. Results Obtained for Phase III: Identificatign of Factors Relating to Teacher Output The purpose of Evaluation Phase III was to: (1) provide a description of project directive and incorporation activities surrounding implementation of the LEAP curriculum; (2) determine the relationship between project directive factors and teacher output; and, (3) determine the relationship between incorporation factors and teacher output. The results obtained for this phase of the evaluation are presented in two sections. The first section describes project directive activities and factors. The second describes incorporation activities and factors. Part I: Results Obtained for Project Directive Factors The intent of this section of Chapter IV is to provide a description of the various project directive activities that surrounded implementation of the LEAP curriculum and to determine the extent to which selected project directive factors relate to the teacher output measure. Although eight project directive factors have been included in this investigation, only three were included in the correlation analyses due to a lack of variability 91 among the teachers relative to the project directive measures. The three project directive measures that are included in the correlation analyses are: (1) teacher participation in the selection of the LEAP program; (2) teacher participation in the planning for implementation; and, (3) the teacher's perceived value of the LEAP inservice program. Although the remaining project directive factors are not included in the correlation analyses, they are discussed below in order to facilitate understanding of activities leading up to implementation of the LEAP program, and to provide an information base to guide future research endeavors. A description of each project directive factor included in this investigation follows. Criteria for selection of the program. The procedures used to select the career/leisure program to be implemented include: 0 Developing a philosophy statements; 0 Developing goal statements; - Conducting a needs assessment to determine the extent to which students needed instruction relative to the goal statements; and 0 Establish criteria for selecting a program to be implemented. 92 The Michigan State Department of Education conducted workshops to assist the district in conducting a needs assessment and to establish criteria for selecting a program to meet the identified needs. The criteria that were used to select the program are as follows: 1. 16. 17. 18. 19. 20. Should address leisure activities; Ease of implementation; Level of application (7-8); Adaptable to various teaching styles; Provide exposure to 100 percent of students; Meets needs of special groups (handicapped); Meets established goals; Allows for budgetary constraints; Portable/easily stored; Provides for evaluation; Has demonstrated success; Availability; Presence or absence of bias; Amount of time needed to use product; Infusability; Addresses life roles; Training, inservice required; Type of material (text, workbook, etc.); Reading level, grade comprehension; and Staffing needs (teacher, counselor). 93 Teacher level ofgparticipation in the selection of the LEAP program. The teachers were asked during their respective interviews to describe their involvement in the selection of the LEAP program. The responses obtained were used to rank order the teachers according to the extent of their involvement. An overview of each teacher's response and the rank score assigned to each follows: Teacher Response Score B & E Were involved in the project since its inception. Attended all scheduled meetings. Were involved in developing the criteria for selection. Reviewed several career education programs prior to the selection of LEAP. 1.5 C Was involved in the project since its inception; did not attend all scheduled meetings. Participated in a limited way in the selection of the LEAP program. 3 D Entered the project after the criteria had been developed. Participated in the actual selection process. 4 E Became involved with the project after the LEAP program had been selected. 5 Number and frequency ofgplanning meetings. The purpose of the planning meetings was to plan for implementation of the LEAP program. The specific tasks included: 0 Selecting the Target Areas and Focus Points for implementation. . Discussing how to implement the Target Areas and Focus Points selected. 94 - Discussing ideas and potential problems associated with implementation. . Developing test items to match the leisure concepts to be implemented. The meetings were scheduled as needed. A total of five planning meetings were scheduled over approximately a three-month time period. Teacher participation in the planning process. All teachers who implemented the LEAP program were involved in some aspect of the planning process, although their degree of involvement varied. An overview of each teacher's involvement in this planning process and the ranked score assigned follows: Rank Teacher Response Score B & E Attended all planning meetings. They were actively involved in the selection of the Target Areas and Focus Points, the various discussions regarding how to implement the program, and in the devel- opment of the test items. 1.5 D Attended four planning meetings and was actively involved in the writing of test items. Was actively involved in the selection of the Target Areas and Focus Points. 3 C Attended only three planning meetings and was actively involved in writing the test items, but was involved in the selection of the Target Areas and Focus Points. 4 A Entered the project after the Target Areas and Focus Points had been selected. Was actively involved in the writing of the test items. 5 95 Inservice training time. A one—half day inservice training session was scheduled for the teachers. The training session was conducted by two health, physical education, and recreation professors from Michigan State University. The inservice session focused on leisure concepts, the LEAP Target Areas, and the format of the LEAP Curriculum Guide. The inservice session also provided the teachers with specific examples of how the LEAP concepts could be integrated into various curricula. All but one teacher (Teacher A) attended this one-half day inservice session. Therefore, no ranking scores were assigned. Value of the inservice. The teachers were asked to rate the inservice session on a four point scale and to make specific comments on their perceptions of the strengths and weaknesses of the inservice session. Following is a summary of the comments made, and the corresponding ranked score assigned each teacher. The ranked score is based on the comments made in conjunction with the scaled score they had assigned to the inservice session. Teacher Response Score B Rated the inservice as "valuable" because it forced the teachers to get their "hands on" the curriculum mate- rials. The information presented was helpful, although there should have been some follow up on developing lesson plans. 1 96 Teacher Response Score C & D Rated the inservice as "somewhat valuable." Both teachers believed there was too much repetition of the content presented. 2.5 B Rated the inservice to be "of little value." Although the background information was interesting, there should have been an opportunity to talk with someone who had implemented the program before. 4 A This teacher did not attend the inservice session. 5 All teachers who attended the inservice session expressed the need to acquire more information on how to better integrate the leisure concepts into their existing curricula. The Project Director considered the weakest aspect of the inservice was that there was no follow up to assure the teachers had the necessary knowledge and skills to adequately implement the program. Resource support. Each teacher had access to a K-12 LEAP Curriculum Guide. The materials were distributed to the teachers prior to the planning for implementation. The guides were purchased through the Career Education Project funds. Community involvement. The community was involved in neither the selection nor the planning processes. The parents did receive a letter informing them that the LEAP program was going to be implemented. This letter presented 97 the rationale for the program and procedures planned for implementing the program. During the interview, both the principals and teachers were asked if they had received any feedback from the parents regarding the letter sent home. Only one parent was reported to have contacted the school. The parent expressed the concern that instructional time would be taken away from the primary subject area. Analysis of the Relationship Between PrOject Directive Factors and Teacher Output Table 18 presents a summary of the ranked scores assigned to each teacher for each of the three project directive factors to be included in the correlation analyses. Also included in this table are the Pearson product-moment correlation coefficients calculated for each factor to determine its relationship to the teacher output ranked scores. A review of the correlation coefficients found on Table 18 shows a high correlation (r==+.87) between the teacher's perceived value of the inservice session and the teacher output score. It appears that the more worthwhile the teachers considered the inservice to be, the more likely they were to implement the program as intended. 98 Table 18. Teacher Ranked Scores on Project Directive Factors and Correlation with Teacher Outputa Project Directive Rankings Teachers' Teacher Teachers' Teachers' Perceived Output Participation Participation Value of Teacher Rank in Selection in Planning Inservice A 4 5 5 5 B l 1.5 1.5 l C 3 3 4 2.5 D 2 4 3 2.5 E 5 . 1.5 1.5 4 Correlation coefficient +.10 +.21 +.87 Teacher participation in the selection and planning of the LEAP program revealed a low correlation to teacher out- put. Given the measures used for this analysis, the extent to which the teachers are involved in the selection and/or planning for the LEAP program does not relate to the extent to which the teachers implement the program. Part II: Results Obtained for Incorporation Factors This section includes a description of the incorpo- ration activities that surrounded implementation of the LEAP program and the results obtained from a series of correlational analyses to determine the relationship between the incorporation factors included in this 99 investigation and teacher output. A description of the Incorporation activities and the teachers' ranked scores assigned, based on the extent to which they were involved in the Incorporation activities, follows. Communicability of the innovation. To determine the communicability of the LEAP program, the teachers were asked to rate the extent to which they considered the LEAP curriculum objectives and directions for implementation were clearly stated; and, to describe the aspects of the program they found to be confusing. An overview of the teachers' responses to these questions and their ranked score assigned, based on their comments, are as follows: Rank Teacher Response Score E Considered "all" of the objectives to be stated clearly and the directions for implementation to be "very clear." The primary con- fusing aspect of the program related to procedures for integrating the leisure concepts into various curricula. l A Considered "all" of the objectives to be stated clearly, but had a difficult time interpreting the directions on how to use the LEAP Curriculum Guide materials. The most confusing aspect of the program was how to best integrate it into existing curricula. 2 D Considered "most" of the objectives to be stated clearly and the directions to be "somewhat" clearly stated. This teacher would have liked to have had more specific suggestions on what learner activities could be presented in the classroom. No aspects of the program were found to be confusing. 3 100 Teacher Response Score B Considered "most" of the objectives to be stated clearly and the directions for implementation to be stated "somewhat" clearly. A considerable amount of time was required to pull from the Curriculum Guide the information needed to teach the leisure concepts. To facilitate quick retrieval of lesson activities would require a different indexing system. 4.5 C Considered "most" of the objectives to be clearly stated and the directions to be "somewhat" clear. The confusing aspect of the program reported by this teacher related to the terminology included in the Curriculum Guide. To understand the concepts to be implemented required a great deal of time. 4.5 The Project Director considered the LEAP program to be well developed, although clarification was needed on how to integrate the leisure concepts into various curricula. The Project Director noted a great deal of confusion on how to integrate the program on the part of several project teachers. This confusion was, to a great extent, attributed to the high project staff turnover rate. Complexity of the innovation. To determine the teacher's perception of the complexity of the innovation, the teachers were asked to rate the extent to which they considered the LEAP curriculum to be difficult to implement and to describe the various modifications in the curriculum they had to make during implementation. 101 A summary of the information collected through the teacher interview and the ranked scores assigned to each teacher based on this summary follows: Teacher B Response The program was not found to be difficult to implement. The program had to be supplemented with leisure activities that related to math. The program was not considered to be difficult to implement. Many of the leisure activities included in the curriculum guide related well to social studies. The major modifi- cation that had to be made was to alter some suggested learner activities because they were thought to be at too low a level for the students. The curriculum materials were not difficult to implement. However, it took a great deal more time to implement the various leisure con- cepts than anticipated. It was difficult to relate some concepts to English. A great deal of planning time was required. The program was difficult to implement as it required too much time to plan and implement. The program was con- sidered to be complex due to the lack of direct relationship of the leisure concepts to math concepts. This teacher reportedly spent a great deal of time planning for instruction to compensate for the lack of suggested learner activ- ities relating to math within the LEAP materials. Score 1.5 102 Rank Teacher Response Score E The LEAP program was found to be very difficult to implement because of the nature of the Title I classes. The class time was short in duration (20 minutes), which interfered with attempting to present the leisure concepts along with the required remedial activities. Also, the learner activities were too difficult for the Title I students. Integration was near impossible, as the integration seemed to interfere with their comprehension of the leisure concepts. 5 The Project Director attributed the most complex aspect of the LEAP implementation to the time required to plan for integration. Because the district's teachers had previous experience in working with goals and objectives, the orga- nization of the LEAP material was not confusing. In fact, it lent itself well to the existing objective based system. Perceived value of the innovation. The teachers were asked to rate the extent to which they agreed with a series of statements relating to the LEAP program. The responses obtained for each teacher for each statement are presented in Table 19. Based on the results presented in Table 19, it appears the teachers, overall, considered the LEAP curriculum to be valuable to their students. The statement that received the lowest overall rating (3.20) related to the extent to which the LEAP curriculum interfered with the students' learning in the primary subject area. The teachers were 103 Table 19. Teacher Responses to Statements Regarding the Value of the LEAP Program Teacher Responsesa Mean a Statements A B C D E Responses The "leisure life role” components of career education is important to the development of the students. 5 S S 5 5 5.00 I believe it is important for students to be able to explore many leisure activities. 4 S S S 5 4.80 I believe it is important for students to relate to leisure roles with life roles of family, citizen, and worker. 3 5 5 S S 4.60 The LEAP curriculum assists the students in relating leisure with other life roles. 4 l S 4 S 3.80 The LEAP curriculum assists the student in identifying leisure activities he/she is interested in. 5 5 5 4 S 4.80 The LEAP curriculum does NOT interfere with the student's learning in your primary subject area. 3 3 5 3 2 3.20 The LEAP curriculum was easily integrated into the existing curriculum. 3 4 5 4 l 3.40 Overall, the LEAP curriculum was worthwhile to the students. 4 S 4 4 4 4.20 The LEAP curriculum helps to: Increase students' awareness of themselves (i.e., their likes. dislikes, values, etc.). 4 4 4 4 4 4.00 Increase students' awareness of others. 4 4 4 4 3 3.80 Increase students' awareness of career opportunities. 4 4 4 4 3 3.80 Increase students' ability to make decisions. 4 S 4 4 4 4.20 Assist the students in making their career plans. 4 4 4 4 4 4.00 3.92 4.15 4.54 4.15 3.85 4.12 aTeacher mean responses based on a five point scale where S-Istrongly agree and l= strongly disagree. more inclined to agree that the LEAP curriculum did 104 interfere to some degree with learning the primary subject area. Another identification of the teachers' perceived value of the LEAP curriculum was that they would like to continue implementation next school year and would like to see implementation of the LEAP curriculum expanded within their school. based on their the additional Teacher C The ranked score assigned to each teacher, responses to both the statement items and items, are as follows: Response The overall response for the statement items equals 4.54. This teacher is interested in continuing implementation next year and is currently working with the curriculum director in an effort to expand implementation next year. The overall response for the statement items equals 4.15. This teacher is interested in continuing implementation next year and would like to see imple- mentation expanded to other curriculum areas within the school. The overall response for the statement items equals 4.15. This teacher expressed the desire to use some of the LEAP ideas next year, but not on such a formal basis as was done this year. She found it to be very strenuous. This teacher would like to see aspects of the LEAP program imple- mented in some other curricula areas next year. The overall response for the statement items equals 3.92. This teacher is interested in continuing implementation next year and would like to expand to other curricula areas. Score 105 Teacher Response Score E The overall response for the statement items equals 3.85. This teacher would not like to implement LEAP with Title I students next year, but would like to implement it with seventh and eighth grade students. She would also like to see it expanded to include more teachers. 5 Support from the principal regarding the program. The principals were asked a series of questions to determine the extent to which they supported the LEAP program. The teachers were also asked to rate the extent to which they considered their principal to be supportive of the LEAP program. A summary of the responses obtained and the corresponding teacher ranked scores follows. Teacher Response Score B The teacher perceived the principal to be "very supportive" of the LEAP program. The principal reported the program to be very important to the students. The principal was willing to provide a sub- stitute teacher so Teacher A could attend the required meetings. The principal is interested in expanding implementation to other teachers in the building. 1 D The teacher perceived the principal to be "very supportive" of the LEAP program. In fact, the principal has even taken over the teacher's class on occasion to allow her to attend the required meetings. The principal considered the LEAP program to be very important as it "supports the school's educational goals." Would like to expand to other curricula areas if the teachers were properly informed and were enthusiastic about implementation. 2 106 Teacher Response Score E The teacher perceived the principal to be "very supportive" of the LEAP program. The principal has provided support to Teacher E by arranging for substitute teachers for the required project meetings. He is interested in expanding implementation providing the enthusiasm for implementation is present. 3 A The teacher perceived the principal to be "very supportive" of the LEAP program. The principal considered the program to be very important, as it supported the school's goals. The principal is not sure whether to expand implementation next year. The teachers would have to be well informed and express an interest in participating. So far, the interest is not there. 4.5 C The teacher perceived the principal to be "very supportive" of the LEAP program. The principal was "somewhat" interested in continuing implementation next year. 4.5 Morale of the teachers in the school. To determine the morale of the teachers in the school, the principals were asked to rate the morale of their staff using a five- point scale (where 5==high morale and l==low morale). Using the same five-point scale, the teachers were also asked to rate the morale of the other teachers in their building and to rate the extent to which they perceived the teachers in their building to be "enthusiastic towards teaching" (on a three-point scale: very, somewhat, not at all). A summary of the responses accumulated and the corresponding teacher ranked scores for this factor follows. 107 Teacher Response Score A Both principal and teacher rated the teachers' morale as a 4.5. The teacher considered the teachers to be "very" enthusiastic towards teaching. 1 C The principal rated the teachers' morale as a 3.0. Teacher C rated the teachers' morale as a 5.0. The teachers were considered to be "very" enthusiastic towards teaching. 2 D The principal rated the teachers' morale as a 5.0. The teachers' morale was rated as a 4.0 by Teacher D. The teachers were considered to be "fairly" enthusiastic towards teaching 3 E The teachers' morale, as rated by the principal equalled a 3.5. Teacher E rated the teachers' morale to be a 3.0. The teachers were considered to be "very" enthusiastic towards teaching. 4 B The principal rated the teachers' morale as a 3.0. Teacher B rated the teachers' morale as a 3.0 and perceived the teachers to be "somewhat" enthusiastic about teaching. 5 Innovativeness of the school. The principals were asked to specify the number of innovative programs that had been developed and/or implemented within the last school year. The responses obtained were used to rank the teachers relative to this incorporation factor. The results are as follows. 108 Rank Teacher Resppnse Score B Within the last school year, this school implemented a new math program, a new history program, and had developed a career planning class which was offered as an elective to junior high students. 1 A A new math program is in the process of being developed. A new social studies text book has been adopted. 2.5 E A new Physical Growth and Development program had been developed within the last year. Also, the students were provided a new Team Forum Class to discuss problems. 2.5 D The school has changed from term to semester schedule. The principal is in the process of attempting to change the teachers' view of the school from a junior high to a middle school concept. 4 C A KelZ coordinator position has just been established. The school was in the process of developing curriculum goals and objectives. 5 Support of non-implementing teachers. The principals were asked to indicate the extent to which they felt the teachers in their building were supportive of the LEAP curriculum. The project teachers were asked whether or not they had talked about this program with other teachers in their building and to then indicate how much interest in the program the other teachers in their building had shown towards the LEAP program. Based on the interview responses obtained, it was discovered that the other teachers knew very little about the LEAP program or even about the Career 109 Education project. A few teachers had mentioned the LEAP program to some teachers in a casual one-on-one setting. Others had talked to the entire staff about career educa- tion, but had not mentioned the LEAP program specifically. Therefore, it was not possible to adequately discriminate between teachers relative to their responses in order to assign ranked scores. Perception of Career Education. To ascertain the teachers' perception of Career Education, they were asked to rate the extent to which they agreed with a series of statements regarding Career Education and the school's role in incorporating aspects of Career Education into the school's program. The teachers' responses to these series of statements are presented in Table 20. A review of Table 20 indicates the teachers were very positive about Career Education and the school's responsibilities to imple- ment various aspects of Career Education. The lowest over- all response equaled 4.2, and the highest equaled 5.0. The individual teacher's overall responses also reveal that each teacher was positive about Career Education concepts. No mean response fell below 4.2. Time spent ongpreparing lessons. The teachers were asked to report the average number of hours they spent on preparing a lesson for implementation. A summary of the responses obtained and the teacher's assigned ranked scores based on their responses are as follows. 110 .meumewwo wamcouumnua one seems hamcouum um mum£3 mamom unmxwa ucw0m|o>wm o co commas m.H m.m o.m m.m m.H swoon omxcmu .muwnomwe h.v o.m o.v m.v w.v o.m uncommon some .mumcomms o.m m m m m m mcmHm umwumo Hemnu ocexma cw mucmpsum use umwmmm . o.v m e v m m mGOflmwomo wxufi ou aueawnm .mucoosum owmwuoca . m.v m m e m m meeuecsuuommo womwmo mo mmmcoum3m .mucepsum unmouocH . m.v m m v m m muscuo mo mmwsmumsm .mucwpsum mummuocH . ~.v m v v m m A.ouw .mesHm> .mexeamwo .mmxea Hausa ..m.ev mG>H0mEGSU MO mmeOHM3M .MUflwUSHm mwmeUGH e ”on hueawnemcommou m.Hoonom on» me DH M newcommmm m a U m c new: mummcommwm nuances ucoEmumum cowumoapm umoumo mo maceumeoumm on mcwumaem mucmaoumum ou noncommom .muenomma .om manna 111 Teacher Response Rank Score 4 hours 2 hours 1% hours 1% hours wmmon wwwNI—I 01m 1 hour Agalysis of the Relatipnship Between IncorporatiEn Factors and Teacher Output Table 21 presents a summary of the ranked scores assigned to each teacher for the incorporation factors included in the correlation analyses. Also included in the table are the Pearson product-moment correlation coefficients obtained from the analyses. Based on the results presented in Table 21, five factors included in these analyses appear to relate to teacher output. Communicability of the program was negatively related to the teacher output ranked scores (r = -.82). This correlation coefficient implies that the more the teachers perceived the program to be easy to understand, the less likely they were to implement the program. A second factor which related highly to the teacher output score was complexity of the program (4 = +.82). Based on these results obtained, it appears that the less difficult a teacher perceives implementation of the program, the higher the teacher output score. 112 ma... 3... o~.+ on... 5.1. oo.+ no... «of ucoauwuusoo ceaueaeuuoo m n m4 m.n v n m m a m m N m.n v n a n n n w a a m m N we a m4 m.v n U m m.n H m a n m4 m4 H m m.n m4 m.~ .— mé v v w v a. scanned mcawemeum sequeosom Hoozum uo cacao: senoOum sewmoum Isuooum uo asumoum mo xcea accuses ca usedm slab useusu uo eeoco>uue>occn uozoeoa no usedmsm uo os~e> muaxenmloo huuuansuwcslsou usmuso escaudeuuom se>aouuem eo>aoouem assuage mama—5 success on scauemeuuoo use euouoem scaueuomuoucn co sououm vexed: essence. 2: ounce. 113 A moderate correlation was found between how the teachers perceived the extent to which the principals in their building supported the implementation of the LEAP curriculum and their teacher output score (r = +.67). These results imply a tendency for teachers to implement a program when principals show support for the program. How a teacher perceives Career Education was also moderately correlated to teacher output (r = -.63). This correlation implies the more teachers considered Career Education to be of value and a responsibility of the school, the less likely the teachers were to implement the LEAP curriculum as intended. A moderate correlation (r = +.60) between the teachers' perceived value of the program and teacher output was found. There appears to be a tendency for teacher output to be greater when the perceived value of the program is greater. A Discussion quthe Appropriateness of the Evaluation Plan The procedures used to collect data for teacher output were most appropriate. The interview technique provided a wealth of information from each teacher regarding implemen- tation and problems encountered during implementation. The self report method also proved to be a good data collection method. Because the teachers could complete this form throughout implementation, the data collected appeared comprehensive. 114 The observation of teachers did not provide any additional information, but did help to verify, to some extent, information collected via the interview and self report instruments. It would have been more appropriate to have developed an instrument to guide the observation and to provide additional documentation of teacher implementation. The evaluation plan supplied a measure of teacher output for each teacher. All but one of the nine teacher output variable measures provided a variance in scores across the five teachers. Although the results obtained for each teacher output variable could be interpreted and did describe how each teacher was implementing the LEAP curriculum for several variables, interpretation of the results could be discussed only in light of how teachers implemented compared to each other. It would have been more appropriate to have deter- mined an expected score for each variable prior to data collection. The evaluation plan provided an adequate measure of student outcome. Although no relationship was found between student gain scores and the teacher output score, the pro- cedures used to determine if a relationship exists were appropriate. The fact that only one teacher output variable (time spent on implementing) related to student outcome, 115 even though the students did learn as a result of the program, could be due to the teacher output variables selected for the evaluation of the LEAP program. There may well be other teacher output variables not included in the teacher output analysis that would have related to student outcome. Overall, the evaluation plan provided a means for discriminating between teachers in the extent to which a project directive and incorporation factor was present. Therefore, for most factors, it was possible to rank order the teachers. The interview technique was an effective and efficient method for data collection as it provided in-depth responses to the items contained on the interview forms and required a relatively small amount of time. In some instances, however, the interview items failed to provide an adequate measure for a factor. For example, the item intended to measure the teachers' involvement in the inservice session proved not to discriminate between teachers. To adequately measure the factor, it might have been more appropriate to have included items pertaining to how teachers used the information presented at the inservice session. Interview questions relating to the extent to which non-implementing teachers supported the LEAP cur- riculum might have been expanded to determine more 116 specifically: (1) about which aspects of the LEAP curriculum had teachers been informed; (2) the proportion of teachers in the building who had been informed; and, (3) the extent to which the informed teachers showed an interest in implementing the program and/or made positive comments regarding the program. CHAPTER V SUMMARY AND RECOMMENDATIONS The purpose of this study was to develop a plan for evaluating innovative programs in education and to determine the appropriateness of the plan by applying it to the eval- uation of a career/leisure program. The evaluation plan focused on teacher implementation and factors relating to teacher implementation. The extent to which teachers implement an innovative program is likely to vary from one teacher to another, as well as vary from what was intended by the developer of the innovation. In order to more accurately interpret student outcomes, it is essential that the extent to which teachers implement the program be determined. To gain a better understanding of implementation, it is necessary that factors relating to implementation be identified. The investigator perceived implementation of an innovative program to be a process comprised of four inter-related components: 1. Project Directive: The process of planning for implementation. 117 118 2. Incorporation: The process of adopting the innovation into an existing school environment. 3. Teacher Output: The extent to which the teacher implements the program as intended. 4. Student Outcomes: The extent to which the program changes student behavior. The plan developed to evaluate innovative programs includes three phases. The first phase centers on teacher output and incorporates procedures used to obtain a description of and a measure of teacher output. The second phase includes procedures to evaluate student outcomes and to determine the relationship between student outcomes and teacher out- put. The third evaluation plan relates to techniques used to measure project directive and incorporation factors and to determine the relationship between: (1) teacher output and project directive factors, and (2) teacher output and incorporation factors. To ascertain the feasibility of the evaluation plan, it was applied to the evaluation of a career/leisure pro- gram. Of interest to the investigator was whether or not the evaluation plan could provide an adequate measure of teacher output, student outcome, project directive factors, and incorporation factors; and, whether or not the plan could supply an indication of the relationship between: teacher output and student outcome; teacher output and 119 project directive factors; and, teacher output and incorporation factors. A summary of each evaluation phase follows. Each summary includes procedures and results obtained for the evaluation of the career/leisure program and a summary of the extent to which the evaluation plan provided appropriate measures. Summary of Phase I: Evaluation of Teacher Output The purpose of this evaluation phase was to: (1) measure the extent to which each teacher implements the program; (2) provide a description of program components implemented; and, (3) identify problems encountered during implementation. The career/leisure program evaluated was the Leisure Education Advancement Project (LEAP). The program was implemented for eight weeks by a small number of teachers (N==5) in five different schools. A total of 211 students were provided instruction in the LEAP program (193 students were in seventh grade and 18 were in fifth grade). The teachers were to integrate the LEAP Curriculum into their existing curriculum area. There were three LEAP Target Areas to be implemented, Target Areas 1, 4, and 8. To identify teacher output variables for the LEAP program, the LEAP Curriculum Guide and content presented 120 at a teacher inservice program were reviewed. Also, the teachers were interviewed in a group setting to obtain a description of their perceptions of how the LEAP program was to be implemented. Nine teacher output variables were identified. Three methods were used to collect data on the nine variables: interview, self report, and observation. An interview instrument containing 12 questions was developed. The item format included a combination of yes-no responses and open-ended reports. The self report form contained a list of the leisure concepts to be implemented. The teachers were asked to document the date they implemented each concept. A 30 minute observation of each teacher was made to validate the responses obtained on the interview and self report instruments. No observation instrument was developed. A scoring technique incorporating data collected from the interview and self report instruments was used to quan- tify the degree to which the teachers were implementing the LEAP program. The scores obtained for each variable were transformed into standard z-scores. The overall teacher output score was represented by an average of the z—scores. The analyzed data showed that no teacher was able to implement all of the 28 leisure concepts selected for implementation. The total number of leisure concepts 121 implemented per teacher ranged from 15 (54%) to 21 (75%) leisure concepts. The extent to which the leisure concepts were inte- grated into the existing curricula varied widely among the teachers. One teacher integrated only 6 percent of the leisure concpets taught. Another was able to implement 50 percent of the leisure concepts. Across all teachers, 38 percent of concepts were integrated into existing curricula. The average per Target Area 1 was 6.8; for Target Area 4, an average of 5.6 activities were implemented; and, for Target Area 8, an average of 2.6 activities were implemented. The average number of learner activities implemented per Target Area for each teacher ranged from 4.33 to 5.67. On the average, the teachers implemented 2.8 learner activities which involved community members in the LEAP curriculum. The community was involved in three ways: (1) by being involved in a career/leisure activity day; (2) by being interviewed by a student regarding how they used their leisure time; and, (3) by receiving a letter explaining the LEAP curriculum. Four of the five teachers implemented only one learner activity that involved the students in the process of plan— ning and/or decision making. The fifth teacher implemented 122 four. The activities implemented included: (1) planning for a career/leisure activity day; (2) planning a leisure resources file; (3) planning a trip; (4) selecting commu— nity members to speak on careers as hobbies; and, (5) determining "why" people decide to do "creative" as opposed "criminal" types of leisure activities. On the average, the teachers implemented 2.4 learner activities relating to self-awareness. The number of activities implemented by each teacher ranged from one to three. The activities included: (1) keeping a daily record of the amount of leisure time they had and the types of activities they did during this time; (2) rating their feelings and attitudes towards various activities; and, (3) thinking about what leisure activities they would do if placed in various environmental settings. Because all teachers taught the Focus Points in sequence, each teacher received the same teacher output score. The number of Learner Activities implemented that provided an opportunity for students to investigate leisure resources ranged from zero to two. There was an average of one activity across the five teachers. The activities implemented that related to the investigation of leisure resources included: (1) developing a file of where to find information regarding selected leisure activities; 123 (2) investigating the history of various leisure activities; and, (3) investigating how to do various leisure activities. The average number of hours, across the five teachers, devoted to the teaching of the LEAP curriculum was equal to 5.67. The range of hours spent on implementation was from 4.36 to 7.00 hours. Averaging the teacher variable standard scores obtained for each teacher resulted in a range of standard scores from +.37 to -.70. Three major problems were encountered during imple- mentation of the LEAP curriculum. First, every teacher expressed the need for more time to implement the leisure concepts. Several school days were lost due to snow and due to a high number of student absences (as a result of a flu epidemic). A second problem expressed by the teachers related to their inability to adequately integrate all the leisure concepts into their curriculum areas. A third problem related to the reported time needed to plan for instruction. The procedures used to measure teacher output were found to be appropriate. The measures for the teacher output variables, for the most part, provided a variance in scores across the five teachers. The interview and self report methods provided ample information to describe imple- mentation and problems encountered during implementation. 124 A weakness found in Phase I of the evaluation plan was the lack of an explicit expected score for each teacher output variable. Prior knowledge of an expected score would have facilitated interpretations of the teacher output variable scores. Summary of Phase II: Evaluation of Student Outcome The purpose of this evaluation phase was: (1) to determine the extent to which the students were impacted by the career/leisure program; and (2) to ascertain the extent to which the student outcome measure relates to the teacher output measure. Student outcome was measured by a 60 item cognitive skills test. The test was administered on a pre to post test basis to students receiving instruction in the LEAP curriculum and to a control group of students. All tests were machine scored and computer analyzed. Independent t-tests were used to compare the difference between the treatment and control groups of students for the pre test and post test scores. To analyze the rela- tionship between student outcome and teacher output, a Pearson product-moment correlation coefficient was cal- culated using the average student gain scores obtained for each teacher and the teacher output score. The independent t-tests revealed there was no signif— icant difference between the control group and treatment 125 group of students at the onset of the program. However, there was a significant difference (alpha level==.004) at the conclusion of the program. The Pearson product-moment correlation between the average student gain scores for each of the teachers and the respective teacher output scores was equal to .02. Analysis of the student gain scores and each teacher output variable measure identified only one variable that moder- ately related to student gain. This variable was the time spent on implementation (r==+.60). The procedures used to measure student outcome were appropriate. The procedures selected to determine the extent to which student outcome related to teacher output were appropriate although no relationship was found. It could be that other teacher output variables not included in this study would have related to student outcome. Summary ovaaluation--Phase III: Identification of Factors Relating to Teacher Output The purpose of this evaluation phase was to: (1) provide a description of project directive and incorporation Vactivities surrounding implementation; (2) identify project directive factors relating to teacher output; and, (3) iden- tify incorporation factors relating to teacher output. Seven project directive factors and six incorporation factors were included in the evaluation plan based on a 126 review of related literature. In addition, one project directive and three incorporation factors, which were related to the LEAP program, were included. The procedures used to measure the project directive and incorporation factors included in-depth interviews with the project director, implementing teachers, and the school principals. Three instruments were developed (one for each of the three sources of information). The Teacher Interview form included 39 items. The Project Directive Interview instrument contained 16 items and the Principal Interview form included 10 items. Each instrument included a combi- nation of scaled response items, yes/no response items, and open-ended response items. To obtain a description of the project directive and incorporation factors surrounding implementation, the data were reviewed and summarized. To determine if each factor included in this study affected teacher output, the infor- mation collected that related to each factor was reviewed. Teachers were given a rank score based on the extent to which a factor was present relative to the other teachers. The teacher output scores were also ranked. A Pearson product-moment correlation coefficient was calculated to determine the relationship between the ranked scores. Based on the results of the analyses, it was found that the teachers used criteria for selecting the LEAP 127 curriculum. Prior to selecting the curriculum to be implemented, goal statements were established and a needs assessment conducted. The teachers' level of participation in the selection of the LEAP curriculum varied. Some teachers were involved from the inception of the project. Others were not involved until after the LEAP curriculum had been selected. Planning meetings were held prior to implementation. The purposes of the planning meetings were to: (1) select aspects of the LEAP curriculum to be implemented; (2) dis- cuss how to implement the curriculum; (3) discuss potential problems associated with implementation; and, (4) develop test items to match the leisure concepts to be implemented. The extent to which the teachers were involved in the planning meetings varied. One teacher attended all meet- ings. Another was involved only in the writing of test items. The teachers attended a half-day inservice session which focused on implementation of the LEAP curriculum. All but one teacher attended this inservice session. Those teachers who attended the inservice were asked to rate the extent to which they considered the inservice to be of value. The responses obtained were varied from "valuable" to "of little value." 128 Each teacher had access to a K-12 LEAP Curriculum Guide. The materials were distributed to the teachers prior to the planning meetings. The community was involved in neither the selection nor the planning process. Little parent feedback, regarding the LEAP curriculum, had been received. The extent to which the teachers perceived the LEAP curriculum to be easy to understand varied from one teacher to another. The Project Director, in general, considered the program to be well developed. The teachers ranged in their responses relative to the extent to which the objectives and directions for implementation were stated clearly. Teachers varied in their responses regarding the complexity of the program. The responses ranged from "very difficult to implement" to "not at all difficult to implement." Complexity, by most teachers, was ref- erenced to the process of integrating the LEAP concepts into existing curricula. The teachers, overall, considered the LEAP curriculum to be valuable to their students. A noted concern regarding the value of the program was the time it took away from the primary curriculum area. The principals, overall, were quite supportive of the LEAP curriculum and were willing to provide assistance 129 needed to support implementation. All teachers considered their principals to be "very supportive." The principals, however, were more varied in the responses they gave to indicate their support for the program. The morale of teachers in the buildings where the LEAP curriculum was implemented varied. On a five-point scale, where 5==very high and l==very low, the morale of teachers were ranked from 4.5 to 3.0. The teachers were considered to be from "very" enthusiastic" to "somewhat" enthusiastic about teaching. The extent to which the schools were considered to be innovative varied. Within the year prior to implementation, the number of other innovative programs developed and/or implemented ranged from three to one. Few teachers in buildings where the LEAP curriculum was being implemented had been informed of the LEAP imple- mentation. It was not possible to rank the teachers on this factor because of the lack of variation in the obtained responses. The responses to the interview items that asked teachers to rate the extent to which they considered various aspects of Career Education to be a responsibility of the school were varied. Based on a five-point Likert scale, where 5==strongly agree and l==strongly disagree, the lowest response equaled 4.2 and the highest equaled 5.0. 130 The individual teacher's overall response indicated that each teacher was positive about Career Education concepts. Correlation analyses were performed to determine the relationship between the teacher output measure and the project directive and incorporation factors (for which a variation in scores appeared). These analyses revealed one project directive and five incorporation factors to be related to teacher output: Project Directive Incorporation How the teachers The communicability of perceived the the program (r==-.82) inservice to be of value (r==+.87) The complexity of the program (I = +. 82) The perceived support of the program (r= +.67) The perceived value of the program (r=+.60) Perceptions of Career Education (1' = -. 63) The evaluation plan, for the most part, provided a means for discriminating between teachers in the extent to which a factor was present. The interview technique provided a description of factors surrounding implementation. In some instances the interview items used to measure some factors could have been expanded to provide a more explicit description of the extent to which each factor was present for each teacher. 131 Conclusions Regarding the LEAP Curriculum Evaluation The conclusions related to the evaluation of the LEAP curriculum are presented in three sections: conclusions regarding teacher output; student outcome; and, factors relating to teacher output. Conclusions Regarding Teacher Output 1. The extent to which the teachers were able to implement the 28 leisure concepts was relatively high. (Three teachers implemented at least 20 [71%] of the concepts and two teachers implemented at least 15 [54%] of the concepts.) The extent to which the leisure concepts presented were integrated into the LEAP curriculum was rela- tively low. (The highest percentage of leisure concepts presented was 50 percent--the lowest 6 percent.) It appears that the method selected for implementation (integration) was not appro- priate for all curriculum areas. Overall, the teachers developed and implemented a relatively high number of learner activities involving the community and activities relating to students self-awareness. 132 Overall, the teachers developed and implemented relatively few learner activities associated with planning and decision making and investigating leisure resources. The extent to which learner activities were developed and implemented for each Target Area differed greatly with the highest number of learner activities relating to Target Area 1, fewer to Target Area 4, and fewest to Target Area 8. The amount of time spent on implementation varied to a large extent across the teachers (from 4.33 hours to 7.00 hours). Problems associated with implementation included: (a) an insufficient amount of time to implement all leisure concepts; (b) time required to prepare for integrating the leisure concepts; (c) conflict in time taken away from primary curricula area to implement LEAP. Conclusions Regarding Student Outcome 1. The students learned, as a result of the program. The students involved in the LEAP curriculum obtained a significantly higher test score (alpha==.004) at the conclusion of implementation than students not involved in the curriculum (even though there was no significant difference between 133 the two groups of students at the onset of the program). The student outcome measure did not relate to the overall teacher output measure. The student outcome measure was moderately correlated to the time spent on implementation (r = +.60). Conclusions Regarding Factors Relating to Teacher Output 1. The more teachers perceived the inservice to be of value, the higher the teacher output score. The more teachers considered the LEAP curriculum to be easy to understand, the lower the teacher output score. The more teachers considered the LEAP curriculum to be difficult to implement, the lower their teacher output score. The more teachers perceived the LEAP program to be of value, the higher the teacher output score. The more the principal supported implementation of the LEAP curriculum, the higher the teacher output score. The more teachers perceived Career Education to be of value to students and a responsibility of the school, the lower the teacher output score. 134 7. Teacher output did not relate to the following factors: a. Teacher involvement in the selection of the LEAP curriculum. b. Teacher participation in planning sessions prior to implementation. c. Morale of teachers in the building. d. Innovativeness of the school. e. Time spent in preparing lessons. Conclusions Regarding the Evaluation Plan The conclusions drawn based on analysis of the evaluation plan are as follows. Conclusions are presented for teacher output, student outcome, and factors relating to teacher output. Conclusions Regarding Teacher Output 1. The evaluation plan for teacher output provided an adequate measure of teacher output (the mea- sures for all but one teacher output variable supplied a variation in the scores obtained). 2. The data collection procedures provided a description of the LEAP implementation. 3. The data collection procedures identified problems encountered during implementation. 135 Conclusions Regarding Student Outcome 1. The student outcome measure was adequate (the test was reliable and matched the objectives of LEAP curriculum). Although the methods for determining the rela- tionship between student outcome and teacher output seemed appropriate, no relationship was found except for one teacher output variable. It could be the teacher output variables included in the study were not appropriate. Qonclusions Regarding Factors Relating to Teacher Output 1. Factors relating to teacher output can be identified. The procedures used provided a description of project directive factors surrounding implementation. The procedures used provided a description of incorporation factors surrounding implementation. 136 Recommendations Regarding Future Implementation of the LEAP Curriculum Based on the results obtained from evaluating the LEAP curriculum implementation, it is recommended that in the future, teacher inservice be expanded to assure that teachers feel comfortable and confident in implementing the program. Specifically, the inservice should focus greatly on how to integrate the leisure concepts into existing curricula. The value of the LEAP curriculum should be emphasized. Finally, the problems experienced by the five teachers during implementation should be shared with those teachers planning to implement programs in the future. If implementation is to be expanded into other buildings, attempts should be made to attain or maintain the principal's support for the program. Involving the principals in the implementation planning stage and keep- ing him or her informed of the activities surrounding implementation might be the means to attain or maintain the principal's support of the LEAP curriculum. Although the difference between the control group and treatment group of students at the conclusion of the program was found to be statistically significant, it is recommended that those in the position of making decisions regarding the LEAP curriculum consider the extent to which the gain made — . 137 by the treatment group of students is meaningfully significant. Does the gain made by the students warrant expanding implementation given the time and cost involved? Because the extent to which the teachers were able to integrate the leisure concepts into existing curricula was low, it is recommended that alternative implementation strategies be investigated. The teacher output variables included in the eval- uation should be reexamined. It is recommended that the description of implementation provided by the evaluation of teacher output be reviewed and the LEAP Curriculum Guide be reviewed in an attempt to identify teacher output vari- ables which might relate to the student outcome measure. It is also recommended that the possibility of weighting certain teacher output variables be considered. Future evaluations of the LEAP curriculum should continue to identify factors affecting teacher output. Although no relationship was found between teacher out- put and some factors, based on this initial evaluation, future implementations might reveal a relationship. This evaluation provided a glimpse of the imple- mentation process surrounding the LEAP curriculum. To continue to gain knowledge regarding teacher output and factors affecting teacher output would require additional 138 investigation. This evaluation has supplied a data base for evaluations of future implementations of the LEAP curriculum. Recommendations Regarding the Evaluation Plan Future use of the evaluation plan developed for this investigation should expand the procedures used to measure teacher output to include the identification of expected outcomes for each teacher output variable. Comparing teacher output scores obtained for each variable to that which is expected would provide a context in which to better interpret the obtained teacher output results. Because certain teacher output variables may be per- ceived to be more important than others, it is recommended that for future application of the evaluation design, the possibility of assigning weights to the variables be considered. It is recommended that a measure of teacher output be an ongoing process. Implementation is likely to change over time. As implementing changes, the instruments used to measure teacher output will also need to change. Because student outcome did not relate to the total teacher output score, but did relate (to a moderate extent) to one teacher output variable, it is recommended that future evaluations using this plan analyze the relationship 139 not only between the total teacher output score and student outcome, but each teacher output variable and student outcome. It would be valuable to determine which aspects of implementation affect student outcome. If the evaluation plan is applied to a program implemented by a large number of teachers, teacher out- put data analysis might be expanded to include a series of independent t-tests to determine if there is a significant difference between high implementing teachers and low implementing teachers relative to student gain scores. This evaluation plan provided a means for determining the relationship between teacher output and student outcome; teacher output and project directive factors; and, teacher output and incorporation factors. This evaluation plan might be expanded to include procedures for determining the relationship between student outcome and project directive factors and student outcome and incorporation factors. It is recommended that measures for the teacher output variables and factors be as specific as possible to produce varied responses. Caution must be used, however, to main- tain as small a number of questions as possible for each teacher variable and factor in an effort to keep the time involved in data collection to a minimum. (A large amount of information collected can be overwhelming to those who provide the information as well as to those who must col- lect, analyze, and report the data.) If a more thorough 140 investigation of a variable(s) and/or factor(s) is desired. it would probably be best to limit the total number of variable/factors to be investigated. Implications for Future Studies on Implementation It is only through gaining knowledge about teacher implementation and student outcome that the most appropriate decisions can be made regarding future implementation of an innovative program. If it is found that students did not learn and that teachers did not implement the program prop- erly, decisions could be made that differ from decisions made, if information regarding only student outcomes was available. Knowing specific aspects of an innovation which were and were not implemented (and knowing why and/or why not) can help direct attention to where changes need to be made in future implementation. Becoming aware of factors that seem to affect teacher implementation can inform decision makers as to what should be altered in the environment in order to facilitate implementation. The evaluation plan developed for the purpose of this study provided a basic model that educational evaluators might follow in their attempts at evaluating innovative programs. The most opportune time to apply this evaluation design would be before the program materials are placed on the market. Considering the cost involved in purchasing new 141 programs, it would be most helpful to those contemplating implementation of a program to have access to information provided by this evaluation design prior to the initial implementation. Ideally, program developers should apply this evaluation design to a pilot test of their program and provide a summary of this information to schools considering implementation. Although the initial cost of an innovative program might be slightly increased, the expense of purchasing programs not likely to succeed in a particular school setting could be avoided. However, in the event a school invests in an innovative program that does not provide such evaluative information, it is essen- tial that they invest additional time and money to conduct such an evaluation. Based on the results obtained from application of the evaluation plan to the evaluation of the career/leisure program, the plan provides a means for measuring teacher output, measuring student outcomes, relating student out- comes to teacher output, measuring factors and determining the relationship between the factors and teacher output. Application of this plan also illustrated the ability of the plan to evaluate an innovative program when only a small number of teachers (N==5) are implementing. However, because the plan was implemented using a small number of teachers, and because it was applied to i_' I 142 only one program, the feasibility and practicality of the evaluation procedures incorporated into the evaluation plan requires further study. Furthermore, whether or not the changes recommended to improve the plan do so, needs to be determined. APPENDIX A A DESCRIPTION OF THE LEAP CURRICULUM APPENDIX A A DESCRIPTION OF THE LEAP CURRICULUM Target No. l The student will understand that most life experiences can be leisure experiences. They can be active or passive, alone or with others, physical or mental, planned or spontaneous, anticipatory or reflective. Focus Points . To understand that almost any life experience can be a leisure experience--it's a matter of personal choice and attitude. . Students will identify the levels of leisure, expe- rience leisure in each of the positive levels, and plan a balance among their own experiences that lead to personal gratification. e To help students translate their own uses of leisure into levels on the leisure hierarchy. . To identify and share leisure for miscellaneous moments. . To participate in and recognize leisure experiences that are: (a) physical vs. mental, (b) individual vs. social, (c) structured vs. nonstructured, (d) participant vs. audience, and (e) momentary vs. long-term; to identify and develop skills and attitudes necessary to participate in these expe- riences to a degree commensurate with the student's abilities and developmental stage; and to identify the need to develop beginning skills in various experiences which later will become more satisfying. Target No. 4 The student will identify, understand and evaluate leisure resources available in the community, state, and nation; the student will develop an appreciation for various modes that individuals have of utilizing these resources. 143 144 Focus Points . Identify people in the family, classroom, school, and immediate neighborhood who can help with special class and personal interests. . Identify people in the community: (a) involved in-- crafts, sports, construction, foreign cooking, chess, or other; (b) having--collections, films of activities that are difficult to see in person, youth agency pro- grams, camping, a fishing experience; (c) knowing-- special interest clubs, student government, serv1ce to the community, field trips, class skills in building programs, or other; and (d) able to--accompany students on field trips; help build, cook, construct; teach baseball skills, snorkeling, golf, dance, or painting. . Write to state and national bureaus for printed materials about leisure interests: bookkeeping, care of unusual pets, outdoor camping, or other. 0 Locate sites on a map of local and state environments that offer special leisure opportunities: parks, overnight camp sites, ski areas, artificial bob-sled runs, amusement parks, mountain-desert-water areas, or other. 0 Use resources to plan leisure experiences--i.e., locate information and help involve students in planning and developing projects. Target No. 8 The student will understand that leisure as well as work, family, and other social roles can help develop a life plan to ensure personal growth and satisfaction. . Identify work assignments and leisure options during regular school days. Design and use a work/leisure plan to meet individual, class, and teacher requirements. APPENDIX B INSTRUMENTS FOR MEASURING TEACHER OUTPUT: A. TEACHER IMPLEMENTATION INTERVIEW B. TEACHER SELF REPORT TEACHER IMPLEMENTATION INTERVIEW Were you able to teach all of the content (Target Areas and Focus Points) selected for implementation? Yes No If no, what were you not able to implement and why? Did you feel comfortable in teaching all content areas? Yes No If no, for what content areas did you not feel com- fortable? Why? Did all of the career/leisure learner activities you presented relate to your subject matter? Yes NO If no, which activities did not relate to your subject matter? How successful do you feel the learner activities were that you taught? (I.e., did the students react favor- ably to the learner activities and do you feel they learned from the activities?) Explain your response. What learner activities would you change, and why? What activities did you do to involve the community? How successful do you feel these activities were? Explain your response. Did you attempt to involve the students in planning or making decisions about leisure activities? Do you feel you provided the students with oppor- tunities for them to become more aware of their own interests in leisure activities? Yes No. If yes, describe the activities; if no, why not? 145 "' 10. 11. 12. 146 In what sequence did you teach the Focus Points? Were you able to provide the students with opportunities to investigate resources and facts about leisure? Yes No If yes, describe the activities; if no, why not? How much time/week did you implement the program? How many weeks did you spend on implementation? 147 >Hm>wuowwmw we madmsoo pom :OHDMEMOMGA mom 0» 3o: woodman monmeo o>wm pom moonsowou endowed seamen moonsOmmu spammed maanHm>m mceocwm How snowmen unsomwa N.H woeuw>wuom endowed ma powwmaUMm on use soaps mm: Hmoofi>wocw so some: Hmcomumm on» mmsomwa gsueaenm mo coeumusums= mo mace mmnomeo momma mcwammwumm CH 0>Huoommw on use ucmsoao some so: use can: mmsomfio use madmamao seaweed Demon ca name» seamen mwoceeuemxm o>eummmc mDMGHEHHm so uafiea one moonwwummxm changed m>euwmom modmbop on poo: monomwo moms: mcflmmmeumm :fl o>euomumo on use some sons mmsonwo can owsmfioa mo mam>oa uneven semen musmwma uneven neocowuwmxe mudwama mo muowwm> m wow owe: mmsomwo musmfioa cw unmouocw as >33 monomea moflue>fiuom endowed daemon seaweed seamen o.H moeuw>wuo¢ assumed mo scammeomoo sumo mummocoo coeumosom summed; msoom usmums seduced on» ucmEdooa .coeuosuumcw mom poms mowuw>fluo¢ .ummocoo cowumosom mummwoa some ooucwEmHmEfl so» sumo use oumowch EmOh Hmomfim MAmm mmmmu