A DESCRIPTIVE STUDY OF A. SYNTHESIZED OPERATTONAL . _ INSTRUCTIONAL DEVELOPMENT MODEL _ REPORTING lTS EFFECTWENESS. - ‘ EFFICIENCY, AND THE COGNITWE AND AFFECTI'VE TN‘FLUENCE OF THE ‘ r DEVELOPMENTAL PROCESS 0N ACUENT Thesis for the Degree of Ph. D. MICHIGAN STATE UNIVERSITY. SPEUGS THEODORE SIAMAS ' I. 9 7 2 CW 3 1293 10065 7174 ‘ LIBRA R Y Michif‘ran State University “xv-3.. This is to certify that the ' " thesis entitled A DESCRIPTIVE STUDY OF A SYNTHESIZED OPERATIONAL I INSTRUCTIONAL DEVELOPMENT MODEL, REPORTING ITS INSTRUCTIONAL EFFECTIVENESS, EFFICIENCY, AND THE COGNITIVE AND AFFECTIVE INFLUENCE OF THE DEVELOP- MENTAL PROCESS ON A CLIENT presented by Spelios T. Stamas has been accepted towards fulfillment of the requirements for Eh. DI degree in Education Mljor professor 0-7639 ABSTRACT A DESCRIPTIVE STUDY OF A SYNTHESIZED OPERATIONAL INSTRUCTIONAL DEVELOPMENT MODEL, REPORTING ITS EFFECTIVENESS. EFFICIENCY, AND THE COGNITIVE AND AFFECTIVE INFLUENCE OF THE DEVELOPMENTAL PROCESS ON A CLIENT By Spelios Theodore Stamas The problems which have beset American education and threaten to undermine it as a viable institution in society have been identified and extensively reported by the mass media and professional literature. Many of the causes for the problems in education have been associated with: (l) the inefficient manner in which American education has used existing financial and human resources, and (2) the ineffectiveness of instruction as evidenced by the high drop-out rates at all levels in the educational process. Consequently, educators must be responsive to increasing public demands for better education. One alternative which appears to have potential for improving the quality of training programs is the instructional development process. For some years, the same basic systematic process has produced significant results in the American space program, many of the national defense systems, business and indus- trial programs and, to a more limited extent, education. Thus, there are réasons for believing that this systematic problem solving approach may be one of the more useful options available to educators. Spelios Theodore Stamas This study revealed that there have been attempts to adapt the sys- tems approach to problems in education through the development and use of instructional models. These attempts are extensively documented in the model-building literature. However. the models for planning the systematic improvement of instruction appear to be too general to be of maximum operational value to instructional developers. Therefore, this study proposed to improve the process of instruc- tional development by generating an operational synthesized model from a review of related research. The study also examined the effectiveness and efficiency of the model's process within the context of one training program. The setting for this study was the Breathalyzer Operator Training Program (BOTP), a_monthly training program for Michigan law enforcement officials. The population included eight different groups. totaling 222, of police officers who attended BOTP schools from October. l97l, through May, 1972. Three distinct research objectives were examined in this study. The first related to synthesizing an operational instructional development model from the related literature and reporting the experiences of using the model with a client. The Provus Pittsburg Discrepancy Evalu- ation Model (Nelson. l970) was used to formatively evaluate consistencies and discrepancies between the model and the actual process of instruci tional development with the BOTP. The second objective focused on statistically measuring the effec- tiveness of the instructional development process in improving student learning over a series of eight monthly training programs. Five dependent Spelios Theodore Stamas variables were identified from the written certification examination and the laboratory performance checklist criterion measures. The two varia- bles related to the written certification examination were subjected to an analysis of variance to determine if significant improvements in stu- dent performance were discernible as a result of the instructional development. Paired comparisons using least significant differences be- tween various combinations of BOTP schools (or months) were then computed to identify significant differences. The three laboratory variables were subjected to 3:3ggt statistical comparisons, between schools, of the percentage of satisfactory performance responses to determine if signifi- cant student improvement (over the October, 197l, BOTP school control group) was evident in the schools during instructional development from November, 1971 through May, l972. The final objective sought, first, to compare the attitude of the client toward the instructional development process with nationwide norms; secondly, to develop and administer instruments for measuring the clients cognitive growth related to the process; and lastly, to report the in- structional efficiency of the process using three criteria. The client's raw score on an attitude instrument was compared to the grand mean score of nineteen national Instructional Development Institutes. Cognitive growth was measured by asking the client to numerically rate both his entry and exit cognitive proficiency on twenty-eight items representing the steps of the synthesized model. Entry and exit mean scores were then compared using the t-test. Spelios Theodore Stamas Several discrepancies were reported between the synthesized model and the process used with the BOTP. Significant differences in student performance at levels ranging from .05 to .Ol were discovered for the three laboratory variables designed to measure instructional effective- ness. However, no significant differences were reported on the variables related to the written certification examination. The experimenter found that the instructional development did not result in instructional efficiency on any of the three criteria used in the study. The client's score of 214 on an attitude rating scale was slightly less than two 5.0. above the grand mean score of 198.7 reported by nine- teen Instructional Development Institutes. The client's mean score on an instrument measuring his opinions re- garding instructional development showed a 3:333; significance at the .Ol level when comparing his entry (October, 1971) and exit (June, l972) cognitive proficiency levels. Responses to an open-ended questionnaire administered after the study was completed, revealed that: (l) the client continued to have a posi- tive attitude toward instructional develOpment; and (2) the client be- lieved that one of the more significant results of the process was the change within himself--an increased sensitivity toward the value of making program changes in instruction and an increased sensitivity toward the quality of his own instruction. The following conclusions are made from the study: (1) The synthe- sized model was effective in improving instruction, in that statistically significant differences favoring the programs under instructional Spelios Theodore Stamas development were obtained on three of the four dependent measures in the replications of the training programs for November, 1971, through May, l972; (2) Instructional effectiveness, as defined in this study, will not necessarily result in instructional efficiency for programs which have not reached the stage of using self-instructional materials, in that a decrease in instructional efficiency was reported on all three of the criterion measures; (3) A positive attitude by the client toward the process can be maintained by providing statistical evidence of the effec- tiveness of instructional development, in that the client viewed the statistical evidence useful in justifying to his employer his continued involvement with instructional development; (4) The synthesized model was, for the most part, a good representation of the instructional development process since few model discrepancies were reported; (5) Attitude toward instructional development is likely to be more positive for those using the process in the field than for those individuals (or groups) exposed to the process in more formal instructional settings; (6) There is a positive relationship between the effective diffusion of the instructional development process to a client and the client's active involvement with the process, in that the study showed significant growth in the client's cognitive proficiency despite the fact that the synthesized model was not disclosed to him during the study; and (7) Effective instructional development can change people, in that the client stated on an open-ended questionnaire that one of the most significant results of the process was the change within himself. Spelios Theodore Stamas The synthesized model provides an operational framework within which instructional developers can consult with instructors regarding the systematic improvement of training instruction. Whether the model can be generalized to other types of instructional systems is a question yet to be answered. However, the instruments used to measure the client's cognitive pro— ficiency and his attitude level relative to the process need further refinement, including tests of reliability and validity. The experimenter concluded that several refinements to the synthe- sized model should be considered by those who may wish to use the model or replicate this study. For example, refinements to the synthesized model need to: (l) include resequencing several steps of the process; (2) contain heuris- tics, or rules of thumb; (3) include prerequisite guidelines which specify to the client what commitment(s) he will have to make if the instructional development is to achieve maximum success; (4) explicate how to perform the functions of the model; and (5) specify criteria for measuring how adequately each step of the model was performed during instructional development. REFERENCE Nelson, Frank 6. ”Models for Evaluation." Monmouth, Oregon. Teaching Research Division of the Oregon State System of Higher Education. 1970. A DESCRIPTIVE STUDY OF A SYNTHESIZED OPERATIONAL INSTRUCTIONAL DEVELOPMENT MCDEL, REPORTING ITS EFFECTIVENESS, EFFICIENCY, AND THE COGNITIVE AND AFFECTIVE INFLUENCE OF THE DEVELOPMENTAL PROCESS ON A CLIENT By Spelios Theodore Stamas A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY College of Education 1972 Copyright by' SPELIOS THEODORE STAMAS 1972 DEDICATION To My Father iii ACKNOWLEDGMENTS The writer wishes to express his appreciation to the many persons who have contributed to the design, development, and execution of this thesis. Particular thanks are expressed to Dr. Kent L. Gustafson, chairman of the guidance committee, for his counsel and special interest in my program and for spurring me on to an intensive study of a practical problem in instructional development. Appreciation is also expressed to Drs. Norman T. Bell and Elwood Miller for their counsel, and to Dr. Allan J. Abedor for his interest, thoughtfulness, and encouragement as an llth hour substitute on the guidance committee. Special thanks go to Dr. Paul N. F. Witt for his interest in my pro- gram, Mr. Jerry Stemler without whose cooperation this thesis would not have been possible, and Dr. Lawrence Lezotte for his key suggestions in the design of the study. Gratitude is eXpressed to Dr. Daniel Jacobson for his valued friend- ship and kindness in keeping an "open-door" at times when a sympathetic ear was most appreciated. Deepest appreciation goes to my wife Connie, my son Ted, and my daughters Gayle and Beth for their patience and understanding throughout the development of this study. To them, I offer my gratitude for the love and encouragement they have provided. iv CHAPTER I. II. III. TABLE OF CONTENTS THE PROBLEM . ........... . .......... Purpose of the Study . . . . . . . . . . . . . . . Procedures ......... . . ............ The Need for the Study . . . . ............. Uses and Importance of Models . . . . . . . . Precedence for the Study ..... . . . . . . Potential Contributions of the Present Study. . Generalizability of the Study . . . . . . . . Hypothesis . . . . . . . . . . . . . . . . . . . : . Additional Research Questions. . . . . . . . . . Assumptions. . . . . . . . . . . . . . . . . . . Definition of Terms. . . . . . . . . . . . . . . Organization of the Thesis . . . .......... RELATED RESEARCH. . . . . . . . . . . . . . . . . . . Development of Synthesized Operational Model . . Review of Related Research. . . . . . . . . . The Synthesized Operational Model . . . . . . . I : I Entry Step: Discussion .......... . . . . Discussion of Model's Development . . . . . . . . . Summary. . . . . . . ..... . ............ DESIGN OF THE STUDY . . . . ........... . . Methods and Procedures ......... . ....... Research Strategy. . . . . . . . . . . . . ...... . Definition of the Population ..... . . . . . . . Collection of Data . . . ..... . ...... . Design Methodology . . . . . . . . . . . . . . . Baseline Data . . . . . . . . . . . . . . . . Model Validation. . . . . . . . . . . . . . : . I The Provus Pittsburgh Discrepancy Model . . .'. . . . Why Was the Provus Model Selected?. . . . . . Instructional Effectiveness . . . . . . . . . Written Certification Examination. . . . . Laboratory Checklist Examination . . . . . Additional Research Collection Procedures. . . . Instructional Efficiency. . . . . . . . . . . Client's Attitude Toward Process. . . . . . . Client's Opinions Regarding Instructional Developmen O O 0 (0-00... TABLE OF CONTENTS-~Continued CHAPTER Page Client' 5 Post Attitude Questionnaire. . . . . . . . . 52 Statistical Hypothesis ............... 52 Statistical Treatment of the Hypothesis . . . . . . . 53 Additional Research Questions . . . . . ....... 53 The Limitations of the Study . ..... . . . . . . . . 54 Summary. . . . . . . . . . . . . . . . . . . . . . . . . 56 IV. FINDINGS AND CONCLUSIONS ........ . . . . . . . . . . 57 Null Hypothesis 1. . . . . . . . . . . ......... 57 Written Certification Examination . . . . . . . . . . 57 Laboratory Variables. . . . . . . . . . . . . . . . . 64 Summary Hypothesis Matrix . . . . . . ...... . . 72 Model Discrepancies and Consistencies .......... 73 Summary Table . . . . . . . . ....... 85 Summary Conclusions Related to Question 1 ..... . 85 Instructional Efficiency . . . . . . . . . . . . . . . 87 Client's Post Attitude Questionnaire . . . . . . . . . . 9O Subjective Data . . . . . . . . . . . . . . . . . 95 Client's Attitude Toward ID. . ............ 96 Client's Entry and Exit Cognitive Proficiency Levels of ID Process ........... . . . . . . . . . . . 98 V. DISCUSSION. . . . . . . .......... . . . . . . . . lOO Overview . . . . . . . . . . . . . . . . . . . . . . lOO Summary of the Development and Application of the Synthesized Model . . . . . . . . . . ..... lOO Implications of the Study to Instructional DeveTOpment . lOl Heuristics . . . . . . . . . . . . . . . . . . . . . . 103 Recommendations for Further Research . . . . . . . . . . 106 Research Leading to Refinements of the Model. . . . . lO7 Determining the Generalizability of the Synthesized ‘ Made] 0 O O O O O O Q C O O O O O O O O O O I O O O 108 APPENDICES A. REVIEW OF RELATED RESEARCH. . . . . . . . . . . . . . . . . 109 B. PRINCIPAL AND NON-PRINCIPAL TEST ITEMS IDENTIFICATION FOR WRITTEN CERTIFICATION EXAMINATION . . . . . . . . . . . . . 187 C. LABORATORY CHECKLIST. . . . . . . . . . . . . . . . . . . . 189 D. ATTITUDE TOWARD INSTRUCTIONAL DEVELOPMENT RATING SCALE. . . 19l vi TABLE OF CONTENTS--Continued APPENDICES Page E. CLIENT' S OPINION TOWARD INSTRUCTIONAL DEVELOPMENT RATING SCALE ....... . . . . . . . . . . ....... . F. CLIENT'S POST-ATTITUDE QUESTIONNAIRE . . . . . . . . . . 201 BIBLIOGRAPHY. . . . . ...... . ......... . . . . . 203 vii TABLE II-l. III-l. III-2. III-3. IV-l. IV-2. IV-3. IV-4o IV-5. IV-6. IV-7. IV-8. IV-9. LIST OF TABLES Composite Matrix of Model Commonalities. . . . . . . . . Attendance Figures for Each BOTP Replication . . . . . . Framework for Reporting Laboratory Variables . . . . . . Framework for Reporting t-test Analysis of Entry and Exit Behavior of the Client. . . . . . . . . . . . . . . Analysis of Variance for the Repeated Measures of the Principal (P1) and Non-principal (NPT) Variables on the Written Certification Examinations . . . . . . . . . . . Analysis of Principal (P1) and Non-principal (NP1) variabIeS. O O O O C O O O O O O C O C C O O O O O O C O Mean Scores on Repeated Measures for the Principal (P1) and Non-principal (NP1) Variables. . . . . . . . . . . . Analysis of Variance for Paired Mean Scores. . . . . . . Least Significant Differences Between Paired Means on the Repeated Measures of the Principal (P1) and Non- principal (NP1) Variables. . . . . . . . . . . . . . . . Matrix of Differences (D) and Corresponding t-test for the Difference Matrix on the Simulator Preparation Labor- atory Variable . . . . . . . . . . . . . . . . . . . . . Matrix of Differences (D) and Corresponding t-test for the Difference Matrix on the Breathalyzer Operation Laboratory variabIe. O O O O O O O O O C C O C O O O O 0 Matrix of Differences (D) and Corresponding t-test for the Difference Matrix on the Checklist Item TBS—[Shore- tory Variable. . . . . . . . . . . . . . . . . . . . . . Summary Matrix of Conclusions on the Null Hypothesis of the StUdyO O O O O O O O O O I O O O O O O O O O O O I 0 viii Page 24 33 48 51 58 60 61 62 63 65 67 68 72 LIST OF TABLES--Continued TABLE IV-IO. Iv-llo Page Summary Matrix Reporting Discrepancies (D) and Incom—i plate (I) Execution of Step Functions in the Synthe- sized Model During the BOTP Instructional Development.. 86 Corresponding t-test Analysis of Entry and Exit Scores on Twenty-eught OBservations on the Client's Opinion Regarding_InstructiDnal Development IEStrumenf} . .1. . 98 ix FIGURE 11“. III-1. III-2. III-3. IV-1. IV-2. LIST OF FIGURES A Synthesized Operational Instructional Development Made] 0 O O O O O O O O 0 O O O O O O O O O O O O O O O Breathalyzer Operator Training Course Schedule . . . . Provus Discrepancy Model Process Flowchart . . . . . . Model Selection Criteria . . . . . . . . . . . . . . . Instructional Development Inputs Into BOTP Schools . . Mean and Raw Score Comparisons Between the Client and the Instructional Development Institutes on the 53317 tudeToward Instructional Development Instrument . . . The Barson Model . . . . . . . . . . . . . . . . . . . Hamreus Systems Development Model. . . . . . . . . . . The Kaufman Model. . . . . . . . . . . . . . . . . . . Kaufman Model Systems Analysis and Synthesis . . . . . Kaufman Model Functional and Systems Analysis. . . . . Childs Model . . . . . . . . . .......... . . The Banathy Model. . ........... . . . .'. . The Stowe-Indiana Model. . . ..... . . . . . . . . The Stowe-Indiana Model. . . . . . . . . . . . . . . . IDI Mada] O O O 0 O O O O O 0 O I O o O 0 O O O O O O O Briggs Design of Instructional Model . . . . . . . . . An ID Model, K. Gustafson, Michigan State University, 197] O O O O O I O O O O O O O O O O O O O C C O 0 C 0 A Systematic Approach to Instruction (Gerlach and Ely) Page 21 35 39 41 71 97 112 114 116 117 119 121 124 126 127 131 132 137 140 LIST OF FIGURES--Continued FIGURE A-14. A-15. A-16. A'17o A-18. A-19. A-ZO. A'Z] a. A-21b. A-21c. A-22. A-23. A-24. A-25. Page The Douglas Instructional Development Model. . . . . . . 145 Instructional Design Model (Kemp). . . . . . . . . . . . 149 Procedures Used in the Development of Human Components of Systems (Gagné) . . . . . . . . . . . . . . . . . . . 150 A Teaching Model (DeCecco) . . . . . . . . . . . . . . . 154 A General Model of Instruction (Kibler, Barker and Mi]eS)oooooooo00000000000000coo157 Stages in the Instructional System Design Process of the CER Model (Alexander and Yelon). . . . . . . . . . . . . 160 Tracey, Flynn and Legere Model . . . . . . . . . . . . . 163 Civil Defense Flowchart Model at First Level of Detail (Gordon) . . . . . . . . . . . . . . . . . . . . . . . . 169 Develop Training Programs Subsystem Model (Gordon) . . . 170 Operate Training Programs Model Subsystem (Gordon) . . . 171 The BUIC II Air Defense System Model (Carter). . . . . . 172 An Empirical Instructional Model (Popham). . . . . . . . 178 "Mini" Model (Abedor). . . . . . . . . . . . . . . . . . 182 "Maxi" Model (Abedor). . ...... . . . . . . . . . . 184 xi CHAPTER I THE PROBLEM Purpose of the Study The purpose of this study is to provide a model which can improve the instructional development process and thus the effectiveness and efficiency of instruction. The study proposes to improve the process of instructional development by generating an operational synthesized model from a review of related research. It also proposes to examine the effectiveness and efficiency Of the model's process within the con- text Df one training program. Procedures To achieve this purpose, the study proposes to: (l) synthesize an operational instructional development model from model commonalities apparent in the related research, from professional contacts with instruc— tional developers, and from selected literature dealing with the indi- vidual components of the instructional development process; (2) apply the synthesized model to the Breathalyzer Operators Training Program (BOTP); (3) state consistencies and discrepancies, if any, between what the model suggests, and what actually occurred; (4) suggest the steps of the syn- thesized model which, on the basis of the discrepancies, would appear to need revision; (5) present statistical and research data designed to measure the degree to which the instructional development with the BOTP resulted in: (a) improved student learning over a series of training sessions or schools and (b) improved instructional efficiency in terms of instructional time, instructional costs, or instructional man-hours; and (6) present evidence which will support or deny positive affective behavior toward the instructional development process as well as cognitive growth in understanding and use of the process by the Director of Train- ing for the BOTP (client). The Need for the Study It is generally recognized that American education is beset with serious problems, some of which threaten to undermine it as a viable institution in society. These problems have been identified and exten- sively reported and re-reported by the mass medias and the professional literature to the American public. For this reason, no purpose would be served by any literature review related to the: (1) financial crisis in public and private education; (2) problems of equalizing educational opportunities for all segments of American society; or (3) need for greater instructional relevance and student instructional alternatives in the curricula of our schools. What is important, however, is that many of the causes for the prob— lems in education have been associated with: (a) the inefficient manner in which American education has used existing financial and human resources in the system, and (b) the ineffectiveness of instruction as evidenced by high drop-out rates at all levels in the educational process. Consequently, educators must be responsive to the public demands and explore or examine new alternatives for improving the effectiveness and efficiency of instructional design, development and implementation. One alternative which appears to have potential for improving the quality of education is the instructional development (ID) process. It is generally agreed by those who call themselves instructional de-‘ velopers, that ID is a systematic process which at a minimum uses learn- ing theory and communication research in the design, development, and implementation of programs which are more effective and efficient in their teaching/learning activities. For some years, this systematic process has produced significant and obvious results in: (l) the American space program as reflected by the success of the Apollo and other space efforts; (2) many of the national defense systems which require systematic training of military personnel as well as a systematic design, development and implementation of early warning attack and weapon systems; (3) business and industrial programs whose success correlates highly with efficient and effective training of sales personnel; and (4) American public and private educa- tion (on a more limited basis) for systematic retraining of teachers, e.g., NSMI (1970); DeCecco (1968); Alexander and Yelon (1969); Gerlach and Ely (1971); Kemp (1971); and Douglas (1971). Thus, there are reasons for believing that an instructional develop- ment process which uses a problem-solving approach similar to those used by government and industry, may befone of the more useful options avail- able to training managers and educators in their quest for quality instructional programs. Interest by these educators in the potentials of ID is evidenced by the increasing use of models which have been developed and reported in the professional and research literature during the past decade (Appendix A). Moreover, instructional models have been used by educators in the past several years to cope with instructional problems. Barson (1965) used his model with four major American universities as a model for systematic development of college-level courses. Hamreus (1968) developed a model which was condensed by the National Special Media Institutes (1970) for use during Instructional Development Institutes designed for teachers, administrators, policy makers, and specialists. Alexander and Yelon (1969) devised their model as a common experiential referent while working with faculty on instructional design. Douglas (1971) formulated his model as an operational plan for instructional development with staff at Burlington County College in Pemberton, New Jersey. Although there is all this interest in instructional models, all these model-builders and users indicate a need for more study and model refinement. As this study will reveal, there have been many attempts to adapt systematic processes to instructional problems by developing models. These attempts are well documented in the model-building literature reported since the 1960's. However, these models for planning the sys- tematic improvement of instruction, or the systematic identification of problems within an educational system, appear to be too general to be of maximum operational value to instructional developers. If the potential of models is assumed, it is important to improve the process of instruc- tional development by advancing the knowledge in model-building theory. Obviously, this study cannot reasonably solve all the instructional problems which have beset training programs in America over the years. Regardless, the results of this study can contribute to the development of better strategies for designing, developing and implementing instruc- tional programs. Therefore, the study is limited to reporting the experiences of the experimenter in using a synthesized instructional development model with a training program, the BOTP for the State of Michigan. One question which needs to be discussed at this point is: Why use instructional systems which involve instructional models? It is there- fore useful to explicate the specific uses and importance of models. Uses and Importance Of Models In _attempting to place the history of model- -building in perspective, Karl Deutsch (1948- 49) states that "men have tended to order their thoughts in terms of pictorial models since the beginning of organized thought" (p. 387). He elaborates: The model itself was drawn from something in their immediate experience, available from their technology, and acceptable to their society. Once adopted, it served, more or less efficiently, to order and correlate the experiences which men had, and the habits they learned, and perhaps to suggest a selection of new guesses and behavior patterns for new or unfamiliar situations (p.387). Deutsch maintains that pictorial models are most useful as a way of order- irm experiences so that more intelligent decisions can be made on prob- able solutions to problems. After briefly tracing the historical precedent for models in society, Deutsch concludes that "later models were drawn by men from work of their hands, that is from processes and things which they themselves could bring into existence, put together or to take to pieces, and which they therefore could analyze and elaborate more adequately in their parts and interrelationships" (p. 387). Alexander and Yelon (1969) see the value of a model as a communica- ”—— ”N m tions tool for providing a Common Experiential Referent (CER) in instruc- tional systems design. They theorize that: When people work together designing systems, they invariably encounter communications difficulties. This is usually because, coming from different backgrounds and having different ways of approaching a task, they tend to view problems differently. Often each employs a different vocabulary, or technical language, derived from his particular area of training or competence, which also impedes communication of ideas (pp. 44~46). In summary, the CER flowchart model is the stimulus and referent from which the instructional developers can move toward a consensus on the best strategies and procedures to follow in the design, development, and evaluation of programs. This common experiential approach presupposes that changes in the referent model may result from the information input provided by individual members of the design team. Generally speaking, the theoretical assumption underlying this approach is that model consensus by the instructional developers will result in greater commitment to the process as subsequent developmental activities evolve. Silverman (1967) recommends the use of models rather than theories of temflfing at this stage in the development of educational technology because, as he contends, "models offer greater flexibility in dealing with what he refers to as field or dynamic forces operating within the environ- ment" (p. 5). He explains further that the "key property of a field is the dynamic one; every part depends upon every other part and parts can- not be studied in isolation from the whole" (p. 5). Silverman emphasizes this thesis by observing that: The best way to proceed in developing a theory of teaching is to begin with what is known about learning in the laboratory and in the classroom by adopting a model derived from a theory of learning and/or from systematic app*oaches to the study of learning in the laboratory. . . . The relationship between the laboratory and the classroom may be improved by the use of models. By model I mean mode of representation. In this sense a model may be a replica. . . . A model tolerates exceptions, but a theory does not easily do so. . . . Models can be useful and yet they demand less commitment to them than do theories. They can be discarded and replaced if shown not to be useful (pp. 4-5). In discussing the values and limitations of using a diagrammatic model as a means of picturing the communications process, Barnlund (no date) noted that: When social scientists try to isolate and order all the elements of a complex event-~that is, when they approach such systems ana- 1ytically--the results are almost unmanageable (p. 86). Barnlund continues by saying that "one advantage of a model then, is the ease with which it handles a multitude of variables and relates their effects upon each other in highly complicated ways--thus preserving the integrity Of the events under study" (p. 86). Further, Barnlund believes the designer of a model is forced to identify variables and relate them with a precision that is difficult for the writer to achieve because of the stylistic demands of effective writing (p. 86). Finally, he maintains that "a diagram or formula can portray at a single glance, and With great transparency, the assumptions and properties of a new theoretical position, thus stimulating the study of alternative approaches“ (pp. 86-87). OfflgghtdndflE1y_11971) view models as a guideline or road map which should be used as a checklist in planning for teaching (p. 12). They explain that a checklist "shows the major components of the total teach- ing/learning system, even though it does not portray the fine details of each component" (p. 12). To Gerlach and Ely, one value of a model is its ability "to visualize, in simplified fashion, something not easily Observed" (p. 12). They conclude that "when a model is used without constraints it can serve as a powerful aid to a teacher in helping him organize his instructional planning more carefully, thereby minimizing the possibility of overlooking essential components" (p. 12). Merrill (no date) describes model-building as a research approach. It is his contention that "a careful analysis is made of the various components of the instructional system, and classification schemes are hypothesized" (p. 6). Using this research approach, often referred to as the systems approach, protects against what Randall (no date) calls "partial solutions to educational problems“ (p. l). Randall insists also that instructional design which is systematic and employs such tech- niques as model-building can be a useful tool for maintaining proper perspectives requisite to productive innovation and change" (p. l). Kaufman (1968) reaffirmed the contention Of Merrill, Randall, and Gerlach and Ely when he noted that the systems approach as represented in .IUSEEUCtI9"a] development modéls "would be an effective and efficient tool to assure that the complex interactions will be properly considered" and that since the educational world is complex, it would seem that a formal problem solving model may serve it well (p. 419). Kaufman's concern for using a formal problem solving model for ordering complex problems to assure proper interaction between all the variables or components related to that problem is the basic premise. of systematic planning.“ Models have also been used to advantage in developing specifica- tions for instructional simulations. For example, Twelker (1969) claims that "A simulation will only be as good as the model on which it is based. . . ." (PP. 5-7). In carrying this further, Abt (1970) maintains that the physical scientist or engineer who experiments in a simulated reality with reduced-scale model of devices or processes uses theory about how things are related to each other to define the model relation- ships, and then experiments with the model to test various solutions in alternative possible environments (p. 11)° Thus, in summary, models have the potential of: (l) ordering experiences so that more intelligent decisions can be made on probable solutions to problems; (2) serving as a common experiential referent from which individual members of a design team can move toward process consensus and commitment; (3) portraying the teaching process in a flexible manner since they demand less commitment and tolerate excep- tions; (4) representing a multitude of variables while preserving the integrity of the events under study; (5) portraying, at a single glance, the assumptions and properties of a new theoretical position; (6) stimu- lating the study of alternate approaches; (7) protecting against partial 10 solutions to educational problems; (8) serving as a checklist in planning for teaching; and (9) simulating reality in reduced—scale to test vari— ous solutions in alternatives possible environments. Precedence for the Study A precedent for the present study was established by Barson (1965) when he developed a model for analyzing instruction and implementing newer media of communications for improving instruction (p. 1). Two years later, he tested the model as part of a USOE study which enlisted the cooperation of four universities (p. 1). Although Barson tested the effectiveness Of the model in analyzing instruction and implementing newer media of communications, the experimenter could find no evidence in this literature review to indicate that the model was revised after the study. Further, the same pattern related to model refinements is evident in the related literature. Few instructional developers have attempted to apply a model to developmental problems for the purpose of reporting consistencies and discrepancies in the model. Of significance also is the fact that few, if any, instructional developers have reported strate- gies for measuring the effectiveness of development resulting from the application of a model. Potential Contributions of the Present Study This stddy is designed to contribute to the literature in the field of Instructional Development by building on the work of Barson and others. 11 first, a literature review was conducted in order to synthesize an instructional development model from the model commonalities observable in the review. The main purpose was to generate a more operational instructional development model than is located in the literature. Secondly, the study was designed to report procedures for measur- ing or assessing the extent to which the instructional development process improved student learning and instructional efficiency. Recent demands for accountability in the learning/teaching process make it imperative that instructional developers, as well as all educators, systematically design and develop procedures for evaluating the effectiveness of their efforts during program development. The need to be accountable was clearly voiced by President Richard M. Nixon in his 1970 Education Message when he observed that as a result of the obvious shortcomings in the quality of American education: . . . we derive another new concept: accountability. School administrators and school teachers alike are responsible for their performance, and it in their best interest as well as the interest oz)their pupils that they be held accountable (Lessinger, 1971, p. Stated concisely, educators need to focus on effective and efficient learning, not the retention of children in school. The ultimate goals should be (1) zero-rejects in learning, (2) greater relevance, (3) im- proved human dignity and selfmimage by dealing not only with individual differences but also individual similarities, and (4) a guarantee that every child shall learn (Lessinger, 1971, p. 14). Lessinger reaffirms that educators have made few attempts to measure the results of student learning or performance as a test of professional competency when he noted 12 that we have now reached the point at which “results are what count, not promises and lamentations" (p. 13). As proof of our failure he reported that: Today, about one of every four American children drops out of school somewhere between fifth grade and high school graduation. In 1965, one of every four lB-year-old males failed the mental test for induction into the armed forces. And hundreds of thousands of parents, particularly of minority children, reacting to this infor- mation, have decided that their children are not stupid--that either some educators are incompetent, or inadequate (Lessinger, 1971, p. 13). Deterline (1971) takes the same basic position on accountability. However, he inserts one additional element into any discussion of the topic when he explodes the myth that teaching means the presenting of information by a competent subject matter expert who, through testing, identifies student deficiencies and blames them on the student. Instead, Deterline insists that teachers "ought to be held accountable for the results of instruction, and rewarded or not rewarded depending on the results" (p. 16). In relating the accountability problem more specifically to higher education, Gage (1971) writes: The current dissatisfaction with higher education as reflected by numerous articles in the newspapers, periodicals, etc. and by reluctant findings by state legislation, suggests that this would be a time when the public is asking that the (Blanket) mandate they have given educators either be returned, returned in part, or at least reexamined. In a sense, the public is asking us in higher education to at least provide them with some evidence that we are taking care of the blanket. That is, this may be a chance to demonstrate to both ourselves and the public that we are as strong as they are . . . the public is going to insist that some evidence be provided that the American college is behaving responsibly (pp. 1-2 . If Gage is correct, the implications of accountability are clear. Instructional developers must examine ways to better prove their 13 accountability through the reporting of evaluation data, including statis- tical and research evidence, in measuring the effectiveness and efficiency of the process. Thirdlx. the study is designed to use the Provus Pittsburg Discrepancy Model as a formative evaluation instrument for: (1) setting the instruc- tional development criteria or standards; (2) establishing consistencies and discrepancies, if any, between program operations and standards; (3) using the discrepancy information to isolate program and instructional development strengths and weaknesses. Fourthly, the study is designed to build on the work of Barson and other instructional developers by developing instruments and strategies for measuring a subject's attitude and cognitive comprehension level re— lated to the instructional development process. Generalizabilitypof the Study The study is generalizable to other Breathalyzer Operators Training Programs (BOTP) as well as to instructional programs which possess the same attributes or characteristics of the BOTP. These attributes are: (l) a five-day, forty-hour structured training program; (2) jointly sponsored training or instructional program; (3) relatively homogeneous (police officers) learner population; (4) lecture-discussion and labora- tory instructional methodologies; (5) multi-membered instructional staff; (6) in-house training facilities; (7) learners motivated by the incentive of job promotion; (8) learners motivated by the effects of non-learning; (9)cHsciplined or controlled learning environments, and (10) general learner population of high-school graduates. 14 Concurrently, the statistical and research strategies used to measure the effectiveness of the instructional development and the forma- tive evaluation design used during the program development hold implica- tions for instructional developers in general. Hypothesis The following null hypothesis is to be statistically tested in the study: H1 There is no difference in student performance on: (1) the written certification examinations, and/or (2) the laboratory checklist sheets, between the baseline (October, 1971, BOTP) mean scores and the mean scores of the BOTP schools from November, 1971, through May, 1972. Additional Research Questions Five additional research questions are to be examined within the context of the experimenter's field experience with the BOTP. Descriptive and Observational data will be used to answer the following research questions: 1. What differences, if any, between the synthesized model and the actual instructional development process used with the BOTP will be revealed by the Provus Pittsburg Discrepancy Model? 2. To what extent did the instructional development with the BOTP result in changed instructional efficiency as measured in terms of: (a) an increase or decrease in actual time of the BOTP 15 through the addition or delection of instructional objectives and/or teaching/learning activities; (6) an increase or decrease in program costs, and (c) an increase or decrease in instructor man-hours? 3. What changes, if any, occur in the client's attitude toward instructional development as revealed in a structured, open-ended questionnaire administered to the client approximately one month after the termination of the experimenter's field experience with the BOTP? 4. What differences, if any, will there be between the raw score of the client and the grand mean score of participants in nineteen Instructional Development Institutes (1015) on the Attitude Toward Ipstrpctiopal Development rating scale instrument? 5. What mean score differences, if any, will there be between the client's entry (October, 1971) and exit (June, 1972) cognitive proficiency level of the ID process as measured by the Client's Opinion's Regprding_lnstructional Development instrument? Assumptions 1. It was assumed that the BOTP needed improvement as evidenced by the initiative taken by the Director of Training for the BOTP (the client) in seeking help from instructional developers in improving the training program. 2. The assumption was also made that using a systematic process (instructional development) during the designing, development and evalua- tion of instruCtion would be better than not using a systematic process. 16 The potentials of using a systematic approach to problem solving can be assumed by the successes which have resulted in: (a) the American space program; (b) many of the national defense systems; and (c) business and industrial programs. 3. The assumption was made that the primary sources of information relative to the instructional development process are ID models reported in the literature. The assumption is supported by the fact that there have been many instructional development models reported during the past decade. 4. The assumption was made that an instructional development model contains a minimum of four components: (a) specification of behavioral objectives; (b) information flow between and among the steps (feedback); (c) flowchart or combination flowchart/narrative description of process; and (d) a recycling process which permits a continuous re—examination of whatever is developed to determine its instructional effectiveness, efficiency, or relevance. 5. The assumption was made that the instructional models reported in the literature were of limited value at the operational level of the instructional development process. Therefore, it was assumed by the' experimenter that a more Operational ID model needed to be synthesized from the review. 6. The assumption was made that the synthesized model would have at least as much potential for improving instructional effectiveness and/or instructional efficiency as any of the 10 models reported in the litera- ture. This assumption is based on the fact that the synthesized model is 17 a composite of process steps common to ID models, most of which are reported to be highly functional by their respective authors. 7. It was assumed by the experimenter that one important element in instructional development is changing people's self-concept. Therefore, the study assumed that the effectiveness of instructional development must, in part, be measured in terms of increased positive attitude by the client as a result of the instructional development project. ggefinition of Terms Instrpctional Development--a systematic process which uses learning theory and communication research in the design of effective and efficient teaching/learning activities. The process contains a minimum of four components: (a) Specification of behavioral objectives; (b) in- formation flow between and among the steps (feedback), (c) flowchart or combination flowchart narrative description of process, and (d) a recycling process which permits a continuous reexamination of whatever is developed to determine its instructional effectiveness, efficiency, and/or relevance. Instructional Developer--used in the singular, it refers to the ex- perimenter; used in the plural, it refers to those individuals who com- prise the team of developers, e.g., the client and the experimenter. Under more ideal circumstances a team of instructional developers might include a client. an instructional media specialist, a learning psycholo- gist and an evaluation specialist. Bglfle-Breathalyzer Operator Training Program for the State of Michigan; monthly (except July and August) training schools for Michigan 18 police Officers. Experimenter (E)-—the author of the study. ' Subject(s)-—police officers who had attended any one of eight BOTP schools for training as Breathalyzer Operators. Synthesized Instructional Development Model--the model designed by the experimenter from the commonalities reported in the related litera- ture; the model used by the experimenter, or instructional developer, and the instructional developers with the BOTP. Operational Instructional Development.Model--a flowchart, or analog, depicting the process steps in instructional development which accompany a written description of the process. The Breathalyzer Instrument--a scientifically designed instrument used in the State of Michigan by certified police officers to measure the blood alcohol content of automobile drivers arrested for driving under the influence Of alcohol. Model Discrepancy--refers to: (l) deviations between the steps included in the synthesized model and the actual procedural steps used by the experimenter with the BOTP; (2) steps of the synthesized model which were not adequately accommodated during the ID with the BOTP; or (3) steps in the model which the experimenter believed were not in correct sequence. Organization of the Thesis Chapter I outlines the purpose and need for the study within the con- text of instructional development, presents a definition of relevant terms, lists statistical hypothesesand research questions and delineates several assumptions related to the study. 19 In Chapter II, the.development of the synthesized operational model is explained. The synthesized model itself is also included in the chapter along with a brief narrative of model entry points. In Chapter III, the design of the study is presented. The nature Of the sample is specified, as well as the analysis of the devices to be used for measuring the research and statistical hypotheses. In Chapter IV, the findings from the data are analyzed, and the conclusions for the hypothesis stated in Chapter I are discussed. Finally, in Chapter V, the implications of the study are discussed, recommendations for further study provided, and heuristics, or "rules of thumb" emerging from the study presented. CHAPTER II RELATED RESEARCH Development of Synthesized Operational Model Review of Related Research The major input for developing the synthesized operational ID model (Figure II-l) was an extensive review of nineteen instructional develop- ment models developed by instructional technologists, curriculum developers, and training managers. To qualify as an 10 model for pur- poses of this study, a model had to contain, as a minimum, the following four components: (a) Specification of behavioral objectives; (b) infor- mation flow between and among the steps (feedback); (c) flowchart or combination flowchart narrative description of the process; and (d) a recycling process which permits reexamination of whatever is developed to determine its effectiveness, efficiency, or relevance. The review also includes a discussion of four non-qualifying but related instructional models. Although these models do not qualify as ID models under the criteria specified in this study, the experimenter reported those models because they contained selected elements of the ID process and/or were deemed to be reputable models. 0f the four models the Abedor Model on prototype development evaluation had particular sig- nificance because it is the only model that represents the prototype development and evaluation process at the operational level. 20 21 um=c_ucou P-~H mtzmwe as. 34‘ s. 8.: g Plugag 43%. 3‘: GIN-g ¢ 22 .l . 3.; s :2"! 1111151114 E. ”iii; 3 gh’fiigiii 'i:*§iisiiiii ' ii: gieN gig: gait £11 liliifill 5:11.152 5 :2 3mil igiiii Figure II-l- Continued 0*? [MT] 23 The review consists of a description of each model, including brief critical analyses of the strengths and weaknesses of the individual models. This review is included in Appendix A. A summary of the models reviewed, point by point, is also presented in Table II-l as a composite matric of model commonalities and differences. The composite matrix was structured in the following manner. First, the experimenter listed all of the different ID steps evident in any of‘ the models reviewed. Secondly, the second dimension of the matrix was designed and included the individual models comprising the review of related research. The last step involved placing X's in the cells. An "X" indicates that ID step is found in the respective model. Once completed the matrix presents a graphic representation of the number of times each ID step occurs in the models surveyed. This frequency count was one source used to determine which steps to include in the synthe- sized operational model shown in Figure II-l. The Synthesized Operational Model The majority of the steps represented in the synthesized Opera- tional model evolved from model commonalities evident on the composite matrix of ID models reviewed in the related research and pre- sented in Table II-l. The experimenter elected to build a synthesized ID model which would, as a minimum, include the steps common to at least three of the instructional development models presented in the literature review (Appendix A). However, in the case of the Technical and Communica- tions Review step (Step 10.0.4), the Abedor Model alone was selected because it represented, in the Opinion of the experimenter, the most 24 x x x x x x x x x x x x x x x x x x x x x x x Q8.. xumnuwmu .mpoxomm .mmw>mm .mumapm>m .ucwsm_aEH x x x. mmuw>cmm ucoanam x x x x x x x x x x x x x x x x. x x x x x x mowuw>wpu< mcwccmm4\mcwcommp cmpmmo x x x x x x x x x x x x x x wnxuouoce amok a uochmcoo x x x x x x x x x x x x x x x x x x x x x x x pascou.mm_mmo pumpmm x x x x x 3m_>mm mcowumuwcsesou w _muwccumh x x x x x x x x x x x x x x x x x x x x x x x .pcou _mcowuuacumcH mmw>mm w zmw>mm x x x x x x x x x x x x x mcwppmm mNaflec< x x x x x x x x x x x x x mwmzaec< xmoh x x x x x x x mm>wuomwno mcwpnmcm x x x x x x x x x x x x x x x x x x x x x x x mumme .mcme--mm>wp -umwno chow>mnmm mo cowgmowwwomam x x x x x x x x x x x x x x x x x x .x x m_F.xm mgpcu .0 pamEmmmmm<-oc¢ x x x x 50—50.; aerucmnH x x x x .1 x x x x x x pcmeummcmz mN_:mmLo x x x x x x x x x x x x x x x x x x x x x x x ammo pumppoo x x x x x x x mpmoo _ecowpuacamcH neocm V UV 8 9 l dAHv G I. DJ VA 9 DJ 8 I S 8 3 X H 8 q I. n 0 J 0 31. m a D. D. a m a n J 0 1.. D. U. D. D. D. a 3 1. J D. d 1.3 .l. 3 q 5 w n J S L. I 0 U l. n w J D. 3 D. 3 U.0X a D. U d 6 1. 1+ 6 M D. I. I: J S O 0 a D. UD. 3 3 .I. D. D. 5 a 1.. D. w a 0 J I U A m U 3 D. 3 I: S U. S P n U I D. O S U. S A U S m mu m “cocoqsou _muoz oDNPmmgucam no .A .mmppwpmcoEEou _aeoz .o xwtuaz mpwmoesou--._-H. a_aac 25 comprehensive treatment of this phase of ID. The experimenter was unaware of any other non-qualifying (or qualifying) ID model which treats the technical and communications review of the developmental process as thoroughly as does the Abedor Model. Checklist II depicts the steps Of the technical and communications review which Abedor states must be con- sidered during ID. The sequencing of the ID steps was determined by: (l) the ID se- quencing pattern evident in the majority of the models reviewed, and (2) the experimenter's best intuitive judgment related to the most logical order of process steps for development with the BOTP. Checklists and decision aids were included as a means of creating a more Operational model than is generally found in the related research. The checklists and decision-aids delineate more precise operations or functions to be followed during some of the more complex phases of instructional develop- ment. Entry Step: Discussion Entry into the model's process is denoted by the step designated Which Step? The purpose of this step is to accommodate a theoretical assumption shared by the experimenter, Gustafson (1971) and other instruc- tional developers that 10 is a non-linear process which could conceivably begin at any step along the model. For example, a client and/or an in- structional developer might enter the process for the first time while evaluating the current instructional program (Step 17.0). This evalua- tion stage will generally lead the client and/or instructional developers into a sequence of somewhat linear steps beginning with Step 1.0 of the 26 synthesized model. The process essentially remains linear until the developer(s) recycles from decision points represented by the diamond symbols on the model or on the basis of feedback obtained during evalua- tion (Steps 17.0 and 18.0). As a result Of evaluating student tests or performances, a client could recognize that there is a problem in his course as evidenced by lower than expected or desired student performances. Thus, the client might enter the ID process by requesting assistance from an instruc- tional developer in Identifying_the Problem (Step 6.0). After acknowl- edging the existence of a problem and discussing its related symptoms, the instructional developer would, as soon as feasible, recycle to Step 1.0 in order to begin the process as prescribed by the linear aspects of the synthesized model. It is also highly probable that a client might initially ask for help in Conducting_a Task Analysis (Step 9.0) of some performance re- quirements in his course or instructional program. By the same token, he might request assistance in Specifyipg Behavioral (Step 8.0) and/or Enabling Objectives (Step 11.0). On the client might desire the expertise of a learning measurement psychologist in Constructinngvaluation Instru- ment§_(5teps 7.0 and 17.0). Regardless of what the reason for the initial contact between the client and the instructional developer(s) the ID process will require eventually a recycling to Step 1.0 of the synthe- sized model. From that point the process follows, for the most part, the specified linear sequence of steps indicated in the model. 27 Discussion of Model's Development Although many 10 models begin with a specification of behavioral objectives, the experimenter felt that the Identification of Broad Instructional Goals (Step 1.0) is a prerequisite step to specifying behavioral objectives because they provide the framework around which to develop more specific Objectives. In many of the 10 models, the identi- fication of the broad and specific objectives is customarily the function of a team of instructional developers, i.e., instructional technologist, learning psychologist, evaluation specialist, and subject matter special- ist. However, for purposes of ID with the BOTP, the experimenter recog- nized that these other specialists would not be available to participate as a team on an on-going basis during the development. Therefore, only the client and the instructional developer (instructional technologist) were listed as the subcomponents of Identifying_8road Instructional Goals (Step 10.0) and Organizing‘the Management (Step 5.0). The experimenter's personal bias regarding the ID process is re- flected in Specifying_ID Opjectives (Step 2.0). In reviewing the related research, the experimenter noted that none of the models attempted to build in a step calling for an evaluation of the effectiveness and/Dr efficiency of instructional programs designed and developed during the instructional development. It is the experimenter's belief that the instructional developer(s) must be accountable to a client for: (1) a statement of the type(s) of course improvements which are intended during the development (Step 2.0), and (2) a plan for effectively evaluating the results of the instructional development (Step 2.0). Thus, this step 28 specifies the goals and direction of the ID. The different types of course improvements and the evaluation data are listed as sub-steps in the synthesized model. At the conclusion of each cycle of ID, the instructional developer must provide statistical and/or research data to support or reject the effectiveness and/or efficiency of the instruc- tional development up to that point in time. Because of the somewhat unique monthly repetitive feature of the BOTP, the experimenter also decided that the next appropriate step in the synthesized model should be Data Collection (Step 3.0). This step is intended to give the experimenter (a non-content specialist) and the client, information on previous BOTP training schools. The data collected at this step of the process is of particular value to the non-content specialist since he will find this data useful in establishing a more effective dialogue with the subject Specialist (client) when Specifying Behavioqgl Objectives (Step 8.0). The synthesized operational model also attempts to reflect the experimenter's personal belief that instructional development includes two distinct facets: (1) the longnrange developmental requirements of a course or an instructional program, and (2) the shorturange or immediate needs of a course during development (Step 4.0). The developers must be able to "loop" in and out Of the main sequence of steps in the synthe- sized ID model. This "looping" permits the developer to consider imme- diate instructional materials production needs, i.e., transparencies, 35mm slides, videotapes, of the existing program while course development progresses simultaneously. . 29 In an attempt to effectively operationalize some of the more com- plex steps of the synthesized operational model, Checklists III and IV as well as Decision Aid II were included because of their utilitarian value in depicting the process functions more completely and concisely. The Abedor Model represents the only model which treats this stage of the process. The model provided the basis for specifying (on the synthe- sized model) the short-term developmental process used when developing instructional materials. This prototype development evaluation process is depicted by Steps 4.14 through 4.22 on the synthesized model. The data collected in Step 3.0 would subsequently be used to Identify the Instructional Problem (Step 6.0) by isolating problem symptoms, preparing tentative solutions, and analyzing discrepancies be- tween what presently exists and what is desired as a result of the instruc- tional development. Once the problem is identified the developer would thereupon Preassess Student Entry Levels (Step 7.0) prior to the start of a BOTP school. Although preassessment of entry levels normally (in most ID models) occurs after the Specification of behavioral objectives and development of relevant learning activities, the experimenter felt that this entry information might be equally useful as input for modify- ing objectives and/or learning activities in order to better accommodate the individual needs of the trainees who would be attending the various BOTP schools. The pre-tést would thus serve to identify potential learn- ing problems and needs in sufficient time to be able to appropriately modify the instruction given to a particular group of law enforcement officers. 30 Decision Aid I depicts the precise student performance goals re- quired in Specifying_Behavioral Objectives (Step 8.0). The format for writing objectives shown in the model is one recommended by Allen, Bowers et a1. (Allen, Bowers et a1., 1969, p. 27). The critical questions . which must be asked to determine if the objectives are indeed measureable (Step 8.1) are presented in Checklist 1. Most of the models reviewed included a Task Analysis (Step 9.0). However, none of the ID models reviewed specified the steps involved in a task analysis as thoroughly (at the Operational level) as Yelon's checklist of subcomponents for this step. For this reason, the experi- menter believed that including this checklist would enhance the operation- al value of the synthesized ID model. The input sources for Revising_Instructiona1 Content (Step 10.0) were identified by the experimenter while monitoring a BOTP school prior to officially initiating the formal phases of the instructional develop- ment. Irrespective of the fact that less than half of the ID models re- viewed included Enabling_0bjectives (Step 11.0), the experimenter felt, from a logical point of view, that any model which emphasizes the im- portance of writing specific objectives at the outset of ID must also give due consideration to identifying the enabling objectives which pre- pare the learner to ultimately perform as required. The remaining steps (Steps 12.0 through Step 17.0) in the synthe- sized model are all present in the majority of the models reviewed in the related research. The requirements of each step are those specified in the IDI Model of the National Special Media Institutes. 31 Summar Chapter II discussed how the experimenter developed the synthesized operational ID model shown in Figure II-l. Chapter III will present the design methodology and the research strategies, population definition, hypothesis, and research questions for the study, data analysis procedures and the limitations of the study. Chapter IV will present findings and conclusions on the stated hypothesis and the several additional research questions. Chapter V will discuss the implications of the study, recommend areas of further study, and present heuristics, or "rules of thumb" emerging from the study. CHAPTER III DESIGN OF THE STUDY Methods and Procedures The research methods and procedures used to investigate the effec- tiveness and efficiency of the synthesized model as well as the cognitive and affective influence of the process on the client are described in this chapter. Three distinct research objectives were examined in this study. The first related to synthesizing an operational instructional development model from the related literature and reporting the experiences of using the model with a client. ) The second Objective centered on statistically measuring the effec- tiveness of the instructional development process in improving student learning in the Breathalyzer Operators Training Program (BOTP) over a series of replicable monthly training programs while, at the same time, maintaining and/or improving instructional efficiency. The final objective sought, first, to compare the attitude of the client toward the instructional development process with nationwide norms and, secondly, to assess the client's cognitive growth related to instructional development. 32 33 Research Strategy The overall research strategy called for gathering and analyzing Objective, descriptive, and experimental (statistical) data emanating from the eight Breathalyzer schools from October, 1971, through May, 1972. The strategy also included the client's descriptive and post-data Obtained by the experimenter in June, 1972. Definition of the Population The population used for the study consisted of an instructional development client and Michigan police officers who attended any one of eight monthly replications of the BOTP during the period from October, 1971, through May, 1972. The week-long Breathalyzer programs, or schools, were conducted at Kellogg Center on the campus of Michigan State University and included a total of 222 Michigan police Officers ($5). The attendance figures for each of the schools is Shown in Table IIIml. Table III-l.=-Attendance Figures for Each BOTP Replication. Oct. Nov. Dec. Jan. Feb. Mar. Apr. A May T l . 2 3 4 5 6 7 8 Subjects SS 26 14 30 29 29 33 29 32 222 .t-Time i). 34 In November, half of the SS who took the final written certification examination were disqualified as subjects because an alternate form Of the regular certification examination was administered to them for purposes not related to this study. Therefore, only police Officers who.were required to pass the regular Michigan Operator Certification Examination were used as subjects. The same basic criteria were used to select the police officers for each of the BOTP schools. For example, the SS were a mixture of Michigan police officers from local police departments, Michigan State Police units, and Michigan Sheriff's departments throughout the state. Secondly, all of the officers were recommended by their department superiors to attend the school. And thirdly, department superiors generally followed the practice of recommending experienced officers who had good potential for making law enforcement a career. The client in the study was the Director of Training for the BOTP schools. Along with being the Director of Training, he was also one of six training staff members who served as instructors. The BOTP train-- ing course Schedule is shown in Figure IIIml. Collection of Data Descriptive data designed to examine the appropriateness of the synthesized model and the client's post instructional development attitude were collected using the basic techniques known as high inference observations (Kerlinger, 1964, p. 510). Using this method, an observer abstracts relevant information from his on~going observations and later 35 .m_:va;um mmcaou acmcpmch couscoao cmnapegpmmcm--._-HHH oczmmm u 9:30 2 9.9.0 3.6 .meu .58.” - and 33 2.3 I an"... cofigvauo m m w u d .u u I I .m .u .u - A... .m e n... c d Yd c B n n Y1 c 8 . u . m m. ... w. a . T... _m a "a... 2.. 3;... m.” m... ... a m .u 3.” ... a 3mm o , . t C C C I. la 0 I. .1 I. Nn' coda—«SEDAN m u m Ma m m» m u H A C T w. H .v.. cw H A C T — .4 3.300 r... m m dm mmm C .m = 96.0 m w m m .95.~0 cmun .m N! u n .n m n M B [ I n a \ M 2:“5 - v88 .. w. m a D 9. D u m m .m m. .m K a can a n g u a I. n S g n n h .I. n c u‘ i n t “3.0... 7352.0 m. m .m... ....n. .m. .m W. n... “M u .u u .u m cm :1... r OMHN u t a s e n v e e k e . s u u c u h «an 7 T emu FIL, COED-0H T T R D D o m m .53.. a . 3:00 3 9.9.0 ~ 9.9.0 2 9.0c0 59.9.0 3 96.0 a 9.0u0 : 9.9—0 u 9.9.0 on; I U z D 4 .3.& :ofimuaoum undo nonrandoun o~"~_ .acoguuoh gonou~< - 4 00:03.5 33h >no—ouflxoh o. cofluavozcu an": .250 «o .fzomEosU ON": cu v 532$ — 2...: >uo~oum5u~£m £392 anno— cofiacwEMxm 30.33... 3104 33 «3.2.3.. o~-.@ 2: _ .h n2. cozngEaxu «35.3...— xcaEEB nozaueFO 09300 - an no. u: o v .u v .35 an”: >43...— Aavncanh savuocDDB >832; >233}. .5 .< 36 makes inferences about variables. Questionnaires and attitude rating scale instruments were two additional sources of descriptive data (Abedor, 1971, p. 85). The experimenter (E) had the dual reSponsibility of interacting with the client (C) as part of a field experience, while also Observing and recording the nature of these interactions and subsequent decisions. Narrative data were collected at each meeting between the experimenter and the client. The Provus Pittsburgh Discrepancy Formative Evaluation Model (Nelson, 1970) was used to evaluate the instructional development by identifying and reporting consistencies and discrepancies between the model and the actual instructional development strategies used with the client during program development. Experimental data related to measuring the degree to which instruc- tional development resulted in improved student learning were collected and extracted from two sources: the written certification examinations and the laboratory checklist examination. For Obvious ethical reasons, copies of these instruments could not be included in the Appendix sec- tion of this study. To determine the attitude of the client toward the instructional development process, the National Special Media Institute's Attitude Toward Instructional Development rating scale was used as a norm against which to compare the client's attitude. This attitude rating scale has been validated by the Instructional Development Institutes (101) for teachers, administrators, policy-makers, and Specialists. An open-ended 37 questionnaire was designed by the researcher for use with a client as a means of obtaining additional information on the client's attitude toward the instructional development with the BOTP. The experimenter searched for a validated instrument to measure the cognitive growth of the client. This search failed to discover any available instrument. Therefore, the experimenter designed a written instrument for use in collecting and reporting information and results on the variables related to the client's cognitive growth. A copy of this instrument is included as Appendix E. Time constraints placed severe limitations on a careful validation of the instrument. The Client's Opinions Toward Instructional Development rating scale instru- ment reports the client's personal assessment of his proficiency with the process at the outset of the development and again after the termina- tion of the experimenter's field experience. Design Methodology" Baseline Data The October, 1971, BOTP school provided baseline data against which to measure improvement in student learning in seven replications of the BOTP. The October school was selected as the baseline data source because it represented the first training school observed by the experimenter. Secondly, the results of the October school were assumed to be typical of earlier schools, from which data was not readily available. Thirdly, there were no schools held in July, August, or September of 1971. Consequently, the October, 1971, BOTP school was the most recent data source available prior to undertaking instructional development. 38 Model Validation The Provus Pittsburgh Discrepancy Model assumes that the evaluation process is one of: (1) setting criteria or standards; (2) establishing discrepancies, if any, between operations and standards; and (3) using this discrepancy information to isolate the strengths and weaknesses of the program (Nelson, 1970, p. 20). For this study, the instructional development process criteria or standards are represented by the steps of the synthesized mOdel. Discrepancies were the deviations from the process depicted in the model. The discrepancy information was used to make recommendations related to using a flowchart model while engaging in instructional development and suggesting which steps of the synthesized model might need revision. The reporting Of the discrepancies and subse- quent recommendations was mainly subjective and based on the observations of the experimenter during a field experience with the BOTP. The Provus Model, according to Nelson, ”seems to hold great promise for educational evaluation" (Nelson, 1970, p. 18). Although no data was found to confirm its validity and its reliability, it was presumed by the experimenter that, because of the general acceptance and reputation accorded the model by evaluation specialists, the Provus model would be appropriate and flexible enough to evaluate the synthesized Operational model and the BOTP. The Provus Pittsburgh Discrepancy Model The basic purpose of program evaluation, as defined by Malcolm Provus in the Pittsburg Discrepancy Model, "is to determine whether to improve, maintain or terminate a program" (Nelson, 1970, p. 20). These three classes of decisions constitute major foci Of the Discrepancy Model. 39 The actual evaluation process itself is one of first setting program criteria or standards; second, establishing discrepancies, if any, between program operations and standards; and finally using this dis- crepancy information to isolate program strengths and weaknesses. After such strengths and weaknesses have been isolated, various types of problem-solving techniques may be applied. These problem-solving tech- niques are utilized in an effort to identify appropriate strategies for overcoming the weaknesses, if the decision to improve or maintain the program has been made (Nelson, 1970, p. 20). The evaluation process consists of moving through stages in content categories in such a way as to facilitate a comparison of program per- formance with standards while at the same time identifying standards to be used for future comparisons (Nelson, 1970, p. 21). According to Provus, this process of comparisons over stages is best understood through examination of a flowchart of that process (see. 5 - __ Ia c .4 . , _ . C 3 Analysis A . ' . based on P ' new Inputs Figure III-2.--Provus Discrepancy Model Process Flowchart. In the chart, S represents a standard; P, program performance; C, com- Parisons; D, discrepancy information, (A), a program change in 40 performance or standards (Nelson, 1970, pp. 21-22). Specific program content which may be examined as input might include staff qualifications, staff preprogram training, student selection cri- teria, student entry behavior, media, facilities, and administrative conditions. Specific types of transactions which might be examined might include student interaction with other students, staff, media facilities; staff interreactions with staff, students, media facilities and the administration; and student-staff interaction or transactions directly related to objectives. Specific output which would be examined would include attainment of the enabling objectives (E0), the terminal objec- tives (TO), the ultimate objectives (U0), and the inter-relationships which exist between the different types of objectives (Nelson, 1970, p. 23). Why Was the Provus Model Selected? This model was selected because the BOTP contained elements of the three generic characteristics of the temporal criteria described in the discrepancy model (Figure III-3). First, the BOTP is permanent in that the training program was already installed as one for which the sponsors had continuous responsibility. Secondly, it has continuity because it is an on-going, continuous project. Finally, it is cyClical due to its recurring, staged development, with provisions for adaptation and/or revisions between cycles (BOTP schools). However, the cyclical charac- teristic was the overriding element in the BOTP since the BOTP inStruc- tional development would most likely require revisions which would be .e_.m...u COFDuapam _aeoz--.m-e.. mes... 41 Tel. It. 9.53m x u n X A ~ a A ~ A N a 3538.3 L . aunoauusum x K N A H N H N _ mmHU ._ w u w n w H H H N A oucauuurdou oxaum W V W a 3 I 9 V A a t. x u S H o 8 e o I 3 3 m n w u w m u m. m a M 3.. a h. m u 1 L . 3 P n 3 n. e 3 u d I I H m T a J 1 .3va m w u m. m u m. m. m m n a. no"... u m. m . 1. I — 8 a T I 8 T 1.. N. u I a n u . .. T 1 1 o a. .A a K o a C . 1D 1' U 7.. P n u u mcuuumuz vouaawom £ 0 .0: _ m cmauuuaxu u u x v i _. aowuuouom oouunouox uaauao huwaunnum H-uonluh cam _ 1 nauouwuu 42 implemented between replications of the BOTP training schools. The model selection chart in Figure III-3 rates the Provus Model as first choice for the evaluation of cyclical programs. The model selection criteria also rates the Provus Model as second choice on the stability criterion. However, the distinction between whether the BOTP is highly defined or highly flexible is not a sharp one. The BOTP, at the outset, had broad general goals, but these goals were not articulated into precise behavioral objectives until the instruc- tional development process commenced. Although it had a feedback characteristic, the feedback was more of a summative type, little of which was used for subsequent program revision. The third criterion was concerned with the type of output desired from the evaluation. The main purpose of the evaluation output of the BOTP was to contribute evaluation output information which could be used in the evolution of the program. The focus was not on dissemination of results to interested publics, nor was the purpose to present an evalua- tion document for purposes of training accountability to the Sponsoring agencies. Therefore, the Provus Model is first choice on this stability criterion as well. The final selection criteria dealt with resources available. These resources are of two types: (1) expertise required for evaluation, and (2) type of financing available. Of the three evaluation models only the Provus Model accommodates evaluation by the staff without the use of expertise from local and national agencies or outside consultants. Here again the Provus Model rates as first choice on this criterion. Finally, 43 the Provus Model was selected because financing can come from a combina- tion of local and federal sources. The BOTP meets this criterion since it is supported by funds from local, state, and federal sources. Instructional Effectiveness Written Certification Examination Principal and NonmPrincipal Variables.-wTo measure improvement in student learning, two variables were identified on the written certifica- tion examination: principal and non-principal test items. Principal test items refers to those questions on the examination related to the specific teaching responsibility of the client. The client identified sixty-one such items on the Certification Examination for which he felt a direct responsibility in his teaching assignments. The remaining sixtyefive test items on the examination comprised the noneprincipal test items, for a total of 126 test items on the Breathalyzer Operator Certifi- cation Examination. Appendix 8 contains a list which identifies the principal and non-principal test items for the written certification examination. Test Rescoring by Experimenter.-~The experimenter worked only with the client and not with the other five regular instructors for the train- ing schools. Thus, it was necessary to rescore the written examinations for the eight BOTP schools to isolate the principal test questions from the non-principal. The rescoring was done by the experimenter using punched IBM scoring masks. Scoring accuracy was verified by the experi- menter through a second count of correct responses on each of the 44 examinations. To minimize the possibility of grading fatigue, the experimenter followed the policy of rescoring not more than two sets of examinations per day. Test Reliability.--The Kuder-Richardson Alternate Formula 21 for average scores was used to estimate the reliability of the principal and non-principal test items on the examination. Using the formula: — notz - RHW' . n 2 number of items rtt - (nml) 0—2 w1th ot = variance Of (P1), (NPZ) t 'R = mean score right WL= mean score wrong it was determined that the reliability for principal (P1) items was estimated to be .68 while the reliability factor for the non-principal (NPI) items was estimated to be approximately .76. It is significant to note that this form of the Kuder-Richardson Formula 21 yields a low estimate of reliability (Guilford, 1965, p. 461). In his own words, Guilford cautioned that: It should be said that all the KuderaRichardson formulas, indeed all the internal consistency formulas that depend on a single administration of a test probably underestimate the re- liability of the test. . . . (p. 461). He reported that, Of all the forms of the KuderaRichardson Formula 21, the form used in this study gives the lowest estimate of reliability. However, he gives no reasons to explain why low estimates are character- istic of the several variations of the KudermRichardson Formula 21 used in measuring test reliability. Test Validity.a~1n contrast, however, serious questions can be raised as to the validity of the written certification examination used 45 as the basis for measuring instructional effectiveness on the principal and the non-principal variables. This validity can be challenged by those, including the experimenter, who take the behavioral scientist position that the criterion test procedure must closely approximate the type of terminal behavior specified or suggested by the objectives of the program. In the case Of the BOTP, two specific types of terminal performance competencies can be deduced from the goals of the BOTP; (1) effective student operation of the Breathalyzer instrument under the conditions Of certification, and (2) satisfactory student proficiency in the techniques of testifying in court cases related to the Breathalyzer instrument. This first type of terminal objective suggests a criterion_ test designed to test the subject in a laboratory environment for pur-- poses of determining his level of proficiency as a Breathalyzer operator. The second objective implies a type of performance criterion teSting environment which permits an examination of the subject's proficiency in effective courtroom testimony relative to the Breathalyzer instrument, i.e., mock court trial. Under these circumstances, it seems that the written certification examination is not the most appropriate instrument within the context Of the goals of the BOTP. Unfortunately, however, no other instrument exists for measuring cognitive growth in testifying under courtroom conditions. Statistical Measurement Procedure.=mThe group mean scores for the monthly replications of the BOTP were reported for both the principal and the non-principal variables. A computer was used to transform the monthly raw test scores into monthly group mean scores on the two 46 variables. For example, in Group NO. 1, consisting Of twenty-six subjects, the means for the two means were 55.08 for the principal and 53.23 for the non-principal; the standard deviations for the two measures were 4.01 and 6.50 respectively. The design called for the use of a two-way analysis of variance with repeated measures to inter- act with the data. The Significance level was set at .05, although it was planned to examine significance at the .10 and .01 levels as well in the reporting. If the F—ratio in the analysis of variance shows an indication of no significance and the experimenter SOSpects that there may be Signifi- cant difference in some of the pairs, he will investigate further. For example, if the group interaction F-ratio reports no significant dif- ference, the null hypothesis--HO: P1 = P2 = P3 = . . . , = P8--would be accepted. This would mean that there is no reason to dispute the null hypothesis. But, if the experimenter suspects that there is at least one difference within the groups, he will make paired comparisons using least significant difference (Steel and Torrie, 1960, pp. 106-107). Laboratory Checklist Examination Laboratory_Performance Variables.--The second source of data for measuring improvement in student learning came from the laboratory Check- list sheets shown in Appendix C. The checklist in Appendix C consists of two discrete categories: simulator preparation and breathalyger operation. From this laboratory sheet, three additional variables were identified. 47 The first category contains a checklist of fourteen performance behaviors whereas the second category is comprised of thirty-seven per- formance check points during the laboratory examination. Because Item #35 on the checklist Sheet represents the Single major laboratory per- formance problem as evidenced by highest error frequency each month, it was decided by the experimenter to include Checklist Item #35 as a third laboratory variable. This item asks the subject to demonstrate his skill in recognizing "good" and "bad" breath samples. According to the client it is the single most important laboratory test item since it can be automatically assumed that the S will not gain certification as a Breathalyzer Operator unless he is able to recognize a "good" breath sample when taken. Framework for Reporting Laboratory Variables.--The three laboratory variables were each subdivided into dichotomous performance levels or standards as illustrated in Table III-2 on the following page. As indi- cated in this table, perfect performance scores by the SS are represented by a score of 14 satisfactory responses for the Simulator Preparation variable and 37 satisfactory responses for the Breathalyzer Operation variable. All scores on the laboratory performance examinations below these two figures would constitute unsatisfactory performance by the $5. The performance checklist (Appendix C) provides for three possible performance ratings by the examiner: satisfactOry, needs improvement, unsatisfactory. However, for this study, the experimenter combined these ratings into two discrete categories: satisfactory and non- satisfactory (Table III-2). Non-satisfactory responses include any rating 48 Table III-2.--Framework for Reporting Laboratory Variables Simulator Preparation BOTP Replication Performance l 2 3 4 5 6 7 8 Oct. Nov. Dec. Jan. Feb. Mar. Apr. May 14 Satisfactory Behavior 15 19 29 24 23 19 21 22 Responses Less than 14 Satisfactory 12 11 l 8 5 15 7 6 Behavior Responses Breathalyzer Operation BOTP Replication )y‘ 1 2 3 4 5 6 7 8 37 Satisfactory Behavior 2 6 14 13 7 9 14 12 Responses Less than 37 Satisfactory 25 24 16 19 21 23 .14 15 Checklist Item #35 _L BOTP ReplTEation 2* 3 4 5"' 6 7 8 Satisfactory Behavior 17 15 24 22 ll 25 22 25 Performance Non-satisfactory Behavior 10 15 6 10 17 9 6 3 49 given by the examiner other than satisfactory. Doing this had the effect of setting the measurement standards slightly higher than the performance levels required during the laboratory examination. For example, if the subject was evaluated as needing improvement on any items on the checklist during the laboratory examination, he was given additional opportunities, Often with prompts, to make the correct response until he performed to the satisfaction of the examiner on those items. Rarely did a subject who was initially evaluated as need- ing improvement on performance items fail to make the necessary correc- tions required for a subsequent satisfactory performance rating. Thus, if anything, the design strategy for the laboratory variables imposed stringent criteria for measuring improvement in student performance, or instructional effectiveness. Statistical Treatment for Laboratory_Variablesu-The laboratory vari- ables were subjected to a £:I£§£.Wh19h provides a statistical comparison of the proportion or percentage of satisfactory performance responses for any pair of months. The method employed is first to calculate the percentage or proportion of satisfactory responses on a variable for each month or school. Secondly, designate these proportions as P1, P2. P . . . , P 3 8' comparison. From that point, compare P1 with Pj with j = 2,.3, 4 . Next, assume the Pl (October BOTP) is the standard of . . . , 8. The tatest is then applied using the following formula: P.-P tC=—~1——-'- where j=2,3,4, ...,8 . . = l—P. PJ 0 ( J) "i .O. /_.Lfi-l—— = the SEM .i 50 The last step is to compare tC with t the tabulated t.in the table, TS d.f. = n—l, at significant levels ranging from .01 to .10. Additional Research Collection Procedures Instructional Efficiency The efficiency of instructional development will be reported in terms of whether or not it resulted in: (1) a reduction or increase in actual time of the BOTP through the addition or deletion of instruc- tional objectives and/or teaching/learning activities; (2) a reduction or increase in program costs, and (3) a reduction or increase in instruc- tional staff manahours. Client's Attitude Toward Process To assess the client's attitude toward instructional development, the Attitude Toward Instructional Development rating instrument in Appendix D was used. This instrument was developed by the National Special Media Institute for use in evaluating the effectiveness of Instructional Development Institutes (IDI) designed for teachers, admin- istrators, policymmakers, and specialists. The experimenter adminis— tered the test to the client in late June, 1972 following C's association with E for eight months. It is planned to compare the client's score on the instrument with the grand mean score of nineteen nationwide IDI'S. The grand mean score for the Institutes would be compared to the client's score on the same instrument. Finally, the mean deviation, standard deviation, and variance would be computed as a preliminary step for 51 determining the standard deviation of the Client's score from the grand mean Of the Institute groups. Client's Opinions Regarding Instructional Development The Client's Opinions Regardiogplnstructional Development (Appendix E) is an instrument designed by the experimenter to measure the client's cognitive growth by comparing his mean entry score with his mean exit score on twenty-eight items related to the synthesized model whose steps represent the process. These mean scores were subjected to a t;test_to determine if significant cognitive growth was evident in the exit mean score when compared with the entry mean score. The computed t (tc) would be compared to the tT for significance at the .05 level. The degrees of freedom (d.f.) for tT is the number of items in Entry plus those in Exit minus two. In the case of this instrument the d.f. is 54. The data would then be presented as Shown in Table III-3. Table III~3.-~Framework for Reporting tatest Analysis of Entry and Exit Behavior of the Client d.f. Comments >4 ,1: .. .. Entry Exit Instrument AdminiStration.-~The Client's Opinions Regarding_Instruc- tional Development instrument consists of a rating scale system for 52 twenty-eight specific items related to the components of the synthesized model. The instrument was administered to the client in late June, 1972, at which time the client was asked to make subjective judgments in regard to his entry and exit cognitive comprehension levels. There was a single administration of the instrument following termination of the experi- menter's field experience. Client's Post Attitude Questionnaire The structured, open-ended questionnaire in Appendix F was prepared by the experimenter for use in determining if the client planned to con- tinue to use the process with future development and to estimate his present level of interest in the instructional develOpment process. The questions and the responses would be presented in the study and the results analyzed by the experimenter using his experiences with the BOTP and his direct contacts with the client as frames of references for as Objective an interpretation of the data as possible under the circum- stances. Statistical Hypothesis Null Hypothesis 1: .There is no difference in student performance on: (1) the written certification examination, and/or (2) the laboratory checklist sheets, between the baseline (October, 1971, BOTP) mean scores and the mean scores of the BOTP schools from November, 1971, through May, 1972, i.e., 53 H I: P P = P = P ,... P with P = October 1971 0] 1 2 3 4 ’ 8 1 BOTP school .(Oct Nov = Dec = Jan...May) P1.P2.P3....P = November, 1971 8 through May, 1972, BOTP schools. Alternate Hypothesis 1: Student mean score performance on: (1) the written certification examinations, and/or (2) the labora- tory checklist Sheets, will be higher for the BOTP schools during the replications from November, 1971, through May, 1972, than mean scores of the October, 1971, BOTP school, i.e., H]: P1 < P2, P3, P4,..., P8 if P P , P ,..., P > P there is a difference. 2’34 81 Statistical Treatment of the Hypothesis H]: Involves an analysis of variance and least significant differ- erences between paired means of the principal and non-principal variables of the written certification examination. The labora- tory variables (simulator preparation, breathalyzer Operation, and checklist Item #35) were Submitted to t-test comparisons of the proportions of satisfactory responses for paired months. Additional Research Questions Five additional research questions are to be examined within the con- text Of the experimenter's field experience with the BOTP. Descriptive and observations data will be used to provide answers to the following research questions: 54 1. What differences, if any, between the synthesized model and the actual ID process used with the BOTP will be revealed by the Provus Pittsburg Discrepancy Model? 2. To what extent did the ID with the BOTP result in changed in- structional efficiency as measured in terms of: (a) an increase or decrease in actual time of the BOTP through the addition or deletion of instructional objectives and/or teaching/learning activities; (b) an increase or decrease in program costs, and (c) an increase or decrease in instructor man-hours. 3. What changes, if any, occur in the client's attitude toward ID as revealed by a structured, open-ended questionnaire adminis- tered to the client approximately one month after the termination of the experimenter's field experience with the BOTP. 4. What differences, if any, will there be between the raw score of the client and the grand mean score of participants in nineteen Instructional Development Institutes (1015) on the Attitude Toward Instructional Development rating scale instrument? 5. What mean score differences, if any, will there be between the client's entry (October, 1971) and exit (June, 1972) cognitive proficiency level of the ID process as measured by the Client's Opinions Regardingyjnstructional Development instrument? The Limitations of the Study The synthesized operational instructional development model was used with a single type of instructional system; namely, a law enforcement 55 training program. The Breathalyzer Operator Training Program (BOTP) was the setting for a year-long ID field experience for the experimenter. It is important to note that the experimenter engaged in instruc- tional development with a client who, at that time, was not an academic- ally ranked faculty staff member of the university. The client, while serving as Director of Training for the BOTP, was also pursuing a graduate program of studies. Another limitation was that the study was confined to testing the synthesized model with only one client. Therefore, it will be essential to subject the model to further examination with other individuals (and/or instructional systems) in order to determine generalizability results. The experimenter's bias while collecting and analyzing observational and descriptive data, may have, unbeknowingly to the experimenter, con- tributed to some measure of data contamination. However, the experi- menter made every effort to be as objective as possible when collecting and analyzing the data. The study was also limited by instrumentation used to collect data on the five research questions and the hypothesis. The experimenter used a formative evaluation instrument which was unfamiliar to him. In addition, the two instruments in Appendices E and F were designed by the experimenter but were not subjected to tests of reliability and validity. Moreover, the study had to contend with variables beyond the imme- diate control of the experimenter, e.g., time constraints on the experi- menter and the client, unavailability of representative samples for field 56 testing prototype materials, and less than most appropriate criterion instruments for measuring instructional effectiveness. Summary Chapter III presented the design methodology and research strategies, population definition, hypotheses and research questions for the study, data analysis procedures and the limitations of the study. Chapter IV presents the findings and the conclusions for the study. CHAPTER IV FINDINGS AND CONCLUSIONS The purpose of this chapter is to report the findings and the con- clusions of instructional development with the Breathalyzer Operator Training Program from October, 1971, through May, 1972. The chapter is designed to present student performance experimental data to support or reject the stated hypothesis as well as descriptive and observational data from the client in discussing the five additional research questions in the study. This hypothesis and research questions will be presented in the same order in which they were presented in Chapters I and III. Null Hypothesis 1 Null Hypothesis 1: There is no difference in student performance on: (1) the written certification examination, or (2) the labora- tory checklist sheets, between the baseline (October, 1971, BOTP) mean scores and the mean scores of the BOTP schools from November, 1971, through May, 1972. Britten Certification Examination Findings.--Table IV-l summarizes the analysis of variance of the repeated measures for the principal (P1) and the non-principal (NP1) variables used in the study. 57 58 Table IV-l.--Analysis of Variance for the Repeated Measures of the _ Principal (P ) and Non-principal (NP1) Variables on the Written Cert fication Examinations. Sums of Mean Source df Squares Squares F Comments r Groups 7 393.58 56.23 1.28 NS Subj-G 214 9389.30 43.88 i Total Subjects 222 3 Repeated ; Measures- 1 45.41 45.41 4.47 S (.05) (P1). (NP1) Rm*G (Interaction) 7 66.72 9.53 .94 NS Table IV-l indicates there is a significant difference somewhere within the repeated measures of the principal (P1) and the non-principal (NPI) test performances as evidenced by a comparison of the computed F-ratio (Fc) of 4.47 and the tabled F-ratio (F .05) of 3.84, d.f. = 1. However, no significant difference is reported in the interaction between the eight groups (schools) in the study. This finding is supported by a computed F-ratio of .94 whereas a tabled ratio of 14.07 is necessary to show significance at .05, d.f. 7,214. Finally, the analysis of variance shows no significant difference between the groups or months. In attempting to discover where the significant differences occurred on the repeated measures of the principal and non-principal variables, mean score differences were further examined using a t-test analysis. 59 Table IV-2 shows that there were significant differences in student performance, in favor of the principal items, on the written certifica- tion examinations during the October and the May BOTP schools. During the October school, student performance on the principal (P1) as opposed to the non-principal (NP1) test items was significant at .05, d.f. = 25, as revealed by a computed t of 2.41 and a tabled M of 2.06. The same pattern of findings are reported for the May school. In May, a comparison of student performance on the repeated measures (P1, NP1) reported significance at .05 as disclosed by a comparison of a computed t_of 2.01 and a tabled t_of 2.00, d.f. = 31. The reasons for significant performance differences during these two months and not during other months of the study are unknown to the experimenter. Further, significant differences in improved student performances on the principal as opposed to the non-principal test items on the eight written certification examinations are revealed in a comparison of mean scores as reported in Table IV-3. Without submitting the mean scores to a t;tg§t_analysis, it can be safely assumed that there has been significantly better performance by the subjects on the principal (P1) test items than on the non-principal (Npl) throughout the BOTP schools. In every school there is improved student performance on the principal test items as is clearly evident in Table IV—3. However, any generalizations relative to the impact that the instruc- tional development may have had on the mean score comparisons of the principal and non-principal variables are inadvisable since the findings 60 mo. “a unmovewcmwm oo.~ Pm Fo.~ .mm. mo.F op.~m m_.mm m ocauwewcmpm “oz mo.~ mm 55.0 cm. _e. om.mm _m.¢m N o_. 5a pcau_cwemam oo.~ mm o~._ No. mo._ 4N.mm om.¢m w pcauwc_=mmm poz mo.~ mm mm.o No. 4N. no.4m m~.¢m m ocauaa_=mwm poz mo.N mm Am¢._v- mm. Aom.v- No.0m _N.mm a op. pa “gnaweecmwm mo.~ a“ mm._ em. no.p -.mm mm.¢m m 5=a8_aw=m_m 5oz o_.N m, Amo.v-. we. “so.v. “c.8m oo.mm N me. an “cau.a_=aam oo.~ mN _e.N “A. mm.P mm.mm mo.mm P mxawsmm Amo.vpp .a.o up cum _az r .a , _nz Pa akom mucmgmemwo memo: mmpnowem> A—azv Paapocwca-coz was AFQV Pmawucagm we mwmxpm=<--.m->H mpnmp 61 mo.mw mm.—w Nm.Nm Pm.~m vp.mm mN.mm mm.pw mm.om mm.pm pmz mm.mw un.Fm mo.mw No.mm mm.wm om.om so.mm om.~m mm.pm pm uww>< mm: .Ln< .Lmz .nmm .cmw .uoq. .>oz .puo m a o m a m N F 332...; mmgzmmoz uwusmaum mmpamvgm> Apmzv Fwawucwgaucoz new Apmv Paavucpea mg» Low mmezmmmz umpmmamm co mmgoom :mwzcn.mu>H mpnmp 62 which manifested themselves during the BOTP schools from November, 1971, through May, 1972, were also visible during the October, 1971 BOTP school. Data associated with the principal and non-principal variables were further analyzed to determine if there were any significant differences between paired means. These data are reported in Table IV-4. Table IV-4.--Ana1ysis of Variance for Paired Mean Scores Group No. BOTP 1 2 3 4 5 6 7 8 N0. of Subj. 26 14 30 29 29 33 29 32 Mean Score-P.l 55.08 56.00 54.33 55.21 54.28 54.30 54.31 53.16 Standard Dev. (P1) 4.01 3.04 3.09 2.96 3.94 4.33 3.88 6.34 Mean Score- (NP1) 53.23 56.07 53.27 56.21 54.03 53.24 53.90 52.09 Standard Dev. (NP1) 6.50 5.51 3.90 5.70 6.65 6.85 4.66 7.35 The data in Table IV-4 was used to compute least significant differ- ences (lsd) between paired means. The lsd is basically a student's 't:t§§t_using a pooled variance as a timesaver over making individual t-tests. For the difference between the two means to be significant at the 5% level the observed differences reported for the repeated measures 63 must exceed the lsd (Steel and Torrie, 1960, p. 106). The findings on the least significant differences between paired means on the repeated measures, e.g., November (2) with October (1), December (3) with October (1), etc., are reported in Table IV-5. Table IV-5.--Least Significant Differences Between Paired Means on the Repeated Measures of the Principal (P1) and Non-principal (NP1) Variables October - Repeated Measures Treatment X lsd di’. 2 3 4 5 6 l 7 8 P1 55.08 .02 -(.74) .13 -(.80) -(.77) -(1.92) 1.69 7 'k * NP1 53.23 2.84 .04 2.84 .80 .Ol .67 -(1.14) *Significant differences. An analysis of Table IV-5 reveals a significant student performance improvement, over the October BOTP school, on the non—principal test items for the November and the January schools. No significant difference in student performance on the repeated measures for the principal items is discernible. Conclusions.--There is a non-rejection of the null hypothesis on the two experimental variables related to the written certification examina- tions. This conclusion is verified by findings of no significant differ-h ence on all of the repeated measures on the principal and most of the . non-principal variables. However, in regard to this, the experimenter 64 suspected that the doubtful validity of using a written examination to measure psychomotor types of skill performances may have contributed to findings of no significant improvement in student learning on these two variables. r. Laboratory Variables Findings.--On the simulator preparation laboratory variable, sig- nificant improvements were found when comparisons were made between the baseline month (October) and seven subsequent replications of the BOTP ; schools. Table IV-6 contains a matrix of differences and corresponding t:tg§5.for the difference matrix on the simulator preparation laboratory variable. The tztggt analysis for the differences between paired combinations of BOTP schools indicates non-significant improvement in student per- formance on the simulator preparation variable during only the November and February schools. The data in Table IV-6 shows significant differ- ence for the remaining paired combinations of BOTP schools with the baseline school (October). The reported significance for these combina- tions is between the .01 and the .05 levels. Of particular importance is the significant improvement at the .02 and .01 levels when March (6), April (7), and May (8) are each compared with the October (1) BOTP school. One final interpretation of the data is that significant performance improvements at the .01 and .02 levels respectively were reported when comparing paired months toward the end of the instructional development cycle, i.e., April with March, May with March. 65 Po. 58 beau_a_=mwm... No. Na pcaopcwcmtm44 mo. Na pcaowcwcmwm. . Nd “mam a N2 mNNo. we. .mmmN-b mmaw . «O. o N NPNO 4..NN.N ..NN.N ommu-p .Laz . NN. NF. 3 m 5mmo (~84.V- .hNN.v- .dNo_N~- emmp-p .naa _NNo. «qo.q- ANo.V- ANN.V- o m we. go. ANN.NV- 1mm. “WIN-“ .caq . 50. oo. mp. No. a N NmNo .lkem NV- Amm.Nv- ANN Np- “No.NV- ANN.NV- Swap-“ .Nwm. NNmo. AN_.V- ANN.V- _5.- Am_.v- ANN.V- o m .Nm._ N¢.F NN», 9..P8.N Nw._ ...5o.oF “map-p n>02 NNNo. m_. N_. ANo.V- m_. N_. mm. o N ...ha.N .9NN.N ..NN.N #0. 6.36m» .«Nm.N “WM” bNNS-» .Nuo MN. m_. co. NN. NP. ._5. No. a _ Am: .Lq< .Lmz .nmm .cmq .uwo .>oz mhom Loegm w n m m e m N vemucmpm .u_nm_em> zaopcgonag cowpmemnoga Loam—sswm 0;» co xvepaz mucmemwewo esp to; “magi“ mcpucoammegou can Any mmocmeoemwo to xwepnz--.m->H wpnmh 66 Significance at the .01 level was also found on the Breathalyzer Operation variable when subsequent BOTP schools were each paired for comparative purposes with the October BOTP school (Table IV-7). The only exceptions to significance at the .01 level were evident for the November BOTP school (significance at .10) and the January school (significance at .05). Significance ranging from .01 when comparing December and March with November and .02 when January and May are each paired with the October school are also reported in the matrix in Table IV-7. As in the case of the simulator preparation variable, there were significant findings among several combinations involving BOTP schools during the last several months of the experimenter's field experience with the BOTP. For example, the student performance on the laboratory variable related to Breathalyzer operation showed (1) significance at the .01 level when April is compared to February and .02 when April is paired with March; and (2) significance at .05 when May and February schools are matched. Corresponding t;tg§t_differences for the difference matrix on the Checklist Item #35 of the laboratory examination checklist sheet are recorded in Table IV-8. Although there was sporadic evidence of sig- nificant improvement at levels ranging from .01 to .05 in several of the paired months up to the April BOTP school, the most important findings related to paired comparisons of the April and May BOTP schools with .earlier training schools. These comparisons are of particular significance to the study because an audio tape, designed and developed specifically to improve sthdent performance on Checklist Item #35 on the laboratory \- .h FM h F‘ In: N _o. Na pcauwcwcmwm499 No. we pcauwewcmwm49 mo. 58 SeauPewcm_m. mm r Am N m N: «moo. mm. pmwmrp .NQ< . “mo.v- a N mNmo _N.— aaNm.N pmwp-p .Lmz . o_. NN. o 0 wake a¢o.N Naamo.N 1mmw “maplu .amu . 7, m_. mN. mo. o m a_mo 6 be“. llaww. nnm._v- Four. Swapnw..=aq memo. 5o. mo. Am_.v- AmF.V- o N ANN.V- mm. . ANN.NV- 6mm.NV- NON.N- Name-» .uao P_mo. . ANo.V- No.v- Am_.v- NNN.V- Nwo.v- o m «*E xxxwfim No; _b . E manqw ummpup .>oz wmmo. «N. om. mo. mo. _N. NN. a N «*«wm.m «xxmm.v ¥*¥_9.N xm—.N «*kvw.m xxx—m.v mm.P wwwuuu .puo um. mw. _N. mp. mm. mm. m_. a F Xe: .Lq< .Lmz .nwu .cmc .uwo .>oz mhom Loggm m N o m w m N ucmwcmum mpnuwgm> agoumgonm4 :o_umemao Lm~apmgaumgm ecu co xwgpmz oucmgommwo ms“ Loy pmmuuu mcwucoammsgou new on mmocmgmmmwo mo xmgpwz--.mn>m wpamh I .. viii . 4 . .11 .. . .5. ul-..~;\ no... pun. X-..e-.Z .6...~.u...~u -.H- u6.- ...a-> ,s u...l~ .g ~..-s..~.,v..cu.u.s.n.~.» ~v.snw Nfi~v .0... v-..~.L..~\ \\\~ \a» .X.~-L..s Fez.-. -. a.‘ \4 ~ ..:o\~\-..\ Po. Na pzaoweazm_m... NO} ow pcaoaewcmwm49 mo. 56 “cau_cwcmwm. 68 mlwr n p M N mwmo. m. mmla N_._ N M a mNNo. 9*.ON.N mo. “mmpnp . m_. mo. 9 mmNo «*«nm.w kkxno.m «*xmm.¢ pmwpup . om. mm. 5N. o m NNmo 9.9Nm.m m_._ me. Amo.mv- Nmmmrp .caq . _N. o_. mo. NN. - a 5 arms mm._ hmrwv. A . - ANN.¢V- ANN._V- pawl-» .aao . mo.-- A_o.v- Aoo.vu A_v.v- A_P.V- o m NNNo *amvnom f..«mm.m fimfim HER CI «mg .32.,me age“ .>oz . mm. mN. 5N. A__.V- m_. om. a N _Pmo ...Nm.¢ ._o.N os._ ANm.N - _N. ..NN.N N¢._ Camp.» .380 8N. m_. F_. AVN.V- mo. N_. ~N_.Vn o _ zmz .La< .Lmz .nwm .cmw .umo .>oz upon. Loggm N N o m N N N caaucatm mpgawaa> zaouaaonad mmN sauH wa_xua;u mg» co xwgpmz wucmgmmmwo ms» go; away.» mcwccoamwggoo ucm on mmucogmmwwo No xwgpmzu-.mu>H mpnmp 69 checklist sheet, was first introduced into the BOTP instructional program during the April BOTP school. (This same audio tape was used during the May school.) An analysis of data in Table IV-8 reveals overall significant improve- ment in student performance at the .01 level on the Checklist Item #35 .3 variable for all paired comparisons involving the May BOTP. The only exceptions were when the May performance levels were compared with those in December and April. Regardless, there was a reported student improve- }fi— fi“ :— ment at the .10 level when the May BOTP was compared to April as well as improvement in the performance improvement of the May BOTP over the December school. Conclusions.--There is a rejection of the null hypothesis that there will be no improvement over the October BOTP on the laboratory variables. Although the simulator preparation variable reported insignificant improvement in student performance during the November (2) and February (5) schools, this might be explained in part by two factors. The first factor is that the instructional development which might have had any in- fluence on performance during the November (2) school was too minimal at that point in time to bring about a significant change. The second factor which may account for the irregularity in performance during the February (5) BOTP school might be explained by the fact that the client did not teach that school. Since no other instructional staff members were directly involved in the instructional development activities, it is reasonable to expect that quality control in the instruction on the simué lator preparation variable was adversely affected. Therefore, the 70 experimenter is of the opinion that performance during November and February might also have been significantly better than that in October if the regular instructor (C) was available to teach the laboratory sections on simulator preparation. Aside from this, the strong signifi- cance at .01 and .02 levels for all combinations of months with October a offered impressive evidence for rejection of the null hypothesis. Moreover, the significant improvement apparent when: (l) the April school performance on this variable is compared with March (.05), and (2) the May BOTP school performance is compared with March (.01) gives further support to the conclusion on this null hypothesis as well as evidence that the instructional quality on this variable may have been somewhat stabilized during the last several BOTP replications. In attempting to relate improvement on this variable to instructional development, Figure IVal is designed to graphically depict the specific changes introduced into the BOTP as a result of the instructional develop- ment. It might be assumed that these instructional changes and materials would contribute to the improved performance reflected by the statistical data related to the laboratory variables used to test the null hypothesis. In the case of the simulator preparation variables, for example, the additional laboratory time introduced into the schedule for the BOTP in December and the 35mm laboratory slides designed and developed for use with the January and subsequent schools seemed to correlate with the schools which first reported significant student improvement on simulator preparation. Thus, it might be concluded that the additional laboratory 71 Oct. Nov. Dec. Jan. Feb. Mar. Apr. May 1 2 3 4 5 6 7 8 Entry Course Syllabus Additional Laboratory Time Laboratory Slides Transparencies Principles of Breathalyzer Audio Tape Breath Samples I Figure IV-l.--Instructional Development Inputs Into BOTP Schools. time introduced in December and the 33mm laboratory slides may have had significant bearing on student improvement in all but the February BOTP school._ This conclusion relative to the potential value of the labora- tory slides is confirmed by significant improvement at the .05 level or better for all BOTP schools under instructional development. The finding also revealed significantly improved student performance during the April and especially the May BOTP schools over the December BOTP school. These findings are important to the study because they lend support to the fact that the audio tape of breath samples which was specifically designed and developed to alleviate the high frequency of 72 unsatisfactory performance on Checklist Item #35 characteristic of earlier schools might have been the single most important factor to explain the improvement. Thus, in summary, it may be concluded that the findings and the analysis of the findings support a rejection of the null hypothesis and an the acceptance of the alternate hypothesis that the instructional develop- ment did contribute significantly to improving instructional effectiveness during the laboratory instruction. Summary Hypothesis Matrix Table IV~9 contains a summary matrix of conclusions on the null hypothesis of this study. Table IVw9.~~Summary Matrix of Conclusions on the Null Hypothesis of the Study Hypothesis #1 Instructional Effectiveness: Principal vs. Nonwprincipal Variables ...... Qualified Non=rejection Simulator Preparation Variable.... ......... Rejection Breathalyzer Operation Variable ............ Rejection Checklist Item #35 Variable ................ Rejection 73 Model Discrepancies and Consistencies Question 1: What differences, if any, between the synthesized model and the actual instructional development process used with the BOTP will be revealed by the Provus Pittsburg Discrepancy Model? In presenting descriptive data related to this research question, the procedures of the Provus Discrepancy Model were followed as described in Chapter III. The steps of the synthesized operational model in Figure II-l represent the standards, or criteria, for the instructional development and any deviations from the model constitute the discrepancies. While analyzing the process used with the client, the experimenter reported whether there were any discrepancies from the model, and the specific activities which took place at each step during actual instruc- tional development with the BOTP. The model, as represented in Figure II-l is intended to be non- linear in the sense that the instructional development could conceivably begin at any point along the flowchart model. The non-linear aspect of the model is depicted by the step called Which Step? Therefore, the logical starting point for applying formative evaluation using the Provus Model is to first identify the entry point(s) on the model for the experimenter and the client and then discuss the operations which occurred in subsequent steps. Which Step?--For the experimenter, the entry point was Step 1.0 on the model. The experimenter met with the client to discuss the possibili- ties of a field experience with the BOTP. During the discussion, the client revealed the broad instructional goals of the program. For the 74 client, the probable entry point was the evaluation step (Step 17.0). The client indicated during this initial meeting that he was dissatisfied with the performances of certified breathalyzer operators on recertifi- cation examinations. Thus, the client made the decision to reexamine the entire training program with the expectation of identifying the problem and then taking whatever appropriate actions suggested by the evaluation data to improve the BOTP. Identification of Broad Instructional Goals (Step l.O).--Two broad goals were identified by the client during the initial meeting. The first goal is to train and certify selected Michigan police officers to effectively operate the Breathalyzer instrument. The second goal is to teach these same police officers how to testify more effectively in court cases involving breathalyzer cases. Since the broad goals of instruction were clearly delineated by the instructional developers, there was no function discrepancy from the synthesized model on this step. Specify Instructional Development Objectives (Step 2.0).--In an attempt to demonstrate accountability for the instructional development process, the experimenter specified his instructional goals for the BOTP: (1) significant improvement in student learning or performance, or (2) in- creased instructional efficiency. Methods for measuring achievement on these two objectives are specified in the two blocks of the model listed at Step 2.0 and the experimental design described in Chapter III. The achievement of these objectives are reported with hypothesis one in this chapter. Therefore, no discrepancies related to this step were evident. 75 Collect Data (Step 3.0).-~A variety of data was collected during the BOTP instructional development. This data took several forms. First, the experimenter participated as a Subject in the October school and monitored parts of several of the other BOTP schools. The client also monitored selected segments of the instruction during all of the schools. r7 Secondly, staff reports from earlier schools were examined and analyzed by the instructional developers. Thirdly, an item analysis was con- ducted for the instructional developers by a BOTP graduate assistant. This item analysis provided an error frequency count on questions related L to the written certification examinations for ten randomly selected. schools operated in 1970-71. The experimenter also conducted an error frequency count on checklist items related to the laboratory examinations for each BOTP school. This was used as formative evaluation data for making subsequent instructional decisions. Fourthly, the experimenter, while participating as a student in the October BOTP school, and as an observer during parts of several other schools, solicited informal input from the instructional staff and participants in these schools. Therefore, it was felt that there were no discrepancies from the requirements of the model on this step. Immediate Instructional Materials Needed? (Step 4.0).--This step was designed as a decision point for determining if there was a need to develop instructional materials to support the immediate needs of the BOTP while simultaneously considering the long range planning and develop- ment represented by the other steps of the model. This decision point in the model "loops" the instructional developer out to a supplemental produc- tion sequence of steps designed to produce instructional materials. 76 This decision point in the model proved useful to the developers during the design and development of immediate instructional materials, i.e., overhead transparencies, 35mm slides, and audio tapes, for the BOTP. Therefore, the experimenter concluded that this step should be a basic part of ID models. Organize the Management (Step 5.0).--For this study, the instruc- ‘ticanal development management consisted of the client and the experi- nueriter primarily. At various times, other instructional developers con- SLzl'ted with the client on matters related to the BOTP as well as other developmental problems in the Highway Traffic Safety Center. Although "“3713! instructional development models recommend the inclusion of an eva'l uation specialist as a member of the 10 team, there did not appear (to the experimenter) to be a critical need for this type of specialist ‘jl‘Y‘i rig the limited cycle of development represented by the field exDelrience. The instructional management had been organized prior to the experi- mehter's arrival on the scene as an instructional developer. For example, 1:1153 'following individuals were regular instructors for the BOTP schools: (1) Dale Dummer (2) Dr. Robert Howenstine (3) Sgt. Francis Korpal (4) Jerry Stemler (5) Floyd Smith (6) Robert Mills The BOTP also had support from their own staff of instructional media 5; lDecialists who produced materials upon request, and made logistical Q l" . hangements for media used during the BOTP schools. However, their 1 c: apacity for designing materials was limited enough to require the 77 developers to use the services and facilities of the Instructional Media Center. Thus, there was not a discrepancy between the synthesized model and the actual instructional development process used with the BOTP on this step. The management was well organized and included the instruc- tional developers, instructors and auxiliary services. The only dis- crepancy which should be reported is the model '5 failure to list .NM 1.3“ 91,1 instructors and auxiliary support personnel as elements in the organiza- tion of management for an ID project. Identify the Problem (Step 6.0).--During this step of instructional development, the client identified symptoms of the problem. For example, he was concerned about the increased number of certified Breathalyzer Opera tors who failed to maintain their certification or to perform as ”V531'1 as was expected; the apparent discrepancies that seemed to exist betWeen the BOTP training and the way Breathalyzer Operators functioned 1" the field; and an intuitive feeling he had that the quality of BOTP instruction was deteriorating. It is interesting to note that the identification of these symptoms (’<:<211i~red during Step 1.0 of the synthesized model. This may suggest that the Defining the Problem step might also appear at the beginning of an i "structional model. The problem was identified as a performance discrepancy between how the subjects are trained in the BOTP schools and the way they actually perform in the field. The instructional development attempted to find ‘Vei;>,:§ to identify the discrepancies and to eliminate the gap between th - a ‘l ning procedures and operational procedures used in the field. 78 Thus, the experimenter believes that there were no discrepancies on both identification of the problem and the proper location of this step in an ID model. However, it is important to preface this conclusion by noting that discriminating between symptoms and problems is one of the more difficult aspects of the instructional development process. Pre-assessment of Entry_Skills (Step 7.0).~~No terminal performance pre-tests were administered to the subjects prior to or at the beginning of the BOTP schools. Nor were there any attempts to identify pre- requisite requirements for successful performance in the school. Pre-assessment of entry skills was one of the variables which the experimenter found impossible to control during this first cycle of in- structional develOpment. The client did not reach the stage of desiring to devote time to this task. However, he did admit the importance of this function in planning for effective instruction. The experimenter concluded that, along with the functional discrep- ancies described above, there might have been a sequence misplacement of this step in the model. From the experiences of the study, it appears that pre-assessment of entry skills would more logically follow the specification of behavioral objectives, and not the identification of the problem as shown in the synthesized model. Although the misplacement of this step in the model did not have serious implications for the BOTP development, it could prove to be a real problem during subsequent stages of ID with the BOTP. Any attempt to identify prerequisite skills before stating the behavioral objectives could be counterproductive. 79 Specification and Measurement of Behavioral Objectives (Steps 8.0 and 8.1) and Identifying Enabling Objectives (Step ll.0).-—Specific terminal objectives were evident from the outset of the ID. However, neither the terminal objectives nor the enabling objectives were specified to the complete satisfaction of the experimenter. The difficulty of mak- gr: ing significant changes in the original BOTP objectives was compounded by . the fact that the instructional developers were functioning independently of the other BOTP staff and administrative units. Efforts by the experi- menter to coordinate the planning and extend the ID to include all human L components of the BOTP instructional and administrative staff failed to H materialize. The client elected, contrary to the suggestions of the experimenter to follow the policy of limiting the ID to only those phases of the instruction which were taught by the client. It was the client's intent to develop his instruction as a "model" for the other staff members to follow in their own instruction. Thus, there was a discrepancy in this step since all elements of the management were not anticipated and represented on the synthesized model. The enabling objectives were generally reflected in the learning activities of the BOTP but were not as carefully developed and specified as the experimenter would have preferred. Nevertheless, the terminal and enabling objectives were specified sufficiently to enable the developers to make significant headway during this first development cycle. Therefore, a discrepancy was evident at this early stage of develop- ment in that the specific and enabling objectives were never specified to the complete satisfaction of the experimenter. 80 Conduct Task Analysis (Step 9.0).-—The experimenter, in his role as an instructional developer, organized a structure for interviewing certified Breathalyzer Operators in the field for data relevant to developing a task analysis of the functional role and the on-the-job behaviors of instrument operators. However, time constraints limited pr the task analysis to telephone interviews of approximately one dozen randomly selected certified Breathalyzer Operators. These same time constraints prohibited a follow-through by the developers on the findings. The client made several attempts to develop task descriptions of the Breathalyzer operation function. However, the task descriptions were not completed during this cycle of development due to time constraints and insufficient task analysis information. The consequence of not com- pleting a task analysis was that the instructional developers remained uninformed of the exact procedures used by the operators in the field. The experimenter concluded, however, that this step of the model should precede the specification of objectives and follow the probiem identification step in the model. Review and Revise Instructional Content (Step 10.0).e-The instruc- tional developers examined the training manuals of several other Breatha- lyzer programs with the intention of using the information for purposes of course revision. Once again, however, the time constraints imposed on the experimenter by the field experience and the client (because of his other responsibilities to the Highway Traffic Center) resulted in an incomplete execution of this submstep in the model. Several significant revisions were made to the BOTP as a result of input information. In November, for example, it was decided to add 81 three hours of laboratory time to the regular schedule to give the sub- jects, or learners, more time to develop laboratory concepts and skills. During the same month, a preliminary course syllabus was prepared by the experimenter and a graduate assistant from the Highway Traffic Safety Center. Both enrolled for the BOTP October school and again monitored the class activities in November to collect the data needed to ' assemble a preliminary syllabus. 1 The value of the syllabus was that it served to make the BOTP f instructors conscious of the curriculum content presented during each : segment of instruction. It was hoped that the syllabus would reduce the instructional duplication and overlap in teaching assignments. The experimenter, while monitoring subsequent schools, observed that the regular instructors were more aware of what was being taught by their colleagues and were attempting to minimize unnecessary duplication in their instruction. Sub-step 10.0.3 of the model called for utilizing student feedback. Since written student feedback was solicited and obtained during the laboratory performance examinations, it was concluded that no discrepancy was in evidence during this step of the ID process as represented by the synthesized operational model. This student input was used to make course revisions. Technical and Communications Review? (Step l0.0.4).--This step was of value to the instructional developers during the design and develop- ment of the instructional materials. Checklist II in the model was use- ful as a guide for examining the content accuracy and the technical quality of the materials produced for the BOTP. 82 In conclusion, there was no discrepancy from the model on this step during the instructional development. Analyze Instructional Setting (Step 12.0).--As indicated earlier in the study, no attempt was made during this limited cycle of instructional development to analyze learner characteristics for purposes of organizing learning experiences to accommodate different student needs while using a variety of instructional patterns, e.g., large group, small group, and individualized instruction. Although its value was recognized by the developers, time constraints restricted significant changes in the over- all instructional sequences of the BOTP. These training schedule changes would be unlikely to occur without the agreement and commitment of the staff personnel who were not involved with instructional development. However, there was no discrepancy in analyzing the physical facili- ties related to the BOTP. Rooms were reserved well in advance of each school and were of sufficient size to effectively accommodate existing patterns of instruction and group size. The only problem with room scheduling was caused by low ceilings in the laboratory spaces. This caused viewing difficulty when using slide projectors and screens. The problem did not appear to be a significant one in terms of adversely affecting instruction. The experimenter concluded that there were no discrepancies at this step of the model. Construct Prototype Test (Step 13.0).--A discrepancy was evident at this point in the process. Many prototype instructional materials had been developed for the BOTP. For example, transparencies were designed to aid the teaching principles related to the Breathalyzer instruction. 83 An audio tape for breath samples and 35mm slides were introduced into the course development. Prototype testing, according to the Abedor model (1971), requires that the materials which have been developed as prototypes, should be field tested on a representative sample of learners. Feedback from these learners is then used as a basis for revising the prototype prior to its use in the classroom. The instructional developers in this study found it impossible to test the prototype in this manner for two reasons: (1) the unlikeliness that the client would want (or be able) to find the time to validate materials by "teaching" a relatively long sequence of instruction to a small group of learner; (2) the unavailability of a representative sample of students at times which would be mutually con- venient to instructional developers and students. Therefore, the validity of the materials was evaluated and revisions made on the basis of observations in the actual instructional setting and data from the performance results on items related to the materials or the criterion measures. In summary, the experimenter concluded that (1) there was a discrep- ancy from the model on this step because a representative sample of students was not used to validate the instructional materials which had been developed, and (2) it is likely that instructional developers will not always have a representative sample of learners readily available to them for testing the validity of prototype materials. Thus, it is more conceivable that prototype evaluation will have to take place within the 84 context of a regular class(es) or through the use of a representative sample of learner. Subsequent revisions would then be made on the basis of student feedback after the use of the prototype materials. Determine Teachipg/LearningpActivities and Methods (Step 14.0).-- A discrepancy also resulted at this stage of development. The experi- 3p menter did not attempt to analyze the learning activities as required | in the Merrill-Goodman manual (1971). The reason for this was that the I teaching/learning activities did not significantly change during this first cycle of development; the main changes were in the development and validation of instructional products which were used with the existing BOTP curriculum. It was unnecessary to use the MerrillvGoodman manual in matching media forms with learning activities because, in this study, the media selection was fairly obvious to the instructional developers. It is likely that the manual would become more useful to the instructional develOpers when that stage was reached where the client was prepared to re-examine and revise the instructional content. However, the experimenter recognized this discrepancy early enough during the process with the client to make necessary adjustments. Thus, even though there was a discrepancy relative to the proper sequence placement of this step in the model, no problems were caused which af- fected the ID with the BOTP. ‘ Schedule Support Services (Step 15.0).--No discrepancies can be reported on this step since the developers, without exception, scheduled equipment and instructional materials well in advance of their use date. In fact, the equipment and materials were generally available a day or two before the use date so that they could be checked out beforehand. 85 Implement (Step 16.0), Evaluation (Step 17.0), Measure Achieve' mgpt(s) (Step 2.0), and Recycle (Step 18.0).--There were no discrepancies at any of these points in the model. The evaluation (Step 17.0) and accountability components (Step 2.0) are well documented in other sec- tions of this study. Recycling would have been a logical step in the r- process had time permitted. Summa ry_Ta b1 e Table IV-lO uses a matrix format to summarily review and report discrepancies and incomplete execution of process steps during the in- Str‘uczi:ional development with the BOTP. For those steps left blank, it can be assumed that no discrepancies from the model were evident. The matrix also includes comments designed to further explain the findings. my Conclusions Related to Question 1 Several conclusions can be made relative to the model used during the BOTP instructional development: (1) On the basis of the ID experiences with the BOTP, the experi- me“her formed the opinion that instructional models should include a phoduction sequence, e.g., Step 4.0, designed to meet the immediate 1 "3 tructional needs of a program while longer range planning and develop- Teht is taking place. Figure IV-l illustrates the various types of 1 “3 tructional materials developed simultaneously with longer range D 1 a hning activities, i.e.,‘audio tapes, slides, overhead transparencies. (2) Definition of the problem is probably the most realistic start- ing . . . . . paint for ID Since the client must first recognize and express a 86 .mpuwnoen QH Hmma cm o.¢ Nuoumwz mpmwempwz manage; was: muumamv Arrmuepmmemm .rzemm: chowpozeumcH muewumEEH o.N_ epuaeem -o.mp immUW>me ueoagam a_zvmmum .c.Npimwem o.e_ me_ee>eeu< zoFFON .mvme mmmcmgo ucmpcou pcmomwmcoem oz imcwcemw4\m:w;umwk .pwo .mvocuwz mewsummp use mmwpw>wpo< cw o.m_ pmwh wazpopoem uozepmcou iceww4\mcw;ummp .wmo sop—om upsogm ampm .mFQEMm r_mEm mo zpwewnMme>mca .mpcwmeumcou we?» o.N_ mcwppwm _mcowuuzepmcH mmxfiec< .o.w mmum zo_—om .umcwammUuwe we on muwmz o.__ mm>wpumwno.m:e_nmcm xewpcmcH ¢.o.o_ 3mw>wm mcowumu iii. -wcsesou use _eowczuwp uozucou .mpcweepmcoo meek o.o_ ucmpcou _e icowpuzepmcH mmw>mz ecu 3mw>mm .quoa o.m mme_e:< xmee puzucou cw o.o eeem eo__oe e_=o;m eeem .eeeeeeeeswee .umcememime ma Op mummz c.w mm>muuwwno _meow>o;wm mo coepmuwewowam .quoe cw _.w amum zo_—ow u_:o;m ampm o.N m_wam xepcm wwmemumLm o.e .aw_eoee NeweeeeH .awumanzm m we mpcmscmwmmm o.m pcwsw new: mecm Lo m>weeeumewsue wee mempm weapon? u_:o;m quoz o.m mums powFFoo c.N me>tuoeweo DH Neeoeew o.F me>eeeeheo mmmem aeeeeem Neeem waem euemm mpcwEEou .oz mmum ucmco sou —mcoz chowpuzeumcH 1" mzoeuuzzm amen mo :oNNL : I) “segue m>m once u e .m.__e..N....ee ea .3 ee 3e Ease e e e «av mmhueequeowNQ 95.8%: Ste: \eeeesmiéic. 26.: 87 need for implementing changes in an existing program. (3) Instructional models are capable of producing significant improve- ment in student performance even when discrepancies are evident, as was the case in this study.. (4) Operational models must include steps which list and describe how 'to execute the various functions, as well as criteria checklists for .1 nu... F: J): UH deteerfinining how well each step has been executed. (5) Instructional models should contain a step similar to Step 2.0 Of the synthesized model as a way of representing the goals of the 1...)”: insizr~11ctional developers and procedures for measuring the achievement of ”1958 goals. Instructional Efficiency Question 2: To what extent did the instructional development with BOTP result in instructional efficiency as measured in terms of (a) an increase or decrease in actual time of the BOTP through the addition or dele- tion of instructional objectives and/or teaching/ learning activities; (b) an increase or a decrease in program costs, and (c) an increase or decrease in instructional man-hours? Findings.--There was an increase in (l) instructional time; (2) pro- SJt~ Ei"1 costs, and (3) instructional man-hours. The increases in instructional time and instructional.man-hours were I: he result of two additions to the learning/teaching activities in the 88 BOTP schedule for which the client had direct instructional responsibil- ity. f_i_r;s_t_, a two-hour open laboratory period was added to the BOTP schedule on Monday nights to afford the subjects more time in the labora- tory for practice. This schedule change necessitated the presence of laboratory assistants as well as the client on several occasions. The main purpose for the client's presence during the open laboratory period was to monitor the activities while, at the same time, assessing the value 01" the change. Secondly, the length of the client's instructional segment on the Principles of the Breathalyzer (Monday mornings) was increased by approximately one hour at the expense of the time normally assigned to instruction on the Metric System. This change was the result of a recom- mehdation made to the client by the experimenter after monitoring the OC to her and November BOTP schools. The instructional costs of the BOTP was also increased by approxi- ma‘t’—ely two hundred dollars. This cost increase was attributed to the deg ‘i gn and development of new instructional materials to deal with speci- F1 Q instructional problems, i.e., transparencies for teaching the hi hciples of the Breathalyzer instrument, 35mm slides for laboratory 1. h Struction on simulator preparation and Breathalyzer operation, and an and ‘i o tape which provided discrimination practice between "good" and “ Ibat!" breath samples. However, the one-time cost of producing these ma t«serials was quickly amortized and regarded as relatively insignificant fig the materials were reused in subsequent replications of the BOTP ChOol. Also, there is no way to estimate what normal costs might have In & terialized irrespective of the instructional development. It must be 89 noted, too, that the instructional developers, on several occasions, discussed tentative plans for revising and redesigning these instructional materials to make them more amenable to self-instructional applications which would be more instructionally efficient in the usage of staff man- hours, However, the time constraints prohibited the instructional developers from designing and developing any self-instructional packages of materials during the first cycle of instructional development. Finally, the time devoted to instructional development by the client and the experimenter might be viewed as a negative efficiency factor. Although it was apparent that the client was dedicating himself to long hours of systematic design, development, and evaluation of his instruc- tion . it could not be precisely determined how much more time the client Was devoting to instructional development than he did prior to the field experi ence. Conclusion.--The findings suggest that the instructional development resulted in increased instructional time, increased program costs, and i "creased instructional man-hours. This increase in instructional man- houY‘s pertained both to the client's instruction in the BOTP as well as to the time he spent engaged in instructional development with the experi- menter. In retrospect, the experimenter concluded that instructional effi- Q‘i ehey is difficult to achieve during instructional development unless: ( 1 ) self-instructional materials are designed, developed and implemented 1 h to an instructional system, or (2) more students are served with the Sam . . «3 staff. These self-instructional materials would have the potential 90 of reducing the instructor's (client) man-hours, in classroom instruc- tion and the effect of amortizing, within a relatively short period of time, the overall costs of the instructional program or system. Client's Post Attitude Questionnaire Question 3: What changes in the client's attitude toward instruc- tional development will be revealed in a structured, open-ended questionnaire administered to the client approximately one month after the termination of the experimenter's field experience with the BOTP? Findings.--Two types of data were collected to report the findings 0" tzkiis question. The first type of data emanated from the responses 91\'€3I1 by the client on the open-ended written questionnaire in Appendix E. "'1 S questionnaire was administered to the client during June, approxi— "EiIZGB‘ly one month after the termination of the experimenter's field e O O xDev-ience. The second source of data was direct conversations and associations with the client. An analysis of the responses on the open-ended written question- ha ‘i *‘e shows that the client: (1) Continued to have a positive attitude toward the instructional Clewlopment process. This finding is based on the client's written r~ esDonses to three specific questions. Question 10 asks: "Would you say .tzf‘ial1t '1 £3 =- for the most part, positive, neutral, or negative?" The client's your present attitude toward the instructional development process 1" es Donse was that his attitude toward the process was positive. When 8 ked his reason(s) for feeling this way about the process, he responded 91 by saying that "It seems to show me logic and results." The client's positive attitude is further confirmed by his response to Question 7 which probed his present impressions of the instructional development process. Although he did not elaborate in detail, the client exhibited signs of positive behavior by stating, "I want to learn more (about the process) and I want to use it." The client's continued positive atti- tude is also reflected in Question 1 when he notes that he plans to mod 1‘ fy or change the BOTP through, as he states, "Reorganization of the tota 1 course, applying the 10 approach to the extent I am familiar . . . continued development of operational skills and use as example . . re- 9V61 uation of all training objectives and organize and develop as needed." The only indication of anything other than a positive attitude by the C] '3 ent is revealed in Question 2 when he reported that he had not attempted tojconvince others of the value of the process. Part of the 8x131 anation for the client's response to Question 2 may rest with his bej ‘i ef that he would prefer to diffuse the process and the merits of the process to others through the modeling of instructional development pro- celeres or behaviors in his own teaching and through the presentation of s ta tistical and research evidence to support the effectiveness of the Ib‘floczess in improving instruction. (2) Viewed the value of instructional develppment as being a process w why as he wrote on QgestionB, "comitsyou and y‘gives; you account- a bi 1 it ." Failure on the part of the client to elaborate further on this s ta tement exposes its interpretation to different, and perhaps even mis- i . . h t:‘E'Erpretations. Therefore, the experimenter, drawing upon his conversa- ts ons and observations of the client during the process, exercised the 92 prerogative of subjectively evaluating the meaning of the response for purposes of clarification and amplification. On several occasions throughout the development the client indicated that his active role in the design, development, and evaluation of new instructional materials increased his sense of commitment to the process. The value of this type of active involvement in product development and validation had an effect on the client by first, his identifying more closely with the instructional materials developed for use with the BOTP instructional Pr‘Og ram, and secondly, his being more sensitive to the potential impor- tance of the materials to the learning activities of the BOTP. More Simply stated, it is quite natural for an individual such as a client to take extra steps to insure the effective classroom use of instructional materials for which he had a direct responsibility in designing, develop- ing s and evaluating, than if the materials had been designed externally and independently of him. Also, it became evident to the experimenter early during the process, tha t the client was impressed by the accountability concept of the syn- theSized model. As a Director of Training, he is personally accountable to his employer(s) to provide visible proof that the instructional develop- me" 1: was improving the quality of the BOTP program. This type of visible proof is especially needed to justify significant changes made in the b ho gram. (3) Felt that the most difficult.aspects of the instructhnal de- v ,3 went process. for him (Question 6) were "teachimoncepts and ppinci- 9‘! es " and "pin—pointing objectives." When asked why teaching concepts 93 were difficult for him to comprehend and/or execute, the client could offer no explanation on the questionnaire. In regard to "pin-pointing objectives," the client highlighted the problem by saying that specifying behavioral objectives was difficult "maybe because I don't really know, or accept what the objectives are." Again, to avoid the possibility of misinterpretation by those unfamiliar with the events which transpired during the instructional development, the experimenter has opted to sub- J'ectively interpret the implications of this statement within the proper context. Throughout the entire instructional development process, the client found it very difficult to precisely specify objectives, as many behavioral scientists would expect. However, as reported in the find- "198 under Hypothesis 1, the two specific goals or objectives of the BOTP ““3"<3 identified precisely enough to permit measurability as evidenced t23’ ‘tlliis study. Nonetheless, the experimenter constantly sought to refine the objectives in even more precise terms as the development progressed. some of the suggestions offered by the experimenter to the client in- VO] Ved significant changes in performance standards as well as the con- di tions of performance. It is believed that the client was unable, at that time, to comprehend the long-range impact of such a change, thus IDE‘*"""t:ially explaining his statement that he "did not know, or accept the obj ectives." Furthermore, time constraints and instructional development, 7 "dependent of the other instructors for the BOTP, made it unlikely that 8 ~ 1 Shificant progress in further refinement of the objectives would I terialize during the limited development cycle represented by the field e 3('E’Et'fience. 94 (4) Reflected the frustrations of many instructors who engage in the process within a limited period of time. In Qpestion 4, the client stated that the least effective instructional development change with the BOTP was "probably the fact that very little got on paper." This statement mirrors the obvious frustration of not having instructional development changes reported on paper. Once more, the time constraints and the early emphasis on instructional materials developed seemed to interfere with the revision of a written course syllabus. Another con- tributing factor was the inability of the instructional developer to significantly revise the specific and enabling objectives. Without any further clarification of the specific intent of the objectives, course syllabus revisions would have had little meaning and would be subject to further revisions in subsequent instructional development. In summary, the experimenter believes that the changes would have been reflected "on paper" during a recycling of the process. However, the time limi- tations imposed by the parameters of the field experience was an important factor in prohibiting completion of written revisions. (5) Recognized that the most.significant change (Question 3)_result- ing from the instructional development was within himself.. This finding is supported by the client's response that “Probably the most important change was with myself, recognizing that change should be made, learning the various approaches in making sound changes, understanding (at least somewhat) the” importance of the various steps and their relationship to each other and to the total instructional process, and becoming convinced enough to pursue instructional development activities." 95 Subjective Data Findings.--The second type of data is subjective and based entirely on conversations between the experimenter and the client. The client exhibited signs of a continued positive attitude at the conclusion of the field experience by (1) taking the initiative to con- tact the experimenter for additional explanations of the synthesized model used during the instructional development with the BOTP (several hours were devoted to this purpose); (2) requesting permission (of the experimenter) to photostatically reduce the size of the model from the original art work, for purposes of reproducing additional copies to use in future development in the Highway Traffic Safety Cénter; (3) asking the experimenter to write a summary report of the findings of the study for use in interpreting the achievements of the instructional development to other staff members; (4) considering the possibility of continuing his graduate studies in the Instructional Development program at Michigan State University as a way of broadening his competency base as a training director; and (5) seeking continued assistance from campus instructional developers on problems related to the Highway Traffic Safety Center. In interpreting the attitude of the client after the termination of ‘the field experience, it is important to report a possible "spin-off" effect of the instructional development. During July, the client, as a result of consultations with an instructional developer at the Michigan State University Media Center, decided to apply many of the same strategies ' Uiiéici by the National Special Media Institutes (NSMI) in designing training ins 12‘? tutes. The Instructional Development Institutes (IDI) USOE proposal, 96 in this case, became a model for revising the original proposal of the Highway Safety Patrol for the training of police officers in other aspects of police work besides breathalyzer training. The findings make it very clear that the client's attitude toward instructional development continued to be positive even after the termination of the field experience. On the surface, at least, there would appear to be no significant change from the attitude exhibited during the instructional development. However, the difference which appears to be significant is that the client is now, out of necessity, assuming more personal initiative in program development as evident by the subjective data reported in the findings for this hypothesis. As the need arises, the client initiates contact with other specialists whose talents could be used to provide input into program needs. Thus, it would appear that a client's continued positive attitude toward ID seemed to reflect the client's satisfaction with the student performance improvements which resulted during the instructional develop- ment, and his increased confidence in the potential of the process for improving the instructional process. Client's Attitude Toward ID Question 4. What differences, if any, will there be between the raw score of the client and the grand mean score of nineteen IDI's on the Attitude Toward Instructional Development instrument? Findings.--The client's score (214) on the Attitpge Toward Instruc- tional Development (Appendix D) was slightly less than 2 5.0. above the 97 grand mean score (198.77) reported by the nineteen Instructional Develop- ment Institutes as shown in Figure IV-2. Grand Mean X 198.77 Range = 74-250 Mean Deviation = 7.78 Variance = 97.48 Standard Deviation = 9.88 Client's Score = 214.00 Figure IV-2.--Mean and Raw Score Comparisons Between the Client and the Instructional Development Institutes on the Attitude Toward Instructional Development Instrument. For the client's score to have been 2 5.0. above the grand mean, a score of 218.53 would be needed. Also important is the fact that the client's score was higher than the mean score of all but one of the Instructional Development Institutes (101) which had a mean score of 214.20. Conclpsjons.--There is a difference between the client's score and the grand mean of nineteen Instructional Development Institutes is re- .jected. This conclusion is based on the fact that the client's score was J'ust under 2 5.0. higher than the grand mean of the Institutes and only .2 of a point lower than the highest mean score reported by the Instruc- t‘ional Development Institutes. The analysis of the findings on this hypothesis might lend support to a broader conclusion that attitude toward ID will be more positive for 98 a client who is directly and overtly engaged in the instructional develop- ment process than attitude likely to be exhibited by individuals (or groups) who are exposed to the process in simulated settings. Client's Entry and Exit Cognitive Proficiency Levels of ID‘Process Question 5. What mean score differences, if any, will there be between the client's entry (October, 1971) and exit (June, 1972) cognitive proficiency level of the ID process as measured by the Client'spgpinions Regard- ing Instructional Development instrument? Findings.--The C's score on the Client'spgpinions Regarding_Instruc- tional Development instrument (Appendix E) disclosed an entry performance of 4.54 as opposed to an improved exit performance mean score of 2.75 on twenty-eight observations (for both entry and exit) relating to his understanding and/or his use of the instructional development process. The t-test analysis in Table IV-ll reveals a significant mean score gain on exit performance at the .01 level. Table IV-ll.--Corresponding t-test Analysis of Entry and Exit Scores on Twenty-eight OEservations on the Client's Opinion Regard- ingplnstructional Development Instrument - 2 X 5? tc tT.Ol d.f. Comment .Ehtry 4.54 .023 Significant Difference 3.31 2.58 54 Between Exit and Entry fixit 2.75 .057 at .01 Level. 99 Conclusions.--There is a difference between the cognitive level of the client before and after instructional develOpment. Based on the number ratings explained on the measurement instrument in Appendix E, the client's exit mean score of 2.75 fell into the proficiency category of understanding/or being able to execute the steps of the instructional development process “moderately well" to "a considerable degree" as com- pared to an entry proficiency level almost mid-way between "did not know or understand the process," to knowing it "only partially" as shown by a 4.54 mean score. Also, the client's exit proficiency level of the process was significantly higher (.01) than his entry proficiency level. Therefore, in this study, the proficiency level of the client regarding instructional development was significantly improved as a result of his involvement with the process during development with the BOTP. The importance of the conclusion is that the findings suggest that a client using the instructional development process prescribed by the model may be able to learn the ID processu-without prior explanation of the exact details of the physical model--by actively engaging in the systematic development of instruction. CHAPTER V DISCUSSION W This concluding chapter has four sections: (1) a summary of the development and application of the synthesized model, (2) the major implications of the study, (3) a discussion of heuristics which emanated from the study, and (4) recommendations for further research. Summary of the Development and Application of the Synthesized Model The purpose of this study was to provide a model which could improve the instructional development process and‘thus the effectiveness and/or efficiency of instruction. The study proposed (1) generating an Opera- tional synthesized instructional development model from the review of the related literature, and (2) applying the model to instructional develop- ment with the Breathalyzer Operator Training Program for the State of Michigan. The synthesized operational flowchart model provided a framework for 'fornntive evaluation of the model components while engaging in instruc- trional development. The model was applied to the BOTP for the purpose of reporting consistencies and discrepancies between the model and the actual process used with a client as part of a field experience. The instructional development field experience had both descriptive and experimental components representing the two types of research 100 101 objectives and questions of the study. The first type of objective focused on explicating the process through which systematic instruction was developed. In this study, the systematically developed instruction evolved from the activities prescribed in the synthesized operational model. The second research objective related to experimentally comparing student achievement and measuring the cognitive and affective development and growth of a client. Implications of the Study to Instructional—Development The findings of this study tend to suggest several implications for instructional developers in the field: (1) Instructional development models are effective in improving instruction, in that statistically significant differences favoring the training programs under development were obtained on three of the four dependent measures in BOTP schools under ID from November, 1971 through May, 1972° (2) There is not necessarily a correlation between instructional effectiveness and instructional efficiency in programs which have not reached the stage of using selfuinstructional learning activities. The experimenter found a decrease in instructional efficiency on each of the tJiree criteria used to assess efficiency. However, the criteria used in tJie study were deemed to be more appropriate for use with instructional programs which have reached the stage of using self-instrUctional learning activities. (3) Positive client attitude toward the process can be maintained by DV‘OViding statistical evidence of the effectiveness of ID. On the 102 open-ended questionnaire the client viewed the statistical evidence use- ful in justifying (to his employer) continued involvement in instructional development. (4) The synthesized model was, for the most part, a good representa- tion of the instructional development process. The formative evaluation revealed few discrepancies between the synthesized model and the actual process used during the instructional development with the BOTP. (5) Attitude toward ID is likely to be more positive for clients using the process in the field than for individuals or small groups ex- posed to the process in a formal instructional setting. This conclusion is evidenced by the fact that the client's score on a validated attitude instrument was almost 2 5.0. above the grand mean score of 198.77 for nineteen national Instructional Development Institutes. (6) There is a relationship between the effective diffusion of the ID process to a client and the client's active involvement with the pro- cess. The study reported significant growth (P < .01) in the client's cognitive proficiency despite the fact that the client had never been shown the synthesized model used tn! the experimenter during the ID process. The experimenter credits much of this cognitive proficiency to the fact that the client was actively involved in the design, development and evalu- éition of all learning activities related to the ID. Thus, it appears he ”Viol have become familiar with the steps while engaging in them. (7) Effective instructional development can change people's self- COnCept. The client in the study stated that one of the most significant "eSUIts of the process was the change within himself: a recognition tha t "sound changes should be made in instruction." The client also 103 reported that engaging in the process made him more sensitive to the quality of his own instructional strategies, thereby making him more effective as a classroom instructor. The study still leaves unanswered several important questions relat- ing to the ID process. First, how much of the reported improvement in learning and increase in positive client attitudes were due to the interaction between the experimenter and the client? Secondly, how much of the results of the study was due to the model? Thirdly, how much of the results were due to the experimenter and his personality? And finally, what would be the results if someone (other then the experimenter) used the synthesized operational 10 model in developing a training program bearing essentially the same attributes of the BOTP? :Any determination as to how much each of these factors might have contributed to the results shown in this study will have to be left to future research. Heuristics As a consequence of participating in instructional development with the BOTP, the experimenter learned by successive discovery certain heuris- tics or rules of thumb, which may be useful to instructional developers. Since these heuristics may be of value to those who might apply the in- structional development process or to other researchers in the field, they are presented at this time. Heuristic l: Actively involve the client(s) in all design and developmental activities of program development, e.g., 104 developing instructional materials such as trans- parencies, 35mm slides, etc. The design, development and evaluation of systematic instruction should be a shared responsibility of the instructional developer(s) and the client. The experimenter believes an instructional developer would be remiss if he attempts to do the product design and development inde- pendent of the client. There are two basic purposes served in requiring the client to be directly and actively involved with the instructional development. First, the client learns the process by doing. He learns what varia- bles have to be accommodated during the protess, where to go for special- ized design and production services, and how to evaluate or validate products. Thus the process is more effectively internalized by a client through this active participatory approach. Consequently, the client is likely to become independent of the instructional developer in a shorter period of time than would be the case if the client were not directly in- volved with all aspects of the development. Secondly, the client is more committed to the instructional develop- ment. Instructional materials which are designed and developed represent tangible products of the client's efforts and creativity. Therefore, he has a greater stake in the way the materials are used in the instruction and nay take extra measures to assure their maximum instructional value. Heuristic 2: Advise the client, at the outset of instructional development, of the implications of committing him- self to the process. 105 The experimenter believes it would be unwise for an instructional developer to agree to engage in instructional development, particularly in long range instruction, without first making the client as fully aware as possible of the meaning of this commitment to the process. For example, the client must realize and accept the fact that instructional development takes time, often more time than the client thinks he can devote. The client must be willing to view the process as one which will require on- going design and redesign of teaching/learning activities. Secondly, the client must realize that he will have to become an active member of the design and development team. He should not expect the instructional developer to assume the responsibilities of program development inde- pendent of the client. Thirdly, the client must be in a positpon to commit reasonable financial and human resources to program development. The ultimate success of instructional development will depend heavily on the availability of dollars for making changes in the instructional pro- gram, as well as the complete cooperation of the human components of the system. Heuristic 3: Don't require any more of a client during instruc- tional development than he is ready to give. The experimenter found it useful to work on the basis of agreement With the client on matters related to what should be done, how, when and lflnder what circumstances. The principal role of the instructional developer, particularly during the early stages of development, should be ‘to advise the client at each step of the process. Nevertheless, the 1'15 tructional developer must respect the client's option to accept or rEJEEct the advice. At the same time, the client should be made aware of 106 the possible consequences of his decision(s). For purpose of maintaining a good working relationship with the client, the instructional developer should be willing to undertake the process under the conditions speci- fied by the client. This will have the effect of strengthening the team effort by giving the client a significant voice in the decision making process. For example, the instructional developer recommended that the client be available in the role of content expert during the design planning of instructional materials at the graphics department in the Instructional Media Center. The one time he was unable to keep a scheduled appointment, the client sent a graduate student assigned to the BOTP to represent him during the planning. Consequently, the materials were designed by people who were not going to be teaching with the materials. Subsequently, when the client attempted to use the materials, he found them to be accurate, but not arranged in the exact instructional sequence he would have preferred. Eventually, changes were made which resulted in additional costs for revising the transparencies, as well as the loss in instructional development time. Recommendations for Further Research This study has raised a number of questions which are amenable to further research. These questions may be classified.as: (l) improvements : tSPE: i 3coEcotoa E09320 I 99.9:me I 30350 8: a. a. E Aw fiancee 2533.685 Embed Sauce 5 Snack. 3:? use x8533... 3:253. 02625 «59: «83m m Sen .0 , 5.38 .0 5... :2a .0 a. 3m. Emv Gm. cozoabms 36:82.58 85:03 .3925: 9.185 tco 9.32:5 $38.50 nuance“... one 381.5 BESS .2 838030 _ 2: 53 a 35.585.“ can 282; >552 . 3292 38“ 232m $28030 28m .9 .3 8. 5 E 8.8%...— w0 Samoa 05 new .305. < uk¢<10 30..“— 133 To Briggs, a formally correct, or behavioral, objective means one written to meet the criteria for behavioral objectives identified by Mager (1962). Mager's three formal criteria are: (a) Given what, the; (b) Student does what and; (c) How well (p. 19). Briggs also identified seven specific sources which must be considered by the instructional developer when selecting objectives. They are: (1) local demands by potential employers who may specify what kinds of skills they can use in 'their operations; (2) professional societies in various academic dis- ciplines who, through committees, develop specifications concerning what should be taught in specific subject areas; (3) curriculum-development projects which often take the initiative in changing from old to new content; (4) teaching research in the subject area; (5) students who can provide input regarding the relevance of instructional goals; (6) tradi- tional course content, and (7) policy research centers which are designed to predict new skill patterns adults will need twenty to thirty years hence to operate competently in the changing society (pp. 31-33). The second step of the Briggs model requires the preparation of , valid tests over the objectives. Briggs believes that a test is valid if it measures what it is supposed to measure, or if it measures the ob- jective for which it is intended (p. 48). In reference to the third stage of his model, Briggs identifies three steps in analyzing the structure of the objective. First, identify subordinate competencies for an objective by asking "What would the learner have to be able to do or to know before he can perform his entire objective, given only instructions as to what he is 134 to do on a test over the objective?" (p. 74). Secondly, identify types of learning as defined by Gagne's Eight Types of Learning (See Briggs, pp. 82-84). The third step is to number boxes for teaching sequences. To do this, you simply start at the bottom and number them in order in which they are to be taught (p. 80). Step 4 requires the designer to identify the learning experiences or skills the student must have already mastered before he will be able to achieve the course objectives. The last step of the design stage (Step 5) for the Briggs Model is to state a plan for dealing with learners who lack the entering competencies. Once these learners are identified using pre-tests, remedial action as well as the three alternatives indicated in Steps 5a, 5b, and 5c of the model can be used to meet the needs of these learners. In deciding how to produce the desired event, the designer, accord- ing to Briggs, must think of the kind of stimulus necessary to produce it: natural objects; spoken words; printed words; theoretical objects or processes described or represented symbolically or in animation; processes (objects in motion); social stimuli (group interaction); etc. Then select a medium (Step 6) which has the right characteristics for presenting the desired kind of stimuli (p. 98). Briggs defines prescriptions as "directions on how the materials are to be developed for each continuous use of the media chosen" (p. 129). Such precriptions include directions to the film maker or other specialist who is to prepare first-draft materials. They Specify the content, as well as the programming techniques to be employed in the way the content. is to be prepared. For example, the designer may prescribe "dissolves" 135 for a film (p. 129). For anyone interested in examining this technique in depth, there is a completed example of a media analysis on p. 137. of Briggs. Step 7 in the Briggs model is the first step in the development phase Of instructional design. Here, the designer develops the prescrip- tions into draft materials, using as much creativity and ingenuity as possible in developing the content for each instructional item. Particular importance is placed on presenting content, posing questions and problems, evoking responses, and providing feedback to the learner and formative evaluation data to the instructor on where a learner has trouble and what the trouble is (p. 164). Briggs offers a list of thirteen suggestions which may be helpful in preparing first-draft materials (pp. 164-165). Steps 8, 9, and 10 of the Briggs paradigm shows the formative evalu- ation stage of the development phase. Formative evaluation refers to the practice of conducting try—outs of draft materials with individuals and groups of learners, followed by evaluation tests, to provide an empirical assessment of materials and to identify needed revisions. It also re- quires the use of performance tests (empirical data) for making decisions long before the draft materials are ready for try-out. The model uses a feedback loop to connect input data for Stepa 8, 9, and 10 for redevelop- ment of materials, if needed. If the results of the performance tests are below expectations, the Briggs Model assumes that the designer will "loop" back either to Step 7 where he might reexamine the first-draft materials or loop all the way back to Step 1 to reconsider the objectives 136 in light of the performance results and proceed, if necessary. to go through the whole instructional sequence again. In conclusion, Briggs suggests that input as a result of data from graduate students, in advanced courses or on the job as possible sources ‘ of data for course revisions during the development stage. The main strength of the Briggs Model is the thorough and compre- hensive narrative description given to explain the input requirements of each step, stage, or phase of the flowchart model. No other instruc- tional model seems to describe the process as completely as Briggs does in his model. If it has a weakness, it may be that it is not a practical operational model for a developer to use with the expectation of easily diffusing the process to others. The lengthy narrative description (some 200 pages) tends to be overwhelming to professional developers, thus discouraging widespread adoption of the model by design or subject matter specialists. Like so many other models, it too implies a linear process. The GustafsonfiModel.--Gustafson (1971) proposed a variation, as shown in Figure A-lz, of the IDI Model. The basic difference lies with the emphasis it attempts to place on the fact that the instructional development process is a dynamic, non-linear process which may not necessarily begin with the definition of the problem or the specification of behavioral objectives (pp. 2-6). He summarizes the philosophy behind the model as follows: First, there is no beginning or end (or at least there shouldn't be). To commence ID activities should not suggest the beginning of the system. for at least part of it predates the developer's initial effort. Further ID should not have an end since whatever is developed must be continuously reexamined to determine its efficiency. effectiveness and relevance. Another, 137 TIME DEFINE PROBLEM I OPPORTUNITY DEFINE RESOURCES MT I DEFINE SETTING I I I V— E ' DEFINE l OBJECTIVES I , IT DEFINE PERFORMANCE l MEASURES ‘, i SPECIFY INSTRUCTIONAL STRATEGIES LT T T T T" I CONSTRUCT I PROTOTYPE TEST I ANALY'ZE PROTOTYPE I . IMPLEMENT / F RECYCLE TV T T T T Figure A-lZ.--An ID Model, K. Gustafson, Michigan State University, 1971. 138 systems attribute is the interdependence of the elements of the system. Anyone who attempts ID rapidly finds out that individual elements cannot be singled out for individual attention since their explication depends on information, decisions and conse- quences occurring within other elements. (Pp. 2-3) Gustafson isolates, in all, four significant attributes of instruc- tional development: (1) time; (2) interdependence of the elements of the system; (3) information flow bgtgggg_elements3 and (4) information flow amggg_elements of the process. In regard to the time factor, he describes the ID process as time-consuming and cites Bachrach's Law "that things take longer than they do" to illustrate the patience re- quired while engaging in the proCess (pp. 2-3). 0f critical importance to Gustafson is the cybernetic concept of feedback as a means of accommodating information transfer from one element to another. He states that "information must flow in both directions between elements and often among elements simultaneously" (Gustafson, 1971, p. 4). In his opinion there is no more important element to consider when planning an ID project than designing, maintain- ing and redesigning the information transfer network within the system and with its external interfaces. This model stresses the importance of human factors in assuring the success of an instructional development effort. Gustafson believes that "without doubt the most important element of the ID system is people . . . to engage in ID is to change people" (p. 6). Ignoring the people one serves during instructional development frequently will re- sult in what he calls "ID casualties" in that they are proud of the product(s) developed but do not wish to go through the ID process again. This is the obvious danger of ignoring the human factor. 139 The instructional development functions in the Gustafson Model are very similar to the functions of the IDI Model described earlier in the study. Therefore, Gustafson's main contribution to model-building theory would appear to be his emphasis on the human factors in instructional development and the non-linearity depictation and description of the process. The Gerlach-Ely Model.--This model (1971) attempts to graphically portray an instructional which Gerlach and Ely say is "a guideline-va road map--and should be used as a checklist in planning for teaching“ (p. 12). They go on to note that "it shows the major components of the total teaching-learning system, even though it does not portray the fine details of each component" (p. 12). However, it does shew the relation- ship of one element to another, and offers a sequential pattern which‘ can be followed in developing a plan for teaching. The interesting feature of this model (Figure A-l3) is the fact that the content and the objectives are specified or identified before any serious consideration is given to the entry skills possessed by those who are the recipients of the instruction. At this stage the critical factor is the development of behavioral objectives for the content matter which can be precisely measured in terms of student performance upon completion of instruction. Once the instructional designer has attended to these prerequisites he gives consideration to his target audience. The teacher needs to know what each student brings to the course as it begins. Gerlach and Ely contend that "Unless the teacher knows the extent and sophistication of what the students know he must plan his course for an 140 Determination of ._ STRATEGY Organization Specilication _i oi ' 4 ol 'fi GROUPS CONTENT ________ _ __ Assesfsment Allocation Evaluation 0 ENTERING 4— 0' a 0' { BEHAVIORS TIME PERFORMANCE Specification "" “““““““ 0‘ “‘1 Allocation OBJECTIVES _i of “'1’ Selection L— of _i RESOURCES Analysis of FEEDBACK Figure A-13.--A Systematic Approach to Instruction (Gerlach and Ely, 1971, p. l3). 141 average student" (Gerlach and Ely, 197l. p. 14). Two sources for assess- ing the entry level of students are cited: (l0 use of available records, and (2) teacher-designed pre-tests. Insofar as the available records are concerned the student's cumulative record will probably include the results of several standard- ized tests which he has taken. These tests, according to Gerlach and Ely, would reveal valuable information about the student's level of in- telligence, his personality traits, and his potential (p. l5). Course grades also indicate potential as revealed by his performance in courses during his school career. A properly designed teacher pretest considers: (l) the student's achievement in the subject to be pursued, and (2) the student's ability to define basic terms in the subject area. It also serves as a checklist on previous learning and is aimed at the funda- mental question which must be answered prior to formal instruction: "To what extent has the student learned the terms. concepts and skills which are part of the course?" (Gerlach and Ely, l97l, p. 15). The next step is to determine the instructional strategy. or method. for using information. selecting resources, and defining the role of students. Two methods are suggested at this point. The first. exposi- tion, the more traditional approach, is one in which the teacher presents information to the student. using such vehicles as lectures, discussions. textbooks, audio-visual materials, student reports, and the teacher's personal experience to present the course information. The second approach is classified as inquiry. Using inquiry the teacher “assumes the role of the facilitator of learning experiences and arranges condi- tions in such a manner that students raise questions about a topic or 142 event" (Gerlach and Ely, 1971, p. l5). The teacher's role is essentially one of helping the.student to be active participants in developing hypothe- ses which can later be tested by use of additional data. Technique refers to the procedures and practices used to accomplish teaching objec- tives, regardless of approach. Examples of techniques are lecture, dis- cussion, audiovisual presentations, and verbal and written reports prepared by students (Gerlach and Ely, l97l, p. 17). The next logical step in the Gerlach-Ely Model is to organize the students into groups for purposes of effective and efficient instruction. To accomplish this. three basic questions must be answered: (l) which objectives can be reached by the learner on his own?: (2) which objectives can be achieved through interaction among the learners themselves, and (3) which objectives can be achieved through formal presentation by the teacher and through interaction between the learner and the teacher? (Gerlach and Ely, l97l, p. 17). The plan for allocating time for the instruction will usually vary according to the subject matter, defined objectives, space available, administrative patterns, and the abilities and interests of the students. However, the teaching plan must take into account the estimated time for completing each event in the teaching strategies and techniques. The allocation of learning spaces is also based on learning objec- tives and the same three basic questions which must be answered in regard to organizing students into groups. Gerlach and Ely identify four formats. of learning spaces. They are: (l) the traditional classroom equipped with thirty to forty student desks, arranged in rows with a teacher's 143 desk at the front of the room as the focal point, and built-in teaching tools such as a chalkboard and a bulletin board; (2) large-group spaces to accommodate groups of various sizes from 60 to 300 depending upon the subject matter and the grade level; (3) small group spaces with movable walls which permit the conversion of standard classrooms into several small spaces in a minimum of time, and (4) independent study spaces equipped with audio and visual for individual rather than group instruc- tion. The final step in the Gerlach-Ely paradigm before undertaking an evaluation of the instructional program to determine its effectiveness is the selection of resources. These resources can be classified into five general categories: (l) real materials and people; (2) visual materials for projection; (3) audio materials; (4) printed materials, and (5) display materials. Evaluation of performance is one of the last elements of the model, but it should be one of the first concerns of a teacher. Some objectives are simple to evaluate. If they are cognitive, observable, and measur- able, there is no difficulty. The real difficulty comes in measuring objectives which are much more complex and fall essentially in the effec- tive domain of learning experiences. However, there are at least two dimensions to evaluation assessments. Gerlach and Ely quote Glaser (1965) who points out: One is to provide information about a student's present be- havior; measurement for this purpose is primarily designed to «_ discriminate between individuals. The second use is to provide information about the instructional techniques which produced that behavior: measurement for this purpose is designed to dis- criminate between instructional methods (pp. 27-28). 144 The terminal step before recycling, if necessary, is to analyze feedback from the students. This analysis may vary in sophistication from simple observing the student's physical reactions to information presented to a formal feedback analysis from measuring instruments designed to measure whether or not the objective(s) was indeed achieved. In discussing the value of feedback Gerlach and Ely state: It is important for feedback to occur as soon as possible after a response has been made. Not only is the feedback to the teacher valuable, but the teacher's feedback to the pupils is supporting. If the student's reSponse is correct, the teacher should confirm. it. Research indicated clearly that such practice facilitates learning. Delay in feedback decreases its effect. . . . The student knows immediately whether his response is correct. . . . He does not have to wait for his pa er to be corrected. The feed- back is almost instantaneous (p. 30) Finally, after analyzing the feedback, the Gerlach-Ely model brings the instructional developer back to the specification of objectives stage for a reexamination of the original objectives and possibly even the content of the course, The limitations of the Gerlach-Ely model are: (1) it relies too heavily on a narrative description to explain the functions of the skeletal graphic model; (2) it implies linearity in the process, and (3) it is not designed with the view that a team of developers would be assisting the teacher during development. The Douglas Model.--Douglas (197l) describes his model as "an operational plan for Instructional Development within a given institu- tion" (p. 46). Specifically, the model (Figure A-14) was developed for ' use with staff at Burlington County College in Pemberton, New Jersey. Like so many other models, it is comprised of three phases or parts. .23: 3%: 28:22.8 528235 2.98 2:...2& 95?. 145 ‘88-- 181.. {I III!!! ll] .1...- Illa-I11! I!!! Illa-ulna...- ilsI-l gill it; flit-ill! lull-la 33:13.... Blue-I3...- IIIIIJII 1.113.}... Ila-.89...- 1!!! I... I... It! III-II. Illa a Ill-sill... If}. Isl: 81.25... in! 8.88.8! .188!!! iii] ‘8. lat-o. Iain: all!!! Sal-.1832... .383. a!!! all 1.3.3.8381! iii ‘i' 3.31 {i «82.!!! III-.- 88.81 glut-II I“!!! is). 0’; ‘8‘ I]: lull 1.11:5 Bus-n. ail-la... Ilsa-sic ,...-III] a ill-nu! lie-Ii 3.5.1.8.... 1.8!... all..." Ill-3! 8...: Ill-Jill.- lilt- pnltltzlt I'll-ll 1"! ill: I} :3 I'll." It 1.8.. 38.....- III-II urn-III. its]: III!!- llaica luau-Ila In!!! a in!!! III-.33.. I! g Ill-I I: Isl ‘I 8.58... .II. all: all: II. I]. as... I! ll. [-8 1i i s s . a s . n . _ n r I .l I .l il Ir lllll L. llllll Li IIIIIIIIIIIII L 146 Part I is concerned with the functional level analysis of what a teacher should do when offering a course of study to his students. It starts with an analysis of student needs, dealing with such factors as: (l) is the content relevant for the student, (2) are societal needs being met by the course, and (3) content requirements (p. 48). The key design element is represented in writing measurable learning objectives and test items before designing and implementing teaching and learning strategies. Douglas believes that, "Evaluation is one of the most crucial parts of the whole procedure" (p. 49). Evaluation has two main purposes: (1) evaluate the student learning as it occurs, and (2) evalu- ate the effectiveness of the instructional design. The evaluation is an on-going instructional development activity. The complexity and the specificity of the Douglas Model become apparent as one examines the three phases of the basic model format just described. Douglas describes the phases as: The steps in each of the phases relate directly back to the functions outlines in Part Ohe.' This means that each time an instructor proceeds through one of the phases of instructional development, he will be repeating the same functions, out at a more complex level of sophistication. In this regard, instruc- tional development must be considered a cyclic, spiraling phenomenon, in which each cycle is based upon the previous cycle, but is distinct in terms of complexity and exactness. (p. 49) A Phase One project is usually designed with only one instructional track. Instructors may use different instructional modes, but every student in the course will generally perform the same learning activities in reaching the objectives of a course. Evaluation of a Phase One pro- ject usually centers on such factors as drOp-out rate and grade distribu- tion, but an equally important factor is how well the students mastered 147 each of the stated objectives. This analysis furnishes the foundation for deciding to repeat Phase One the next time the course is offered or to proceed to a Phase Two instructional development project (Douglas, 197l, p. 49). Phase Two is a much more precise, experimental type of instruc- tional development. It is characterized by the concept of validation. Each step is predicated upon the idea that the learning experiences provided should prove themselves to be valid when they are carefully reviewed. Each part should function adequately to insure that acceptable levels of learning are being attained (Douglas, 197l, p. 49). This level is different from Phase One in that the instructional development projects are of a comprehensive course syllabus and a multiple track instructional design. The syllabus is a very explicit statement of knowledges, skills, and attitudes which are to be developed during the course of study. It is equally explicit as to how the course is organ- ized and the activities which a student must accomplish in order to complete the course. As in Phase One a decision must be made at the end of the evaluation stage as to whether to modify and repeat the project, with the other option being to move to a Phase Three project or imple- ment as is. From the standpoint of time it is estimated that Phase Two projects take up to a year to complete. Phase Three is a highly experimental procedure in which the entire development process is completely reviewed, and research and ekperimental techniques are utilized at the application level. Learning needs are assembled in terms of institutional philosophy and goals, appropriate 148 content, student input, and societal impact. Validated course goals are generated which take each of these factors into consideration (Douglas, 1971, p. 50). Whereas, Phase One and Phase Two deal with more immediate instructional needs, Phase Three projects are definitely long-range propo- sitions. They may take two to three years to complete. This model represents a new concept in the model-building theoretical literature in that it presents alternate and systematic strategies for developing curriculum over an immediate, intermediate, and long-range basis. Its only apparent weakness is that it depicts only the what to do and not the how to do functions in the process. The Kemngode1.--And finally, Kemp (1971) devised a plan for instruc- tional design which consists of eight discrete steps: (l) List topics, stating general purpose for each one; (2) Enumerate the important char- acteristics of the student group for which the instructional will be designated; (3) Specify the learning objectives to be achieved in terms of measurable student behavioral outcomes; (4) List the subject content that supports each objective; (5) Develop pretests to determine the stu- dent's background and the present level of knowledge about the topic; (6) Select teaching/learning activities and the necessary instructional resources that will treat the subject content to achieve the objectives; (7) Coordinate such support needs as budget, personnel, facilities, equipment, and schedules to carry out the instructional plan, and (8) Evaluate student learning in terms of the accomplishment of the ob- jectives, with a view to revising and reevaluating any phases of the plan that needs improvement (p. 9). Upon completion of the evaluation stage, 149 the designer recycles back to the appropriate step list in the model represented in Figure A-l5. Kemp's process is somewhat inadequate as an operational flowchart instructional design model. Without the narrative descriptions (of the steps) which accompanies the model, an instructional developer would be hard pressed to execute the sequential functions identified and depicted linearly in the model in Figure A-lS. Topics and General Purposes i Student Churncrerisrics i Learning Objectives i 7 Subiect Content i Pre-Test i _* Teaching/[earning Support Actrvrtres and Resources Services i Evaluation Revise Figure A-l5.--Instructional Design Model (Kemp. 1971. P- 10)- Curriculum and Psychological TeachingiModels Gagne Model.--Gagné (1962) devised a model which plans for the human components in a system development. It divides systems develop- ment into three principal stages: the design stage, the development stage, and the system testing and operation stage. These stages are graphically displayed in Figure A-l6. 150 SYSTEM DEVELOP“ Statement of sister-i purposes Advanced design; operations design Assignment of functions to man and machines MACHINES DESIGN STAGE Task description Task nelysis .16 but. DEVE LOPMEHT 1mm STAGE Training devices hm:- m Convicted components ————————————— Tea- m TESTING STAGE System training Sut- evaluatton OPERATIONAL STAGE Figure A-16.--Procedures Used in the Development of Human Components of Systems (Gagné, 1962). Systems design begins with a statement of purposes for the system, one or more "missions" the system is expected to perform. The purposes set the stage for the derivation of what the system's characteristics will be. Before going any further, systematic plans must be made for how the system is to work, and this means not only that the machines must be conceived functionally, but that there must also be a design for operations. Operations are prospective events that human beings 151 do with and to machines. At this stage wise decisions must be made re- garding the functions of the subsystems, the major parts of the total system, and the ways in which they may be connected together to fulfill the system goals. Along with these decisions, some highly important . judgments are made with regard to human beings (Gagne, 1962, p. 3). Assuming that wise decisions have been made at this early stage of planning, the process of design and development is now ready to follow two parallel paths: machine components and human components, both of which interact at many points as development proceeds (Gagné, 1962, p. 5). Once the purpose and function of a subsystem has been stated, the designer of the human components can then proceed to describe in specific terms the nature of the human functions. This job is the task descrip- tion, whose basic role is to provide the kind of information to which all subsequent plans for human beings in the system must constantly be referred. These are the statements which specify exactly what it is that the man-machine combinations comprising the subsystem are doing (Gagné, 1962, p. 5). Task descriptions lead to the two activities that underlie the rest of the designer's work: to design jobs and to undertake the task analy- sis which makes possible decisions about the techniques to be used in .achieving the human behavior required for these jobs. In regard to designing jobs, consideration has to be given to the number of tasks, their length, and their physical location within the subsystem. The analysis of task is undertaken to determine to what extent each kind of 152 human behavior required can be achieved with the use of the various techniques available: by providing job aids (job "supports"), by selection, by training (Gagné, 1962, p. 5). Finally, according to Gagné, the designer of the human components has to ask some basic questions. For example, can the required behaviors be achieved by providing job aids to facilitate human performance? Can they be obtained by selection of people with the right kinds of fundamental abilities? (h-to what extent must the capabilities needed be established by training? (Gagné, 1962, p. 5). During the development stage Gagné identifies several procedures. Job aids can be developed to provide for storage of information beyond the capacity of the human memory, or to serve as external cues for the instigation of behavior required in systems tasks. Most commonly, these additions take the form of checklists and instructional manuals. Aptitude tests can be developed or chosen from existing stock to measure the basic abilities that have been identified with a program of personnel selection and classification. And procedures can be designed for indi- vidual training, based upon psychological principles of learning relevant to the kinds of performances needed (Gagné, 1962, p. 6). A variety of purposes must be served by procedures of training: (1) individual training which pertains to the performance of a man in relation to a machine or to a set of tools, (2) team training designed to train men to communicate with others in ways which will bring about the most efficient attainment of system goals under a wide variety of condi- tions, (3) system training which focuses on the idea of having human 153 beings acquire and refine their competencies in interactive and communi- cative techniques, while considering the system as a whole. Training devices have an important role to play at this stage. They not only establish the specific skills of machine operation, but also, in the form of simulators, for the conduct of team training as well as systems training (Gagné, 1962, p. 7). The final consideration at the developmental stage are performance measures. A means must be provided to measure the results of training-- to determine whether the desired capability has in fact been established. In the Gagné model two other characteristics should be noted. The first is the fact that interaction between the lines of development for machine and for man must occur all along the way, and secondly, "testing" throughout every stage of development. Gagné notes that these two characteristics are often referred to as the "human engineering evalua- tion" (pp. 7-8). Once the design and the development are complete the system is ready for implementation and subsequent evaluation. The testing stage provides the data needed to make decisions regarding any desirable revisions. It is essential to know what it is that human beings are supposed to do, even if they are highly skilled. This means that standards of human performance and measures of human performance must enter crucially into the decisions that are made during the testing of a system. The final function of a systems development is the human function. Gagné describes the human function as "varieties of transformations which the human being, considered as a systems component, performs upon inputs 154 to produce outputs" (p. 53). Once they have been identified and described by considering a typical unitary response system, composed of a human being who is presented with an equipment array and a set of controls that include manipulable buttons and knobs, the human function becomes one of transmission of information into the system. The flowchart for the model is of limited operational value to an instructional developer without the elaborate narrative description Gagné provides to explain each component. Also, the model, like so many others, is characterized by a definite process linearity. The primary contribu- tion of the model is the emphasis on the human factors and their relation- ship to the machine components of a system. DeCecCo TeachinglMode1.--On the assumption that the best substitute for a theory of teaching is a model of teaching, DeCecco (1968) modified a stripped-down teaching model developed earlier by Glaser (1962). The DeCecco model divided the teaching process into four uncomplicated com- ponents which conceptualize the teaching process (p. 11). The model is graphically depicted in Figure A-l7. A w B c 0 Instructional Entering _J Instructional _. Performance Objectives Behavior Procedures Assessment l T' 1 w_.._ Figure A-l7.--A Teaching Model (DeCecco, 1968). Instructional objectives (Box A) are those the student should attain by completion of a segment of instruction. In theory, objectives can 155 vary in scope and character from the mastery of a spelling list to the acquisition of Greek virtue. DeCecco uses Mager's criteria for writing good behavioral objectives. Entering behavior (Box B) describes the student's level before instruction begins. It refers to what he has previously learned, his intellectual ability and development, his moti- vational state, and certain social and cultural determinants of his learning ability. Although the model gives priority to the selection of instructional objectives over the assessment of entering behavior, in practice these two components must interact (p. 12). Instructional pro- cedures (Box C) describe the teaching process; most decisions a teacher makes are on these procedures. DeCecco contends that instructional procedures must vary with the instructional objectives. Also, instruc- tional procedures must vary depending on whether the teacher is teaching skills, language, concepts, principles, or problem solving. A complete strategy is presented for dealing with each type of learning activity in the narrative description of the model. Finally, performance assessment (Box D) consists of the tests and observations used to determine how well the student has achieved the instructional objective. If perform- ance assessment indicates that the student has fallen short of mastery or some lesser standard of achievement, one or all of the preceding components of the basic teaching model may require adjustment. The feed- back loop shows how the information provided by performance assessment feeds back to each component (DeCecco, 1968, p. 12). The DeCecco Model is of doubtful value as an instructional develop- ment model. Perhaps its best use is as a model which delineates one 156 substep of a total instructional development model, i.e., the teaching process. GMI Model.--The General Model of Instruction (1965) is a procedural guide for designing and conducting instruction. The model is, as others claim it to be, applicable to all levels of education (e.g., elementary, secondary, higher), all subject matters (e.g., English, science, art, vocational), and any length of instructional unit (e.g., one hour, one week, one semester) (Kibler, Barker and Miles, 1970, p. 2). According to Kibler et al., the major philosophical premise under- lying the model is: The goal of instruction is to maximize the efficiency with which students achieve specified objectives. The model is based on a technology of instruction which has developed in the past several years from the research and development in three areas-- experimental psychology, military training, and programmed instruction (p. 2). The three individuals who have contributed most to the development of the GUI Model are Robert Gagne (1965), Robert Glaser (1965), and James P0pham (1965) (p. 2). The two major functions of the model are (l) to guide instructional designers and teachers through the major steps in designing and carrying out instruction; and (2) to provide an overall structure with which to view and study the instructional process (Kibler, Barker and Miles, 1970, p. 2). A flowchart diagram of the model is shown in Figure A-lB. The model presupposes that the function of behavioral objectives is for planning instruction, not for informing others of instructional intentions. The selection of appropriate objectives usually is based on 157 _Tf Instructional Pre- Instructional Objectives Assessment “J Procedures "’ Evaluation I _' I I“ _._' Figure A-lB.--A General Model of Instruction (Kibler, Barker and Miles, 1970, p. 3). the following factors: (1) what the students are able to do before be- ginning the unit; (2) what the student should be able to do in instruc- tional units that follow the unit of concern, and what they should be able to do after completing their education; and (3) the available instructional resources, including the instructor's capabilities with his subject matter. Daring the selection the classification taxonomies of Bloom, Krathwohl, Gagne, Guilford and other are applied to determine the level or type of human performance desired. Once a set of objec- tives has been selected, the instructor should perform a behavioral analy- sis in which he determines what a student will do to demonstrate achieve- ment of the objectives. The actual components to be examined in a behavioral analysis are: (l) the important stimuli to which a student responds; (2) the important responses made; and (3) the criteria which the responses must meet to be considered successful. Such an analysis can be performed by observing students who have already achieved the Objectives as they exhibit the desired behaviors. Previous students can be interviewed, and the products (tests, paper, etc.) they produced can be examined (Kibler, Barker and Miles, 1970, p. 4). 158 In specifying behavioral objectives under the GMI model the three elements recommended by Robert Mager in "Preparing Instructional Objec~ tives" (1962) are used: (1) observable behavior; (2) conditions under which the student will be expected to demonstrate achievement of the objective; and (3) criteria for evaluating the success of the student's performance. Prior to beginning a unit of instruction, it is desirable to deter- mine (1) how much of what is to be learned in the unit they already know; (2) whether they have the necessary behavioral capabilities for the instruction to follow; and (3) the instructional activities that should be prescribed for each student. Of course, the assessment should be based on the specific instructional objectives specified for the unit. The results of this assessment should indicate (1) whether any students may omit any of the objectives of the unit; (2) whether any students should be required to master prerequisite skills before beginning the unit; and (3) what specific instructional activities should be provided for specific students (Kibler, Barker and Miles, 1970, pp. 6-7). The design of instructional procedures involves selection of avail- able instructional materials, preparing new instructional materials when necessary, and developing a sequential plan which appears to be the most efficient for achieving the stated objectives. Decisions should be based upon research evidence when it is available. At this third stage of development, the model list ten generaliza- tions, or principles, based on research evidence which should be con- sulted in designing instructional activities. They include: 159 (l) pre-learning preparation of the student; (2) motivation; (3) provid- ing a model of terminal performance; (4) active responding by student; (5) guidance by the instructor; (6) practice using newly learned be- haviors; (7) frequent and prompt knowledge of student responses; (8) graduated sequencing of instruction from the simple to the complex, from the familiar to the unfamiliar; (9) accommodations for individual differences in students; and (ID) classroOm teaching performance skills in stimulating interest, explaining, guiding, identifying and adminiS+~ tering reinforcers, and managing classroom behavior (Kibler, Barker and Miles, 1970. pp. 8-9). When students complete an instructional unit, they are evaluated to determine whether the instruction was successful in achieving the unit's objectives.. Typically, evaluation involves using tests and instruments to measure the acquisition of knowledge, skills, and attitudes. Frequently, it is necessary to specify or describe student achievement. Changes in the objectives, the pre-instruction evaluation procedures, the instruction, or the post-instruction evaluation are to be made on the basis of the evaluational results (note the feedback loop on the flow- chart). In addition to making changes based on observed results, instruc- tors should make modifications on the basis of new developments in materials and techniques, new research findings, and changing values (Kibler, Barker and Miles, 1970, pp. 13-14). The results of evaluation also can be used to inform students and other interested parties regarding the degree of success each student achieved in the unit. However, since all students may be required to 160 master all the objectives, this information may consist of only an indi- cation of the different lengths of time each student took to complete the unit (Kibler, Barker and Miles, 1970, p. l4). The General Model of Instruction is graphically uncomplicated and therefore useful as a general decision-making guide in planning instruc- tion. A major contribution of the model is the list of ten generaliza- tions or principles regarding effective instruction. CER Mode1.--Alexander and Yelon (1969) designed an instructional system which involves going through the series of stages shown in FigUre A-l9. I, { feedback analyze specify evaluate design and test current , system .__S.Ialternative__,,develop ‘ itsystem system objectives procedures system (or sgb- parts .. T— T Figure A-19.--Stages in the Instructional System Design Process of the CER Model (Alexander and Yelon, 1969, p. 45). recycle -———J The modei, although conventiOnal in many respects, makes a signifi- cant contribution to model-building theory. It is the contention of Alexander and Yelon, for example, that the model should serve only as Common Experiential Referent (CER) for the development of mutually 161 acceptable strategy by which a design team can undertake instructional development. The model assumes that each member of the instructional design team enters into the team effort with different backgrounds and different ways of approaching the task. Furthermore, Alexander and Yelon believe that "often each employs a different vocabulary, or tech- nical language, derived from his particular area of training or compe- tence, which also impeded communications" (p. 44). Thus, the purpose of the CER is to facilitate communication among the members of the design team, speed up the design, development and productive process, and to increase mutual satisfaction of the team members with the ultimate product (Alexander and Yelon, 1969, p. 44). In essence, the model serves, as Alexander and Yelon insist, as a springboard for model acceptance or revision by team members, thereby gaining greater under- standing of and commitment to the process by the members than would other- wise be possible. The strategies related to gaining a commitment to the process from the design team members is a valuable feature of the CER Model. The limitations of the model are much the same limitations recognized with many of the earlier models reported in the study: (1) implied linearity of the process; and (2) limited operational value, other than as a strategy for obtaining agreement from the outset of the instructional development. Models Developed by Training_Managers Tracey Model.--Tracey, Flynn and Legere (1968) applied the systems approach to the improvement of vocational education in secondary schools. 162 Their systems model is flowcharted in Figure A-20 and shows a "closed loop" system which is self-correcting and contains fifteen steps. Their model is a modification of the MINERVA Model (1967) which was developed as a United States Army instructional systems program to analyze and renovate the total training efforts of the United States Army Security Training Center and School. The cycle starts by analyzing market needs and ends by evaluating the student after graduation in a continuous process of evaluation and revision. There are three major phases in the cycle: (1) Determination of. systems requirements; (2) System development; and (3) System validation. In determining the system's requirements several types of input information will assist the designer(s) in establishing the goals and functions of the vocational training program. Since the specific mission of a vocational school is to produce the kind of Skilled workers needed nearby, the logical starting point is to analyze the consumer market. An analysis of local needs should embrace (1) firms within commuting distance of the school, and (2) skills needed by industries which the community hopes to attract. The analysis must answer several questions: (a) What skills are required? (b) Where do the skilled workers come from now? (c) How well-trained are these workers? and (d) Will public and private organizations cooperate in developing new educational pro- grams? (Tracey, Flynn and Legere, 1968, p. 19). Secondly, if vocational programs are to do their jobs, they must be . based on an analysis of job requirements, not what someone recalls that 163 .momp .quoz ecemmd uca :=»_m .xuomehui.o~i< meamwu now-anmuoflli maiaozomu .. In. .1111Ji.-lil flit- “C-— .I'SLflnu- -. I. J . aunonuuqacomm meanwnuum Aueueuuub «It. bYl ii: ean_ A 1y 33:. A. - __ . A I on>auc< a . .m noauoauumcH.IIi coquuauuuau «II. nouuaaaevu A . riiiiiuniiiiit riiiiiiiiiiiiv ii «weapons: It ”adequosuuucnwwi «vacuum A L A amounuum * ucouaoo wlxmweaoauuauumcu _ onusou smte 9 uuoaom M uooaomig . a r The . fii (II . H uumuH mopwuuofipo U ouameomuom managemuom. m uoauumcou oumHnauomm non uuuddoo_ rilllolu i... -11 ...... ZMHmwm omHzH¢MH mun UZHH xuhmwm QZHMH<¢H umh qumoqupwo MquxumHDoum UZHZH.u--.e_m-< weaned @ i A ..: o.w 0.0 E986 mzsaomm 838mm maémmo 2359 0555? maHHUNHmo 92¢ onmmHz zmemwm :mHAm8ii.£~i< 953... ~.n c.@ «.0 .382 95:43. 2»: _ $.58 EB .. D - . i ., w c.e.o n.c.o _ . 3.58 28.5.8 . :33: amusing?— _ a Mifl «J... 1... m Tn... ”.958 3.55 4. ~62? ,_ 505:. hwy—tux.— .333 . iii .4 . ~38 35:52: A .q 4A6 . 5.3.55 2.58.5 Eng»... H .o 2385. 333.5» n a o in... «53:5 5.2.8 9.3.3.. .11 52.523...— 5: ~ 83.5.8 . . , . I'J . ~ a o u a... N a $8533. 32.8»..8 2e35, ”ECU SOnhUn—flhmlu Ua .— .r. :54 G HIM E—O twat” ”hammo nausea soda... 4 z » ages: 25:5 .8325: _ \ . n oJ n. a a.“ © © 171 6.! counts mutt: W use man: comes gnu“; man roman tum-c noctu- @d mom usouacu onus A. A8816. m new" stun-um l and mum 1.1.1.11‘ 1 rectum outrun 1.“ near mun I slut-r mum on 'KIIDIII.‘ l ‘ 1.1.1.3] I "sour-um cumu 7.“ cum sworn-r - [ 1.1.1.3] l “mu success It PIOVTDI muons attain-mm mm, 1.2.t.s]' m1, scone. I nuns 1.3.: rust-oer ustc count 1 1.1.1.s] L “a“ 1.1.i.s[' asstcn a successm owls: corms-nos. position no - 7.2.1 norm-r autumn I 7.).s W runs an mu [ 1...",qu " on Fuse locum-n. cam mus-r 7.2.2.1r 1.3.sl mm run-inc W .2.z.2,~ mm m urn mum L roe ornate-u. runs I cooper mean no "81111,er , 1.1. L tun-sun soon-cm owns-s 1.41... ,._1) 1.2.2.; mm scum assesses L - l “'1" ms 7° 1.2.z.s| 1 "In?!” nocm 7.1.1 CROSS-TIMI smarts 1 [ 7.2.2.s| - 1 I mu runs L mun n Jon-exams: 1‘ 7.2.).1r i I llsnucr nmsm counscs L 7.2.3.2r m sumac mm: nut L F test We 1.2.1.3]. I 1.).I T concoct ormrtom. 1. sum unctsls ,.LL‘I‘ comm omrtovuu. 1 run an menses ,.LMJ‘ L J 1' mu m srsrnt comer sun-vise cactus L I not an uncut J TJJJJ' 1.3.9 , ma sacrum. nmtsu L near courier union-inns muss L 7.24.: calm nucrsz I. F mm ness. unctscs unusual consumes 7.2... r "I, rum runs ’- 1:“ 1+5)“; 6965 Figure A—21c.--Operate Training Programs Model Subsystem (Gordon, 1969, p. 42). 172 83.35.. 05.00:. p... 3.0;: 0:. .3 30:035.... UCWF»..OO 20.2343 .Amm .a .mom_ .cascmuV . _auoz smsmam ameacmo ce< HH uHam m:e--.--< a2=m_a a COCOEOFOK-Em .0 .‘J. .0523 £59.!ou .323ch .550: we}?! .0 :33“?! s... 299.82.... 0. 82. 26.6... at» xvdoouuu ~ ~ S 30993 to: his. ‘Ucéu :83 .2: :0 Etc...“ 15 1.3 5008 inc 390.3 .0 £201 .30 J50; 203(hzgfli. 5.... a 53.0..qu 2.8:...95: 3 o. 31.099000 089% o. 3.0:; STUD... >.GB( mm): (23» 3 no 20: Uudm 021 265421 g £30.; .120 05.13... E)..U.:w :OU. “was? .0106. 32:“ s 7,): wt: ‘s..u¢ no Opticio no ‘rvoen inc 0332.. :2 use .5906 .0 29:0ch $2.42.: 1. 3:33:4C :‘.°¢0 .0. 250“ 039:3 IE! 3 o. .3295... ‘1 s 259.28 All a 3.82 s. .8 63.. c. .98 2:00 .0 stress ozz mu): Um“ no i\ >020... 160 .95..."- xoccoc: .33.}... 173 The "what to do" requirements are clearly delineated in the indi- vidual boxes of the BUIC II model. Regrettably, however, the model does not explain how to execute all the steps of an implied linear sequence of steps. Other Related But Non-Qualifying Models The Taba Model.--Taba's Model (1962) is based on an assumption that there is an orderly process which will result in a more thoroughly planned and a more dynamically conceived curriculum. This order might be as follows: Step One: Diagnosis of Needs Step Two: Formulation of Objectives Step Three: Selection of Content Step Four: Organization of Content Step Five: Selection of Learning Experiences Step Six: Organization of Learning Experiences Step Seven: Determination of What to Evaluate and of the Hays and Means of Doing It. (Taba. 1962, p. l2) Step One refers to diagnosing the gaps. deficiencies, and variations of the backgrounds of students as a prerequisite for determining the level on which objectives can be reached by a particular group of stu- dents and the emphasis that may be required in the light of their experi- ence. Formulation of clear and comprehensive objectives (Step Two) provides an essential platform for the curriculum. Perhaps the most difficult task of this step is to translate the general objectives into specific objectives in light of what the unit encompasses and what the analysis of needs indicate (Taba, 1962, p. 350). In large part the objectives deter- mine what content is important and how it should be organized. A unit is 174 likely to generate rich learning if the areas of objectives for it are fairly comprehensive and include some materials on each of the follow- ing: (1) Concepts or ideas to be learned; (2) Attitudes, sensitivities, and feelings to be developed; (3) Ways of thinking to be reinforced, istrengthened. and (4) Habits and skills to be mastered (Taba, l962, p. 350). Step Three requires selecting the content. Basic consideration must be given to the central topic and its dimensions, the focusing ideas in light of which the topic or the unit will be developed, and the specific facts and details which will serve to develop the focusing ideas. Topics must be worthwhile and have a rationale to support their significance. In determining the structure of the topics, the criteria of significance and validity of the content are applied and implemented, as are the criteria of learnability and appropriateness to the instruc- tional needs and the developmental levels (Taba. 1962. p. 352). In providing perspective to particular areas of content and ideas, according to Taba, represent "the essential knowledge that all students should master" (p. 354). These ideas guide the selection and organiza- tion of specific information and its interpretation. For all practical purposes, a list of ideas provide a check against including the irrele- vant and insignificant, whether introduced by the teacher or by the students (p. 354). The specific content should be a valid example of the general idea. have a definite logical connection to the idea, and not just be vaguely related to the topic (Taba, 1962. p. 356). Finally, the development of 175 ideas and the sample content requires the assistance of a content special- ist to evaluate the validity and significance of the ideas and to check the adequacy of the sample. In organizing content in the Taba Model (Step Four), the content needs to be arranged so that the dimensions of inquiry are in a sequential order according to a feasible learning sequence. The topics, the ideas, and the concrete content samples need to be arranged so that there is a movement from the known to the unknown, from the immediate to the remote, from the concrete to the abstract, from the easy to the difficult (p. 359). The first rule to observe is selecting the learning experiences (Step Five) for each idea and its sample content is that each idea should serve some definite function. The learning experiences must have a definite relationship to the objectives (Taba, 1962, pp. 363-364). Generally speaking, a sequence of learning experiences involves at least three main stages (Step Six). At one stage the learning activities are essentially introductory, for opening up, for orientation. These include activities which (a) provide diagnostic evidence for the teacher, i.e., feedback from students on strategies for studying a unit, (b) help the students make a connection with their own experiences, (c) arouse interest, (d) provide concrete descriptive data from which to get a pre- liminary sense of the problems to be dealt with, and (e) create involve- ment and motivation. In this sense an opener has a broader meaning than the usual setting of an environment for learning. Much of the key for success at this stage lies with planning with the students (Taba, l962, p. 365). 176 At the second stage learning activities are designed to develop various aspects of the subject and to provide the needed factual material: reading, research, analysis of data, committee work; study of various kind. Development and analysis need to be followed by the type of assign- ments and activities which help the students to generalize, to put their ideas together and reformulate them in their own terms, to compare and contrast, or to formulate conclusions. While the developmental activi- ties require much individual or small group work, it is more profitable to formulate generalizations and discussion by the entire class (Taba, l962, p. 367). Finally, there are activities designed to apply what has been learned to assess and evaluate, or to set what has been learned into a larger framework. What do these ideas mean? How do they relate to other ideas? .How did we work? What could we do better, or differently, the next time. Another form of summary, testing, or synthesis of what is learned is applying what is known to a new situation, in a new context (Taba, l962, p. 367). Evaluation (Step Seven), according to the Taba Model, consists of "determining the objectives, diagnosis, or the establishment of base lines for learning and appraising progress and changes" (p. 377). Naturally, all this is much more accurate and objectives if the evalu- ator's judgments are based on evidence. Much evaluation is actually continuous diagnosis, accompanied by comparison of results (p. 377). After the outline of the unit is completed in writing, it is neces- sary to check the overall consistency among its parts (Step Eight). 177 Are the ideas pertinent to the topic? Does the content outline match the logic of the core ideas? Is the sampling of detail as sharp as it could be? Do the learning activities provide genuine opportunity for the development of the content ideas? Does the sequence of content and learning experiences flow? Is there proper cumulative progression? Is there a proper balance and alternation in the modes of learning: intake and synthesis and reformulation, reading, writing, oral work; research and analysis? Are there a variety of expressions, such as dramatization, creative writing, construction, painting? (Taba, 1962, p. 379). A check is also needed as to whether the organization is suffi- ciently open-ended to provide alternatives both for content detail to be used and for ways of learning to allow for Special needs. Some students may.need an abundant opportunity just to open up and talk. Other groups may be beset with interpersonal difficulties. They may require consider- able emphasis on training in the ways and means of groups work (Taba, 1962, p. 379). Finally, there are practical considerations. While it is important to conceive a unit of work first in the most ideal terms, its final shape should take due account of the limitations of a given school situation, of which there are many. For example, needed materials may not be avail- able, or teachers may lack the proper background for teaching certain things (Taba, 1962, p. 379). Taba's Model is equally limited in operational value. First, it does not graphically show the relationship of the components to each other, nor the feedback process commonly associated with the process. Secondly, 178 the length of its narrative description of the eight steps precludes facility in applying the model during development. Thirdly, the model places too much emphasis on process linearity. Lastly, the Taba Model is more comprehensive in discussing what to do as opposed to how to do it. The ngham Model.--Popham (1970) proposed an empirical teaching model which is similar to the GMI model and is based on the notion that a teacher should be, among other things, a highly skilled technician who systematically improves the quality of his instructional efforts (p. 9). The model is illustrated in Figure A-23. Specify Pre-assess Select Learning Evaluate Objectives ’ ‘ Activities "’ Figure A-23.--An Empirical Instructional Model (P0pham, 1970, p. 19). Instructional decisions in this approach are based on what happens to the learners as a consequence of instruction. The first step is determining what is to be achieved by specifying objectives. According to P0pham, the teacher should describe his instructional objectives. The objectives should be stated in terms of how learners are to behave after the instruction, that is, what they can do after instruction. In other words, objectives should be stated in terms of observable student behavior. The final requirement of this step is to apply certain learning princi- ples, drawn largely from psychology, to increase the probability that learners will attain a target behavior (p. 14). An example of a learning 179 principle is: giving the learner an opportunity to practice the behavior called for in the instructional objective. A second step in the systematic planning of instruction is to pre- assess the learner's entry level (Popham, 1970, p .12). This step may reveal that pupils already possess the behavior the teacher had original— ly hoped to teach. In this case, the original objectives can be revised upward or new objectives can be substituted (Popham, 1970, p. 13). After the teacher has modified his instructional objectives accord- ing to the results of pre-assessment, the third step is to select learn- ing activities which would achieve those objectives. For example, there are certain learning principles, drawn largely from psychology, that have been shown to increase the probability that pupils will attain a target behavior. The skilled teacher will master a number of these principles and will select learning procedures accordingly (Popham, 1970, p. 14). The final step in the empirical instructional model is evaluation. Evaluation is accomplished by observing post-instructional behavior of pupils. Poor post instructional performance tn! pupils generally reflects inadequacies in the instructional sequence and/or the quality of the instruction (Popham, l970, p. 17). Popham maintains that the value of the empirical scheme is that, regardless of an individual's teaching style, it provides a procedure whereby the teacher, as a technically skilled expert, can, over time, systematically improve the quality of his instruction (p. 20). The model is even less specific than the general Model of Instruction about the functions of the four stages of instructional design. Conse- quently, its operational potential to an instructional developer is 180 questionable. Another limitation of the Popham Model is that it does not contain a recycle or feedback step. The Air Training Command (ATC), located at Randolph Air Force Base, is responsible for over 3,000 basic and advanced training courses. Over 200,000 enlisted and officer personnel are enrolled in these courses each year. In meeting their responsibilities, the ATC has introduced a number of strategies designed to individualize Air Force instruction with media. One of the more significant strategies is the Instructional Systems Development process which has six components: (1) Analysis of system requirements. Data secured by questionnaire interviews, job observation, and information provided civilian hardware suppliers are used for analysis. The analysis includes a delineation of the job itself, the personnel required to perform it, and the environ- ment in which the job is conducted. The results are utilized to develop a task list (Neft, 1972, p. 37). ' (2) Definition of the educational or training requirements. This step involves the delineation of the nature of the specific student population. In addition, the cost of their training is estimated (Neft, 1972, p. 37). (3) Development of objectives and tests. Behavioral objectives are developed according to Mager's model and criterion reference tests are employed to measure student attainment (Neft, 1972, p. 37). (4) Planning, development and validation of instruction. Instruc- tional sequences are derived from analysis of objectives developed above. A variety of instructional strategies are employed to achieve these objectives. All strategies employ active responding and student feedback. 181 Validation is sequentially conducted with individuals and small groups before operational tryout (Neft, l972, p. 37). (5) Conducting and evaluating instruction. Courses are evaluated by two methods--internal and field. Internal evaluation involves review of course documents, observations of training, and the evaluation of student responses. The field method involves evaluating the graduate's job performance in a field command (Neft, 1972, p. 37). (6) Feedback. All the five previous steps are linked by a feedback loop. Modifications are made as appropriate (Neft, 1972, p. 37). This model, however, is of limited operational benefit to instruc- tional designers since it fails to consider completely enough the how of the process. A second serious drawback is the exclusion of enabling objectives as the vehicle component for attaining the terminal objectives. Abedor Model.--Abedor (l97l) developed and validated a flowchart or analog model prescribing specific formative evaluation procedures for try- out and revision of prototype multiamedia selfuinstructional learning systems. After devising an initial model developed from a review of the literature on formative evaluation, Abedor used feedback from interviews with seven faculty members who had previously developed (and revised) multi-media lessons as the framework for devising two revised versions of the model. The first was what he called the MK II "mini" model shown in Figure A-24. The "mini" version, according to Abedor, is "highly simplified in order to facilitate conceptual understanding of the pro- cess" (p. 77). The second revised version, the "maxi" MK II model in 182 3.x“ .A_Nmp .soean \ flxmmmmmwx max“ logosa o.m / / m 8.33. \ \ “— o.m N.P mgcmusum seem mpsoxg» o cowuumppou pcmuzum z mung sow mu_pmwmo4 o.m o.¢ ucmanFm>oo mmePmc< cowmw>mm memo o.~ p.p mgcmaxm sock pcmsmmmmm< cowpump—ou Pmuwcxuwh mung toe mowpmwmob cowpmwumEmm empnocm mwmxpwc< EmFroa no.6.6.._p=meH Emsnoea 56.2 cmucu / 183 Figure A-25 is highly detailed and intended for use by consultants or with faculty who are intimately familiar with the "mini" version (p. 77). A detailed explanation of the "maxi" model was included in the Appendix section of the Abedor Study and is summarized in the following paragraphs. The model begins with a prototype which specified that all the instructional materials have been completed without having obtained feed- back from technical experts, or students of the target population (Abedor, l97l, p. 172). In Box l.l.l the evaluator must decide whether he wants feedback on technical problems in the form of a verbal debriefing, a written report, a rating scale, questionnaire, or other device. When these decisions have been made, three types of consultants--subject matter experts, media specialist, and evaluation specialistv-are selected (Box l.l.2) and briefed about the type of information desired and the format to use in obtaining feedback (Abedor, 1971, pp. 173-174). The prototype materials and content are then reproduced and distributed to the selected consultants (Step l.l.3). Step 2.0 reflects the technical review data required of the consul- tants as they interact with the prototype materials. The components are precise and relatively self-explained by the model. The Collect Student Tryout Data Step (3.0) is performed chronologic- ally after one complete cycle. The discrepancies in the prototype materials are analyzed (Step 4.0) on the basis of the feedback data. Deficiencies are then listed in rank order of their seriousness (Step 4.l), after which a tradeoff analysis is conducted to determine the rank order of the problems, assess the probable causes, and select a feasible "cum It'll" ”YA Hm Alul 6 [OK I'- QIAUTY mun mu CANAL Mums ‘ ‘01 N COI- Alflf’TID 6.1.) b IIIGJIC'! Q Pl! ”I? new ft“! "on." "“9 AT? '1 .\' OASVI‘ K Ill! IIVII IMI’I! 4. , 2 [IT IM‘I’ID C")! N QIVII la Q at 184 l IIAIL all :‘NLICT (0'1" NI. at “1‘7 law 'ILIV H‘T 1'01. \IIJTH 'if‘l .‘ " (- LIS‘I’ MI ICII‘K I" l.\ “5! fl“. 9.1 TWIN? “L1!!! 6.2 W mono-co MC!!!" '0 LACK WWII“! m1 1'!“ 6 “‘1' can "noun um Ashton-u can 5 Put em III". '0". Figure A-25.--"Maxi" M0d91 (Abedor, 1971): 185 solution within the constraints of the formative evaluator's resources and ability. For those problem and solution thus selected, a decision is made to "go," to commit additional resources to remediation, and the process enters the DEVELOP REVISION stage. For those problems which did not warrant revisions, e.g., a ”no-go" decision, the process enters the RECYCLE step (Box 6.0) which asks the question: Is the prototype material ready or is additional feedback warranted? (Abedor, l97l, pp. l84-l85). At this point in the process, the formative evaluator must develop revisions (Step 5.0). Content and the treatment related to overall style of presentation, e.g., inductive, deductive, humorous, satiric, or expository, are submitted to revision in message design on the basis of feedback from students during the debriefing. Thereupon, the message complexity is evaluated to assess whether or not there is information overload to a learner's information processing capacity. Some dimensions of message complexity are sense modality, redundancy, word/picture rela- tionships, and rate of presentations. Sense modality refers to whether the audio and visual sense modalities are used simultaneously or sequenti- ally. Redundancy is the repetition of an idea with a sense modality. Word/picture relationships are examined to insure that either words or pictures are related and do not dominate or compete with each other in the message. Rate of presentation is defined as words per minute or visuals per minute, irrespective of language difficulty or visual complex- ity, or idealogical content of the message. The best source of informa- tion on the rate of presentation is the learner (Abedor, 1971, pp. 186- 187). 186 Box 5.2 deals with frequency of response, format for responding, and response type. In general, responses and feedback should be frequent enough so that the learner is aware of his progress and deficiencies. Again, the student debriefing is the ideal source of information to determine optimal response/feedback frequency. A response and feedback can be accomplished in a number of ways: erasing answer sheets, write-in, multiple choice, or a motor performance. Response type may be classified as "enabling" or "criterion." Enabling responses are designed to allow the student practice on the component learning tasks. Success on enabling responses followed by a failure on "criterion" responses indicates in- sufficient practice (Abedor, l97l, p. 189). The final step is to revise evaluation instruments, objectives, or the materials depending on the feedback obtained from the debriefing. The validation strategies of the "mini" and "maxi" verSions of the MK II model are of significance to the instructional developer who is looking for specific methods of testing prototype materials. For example, the variables for collecting technical review data are not specified as clearly in other models as in the Abedor study. There are no apparent weaknesses in the process other than the difficulty, in some cases of instructional development, of getting a representative sampling popula- tion of students to validate materials. Often, students are not readily available, or time will not allow this kind of careful validation pro— cedure called for in the model. Ideally, the strategies for prototype (materials validation are excellent. APPENDIX B PRINCIPAL AND NON-PRINCIPAL TEST ITEMS IDENTIFICATION FOR WRITTEN CERTIFICATION EXAMINATION APPENDIX 8 Michigan Breathalyzer Operator Certification Examination 1971-1972 Principal Test Items Non-Principal Test Items (Numbers) (Numbers) White Test Pink Test White Test Pink Test Form Form Form Form 4 1 1-3 2 7 4 5 .3 8 9 6 5—8 9 10 lO-18 11-15 l9 l6 21 18420 20 17 22 22 23 21 24 24 25 23 28-34 26 26 25 37-39 27 27 28 41 29 35 31 42 30 36 32 48 34-36 40 33 49 38-40 43 37 51-53 42-44 44 41 55 46 45 45 57w60 48 46 47 62 50 47 49 67-70 51 SO 52 75-78 53 52 55 80 54 54 60 81 56 56 62 83 57-59 61 64 84 61 63 67 85-87 63 continued 187 188 Principal Test Items Non-Principal Test Items (Numbers) (Numbers) White Test Pink Test White Test Pink Test Form Form Form Form 64 69 97-100 65 65 70 103-106 66 66 71 112 68 71 74 72 72 76 73 73 78 75 74 79 77 79 80 82 82 81 83 83 84 89-91 88-96 86-88 93 101 92 95 102 94 96 107-110 97 98-lOO 113 101 103 114 102 104 104 106 105 112 107-111 113-115 APPENDIX C LABORATORY CHECKLIST APPENDIX C MICHIGAN BREATHALYZER OPERATOR TRAINING PROGRAM Laboratory Evaluation Sheet Name Date Department __ s = satisfactory n = need improvement u = unsatisfactory Simulator Preparation _a—l—l N—‘oomeU'I-pWN-d o 13: 14. Brea 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. Rinse flask Handle vial cap carefully Rinse out vial twice Replace vial cap when rinsing Level of meniscus Mix solution properly Recheck level of meniscus Rinse flask when finished Wet rubber gasket Check simulator for leak Check temperature of simUTator Equilibrate simulator Place mouthpiece between simulator and breathalyzer Adequate sample thalyzer Operation Fill in preliminary information on test form Check galvanometer lock Zero galvanometer if needed Check temperature Read both ampoules Record control numfier Gauge both ampoules Wipe both ampoules Elean Leave ampoule in gauge wfién breaking Use protection when breaking Properly dispose of ampoule cap Regauge test ampoule Handle bubbler tube properly Check length of bubbler tube Proper adjustment of b ubbler tube Balance from left to right continued 189 31. 32. 33. 34. 35. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. . Replace breath tube 190 First pointer setting Proper timing purge Adequate purge Second pointer setting Recognized "bad" sample Proper timing-sample Interpret reading properly Record reading Sign report form Remove test ampofile and bubbler Replace rubber sleeve ’77 Dispose of test ampoule properly Remove comparison ampoule Flush out inStrument when finiShed Turn selector valve to "OFF" Move carriage to left Use of check list: Delayed start Ahead or behind'with checks Does not check properly Other ’ Comments & Recommendations: Examiner APPENDIX D ATTITUDE TOWARD INSTRUCTIONAL DEVELOPMENT' RATING SCALE ' APPENDIX D Check One Check One Male [:] Teacher [Z] Female E] Administrator [:1 Specialist [:J ATTITUDE TOWARD INSTRUCTIONAL DEVELOPMENT* Definitions Instructional Development or 1.0. is a systOm approach to solving instruc- tional problems. It involves a definition stage where the problem and all related instructional elements and resources, including management organi- zation are identified; a development stage where the behavior necessary to solve the problem is specified in measurable terms and a prototype learn- ing experience is developed which employs the most effective methods and media that learning theory and practical experience can suggest; and finally, it involves a testing and application stage where the prototype system is tried out and revised repeatedly until some version(s) success- fully teaches the desired behavior. Only then is the resulting system used by teachers who have been thoroughly trained to use it properly with qualified learners. Instructions When you answer the following statemynts please try to express the.way you honestly feel about this idea of instructional development or 1.0. Your answer is correct if it expresses your true opinion. PLEASE ANSWER EVERY ITEM. In each case encircle the letter which represents your own ideas as follows: SA if you agree completely with the statement - A if you agree in general but wish to modify it somewhat U if your attitude is undecided D if you disagree but with certain modifications D S if you completely disagree *Produced under a grant from the U.S. Office of Education, Bureau of Libraries and Educational Technology, Division of Educational Technology, Media Specialist Program. Produced for the National Special Media Insti- tutes by Jack V. Edling. © Copyright, National Special Media Institutes, 1971. Unit 10 Module 3 191 11. 12. 13. 14. 15. 16. 17. 18. 192 I.D. should be a part of the professional prepa- ration of all teachers. I.D. places too much emphasis on programming, media and technology. I.D. makes one realize that you have to be specific SA SA SA on problems and objectives to communicate effectively. . I.D. really gives primary consideration to the learner's needs. I.D. is a waste of time. I.D. is so significant that it is urgent to promote its wide adoption. I.D. allows each child to start from where is is and progress as far as he is capable. I.D. enables children to find capabilities within themselves that they wouldn't have been able to find without it. I.D. is nothing new. I.D. seems like a better solution to our problems than anything else currently being considered. I.D. will be ineffective unless all members of a team have a thorough understanding of the system and are committed to it. I.D. is a flexible approach that allows for ex- pansion and change. I.D. is simply the old problem-solving method. I.D. is the most challenging idea in education at the present time. I.D. is the only really effective way to evolve a relevant curriculum. I.D. requires too many alternatives to be practical. I.D. enables the teacher to better see the pur- poses of his instructional program. I.D. cannot be compared with traditional approaches to improving instruction. SA SA SA SA SA SA SA SA SA SA SA SA SD SD SO SO SO SO SO SD SO SO SO SD SO SO SO SO SO SO 19. 20. 21. 22. 23. 240 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 193 I.D. will work only when everyone directly involved in instruction is favorable and familiar with it. I.D. requires concentrated effort at first but it becomes less demanding as it becomes better under- stood. I.D. is something every educator can use. I.D. enables people to better work together to meet the needs of students. I.D. enables teachers to develop new and more effec- tive methods for meeting student needs. I.D. may have some advantages but I haven't been sold completely on it. I.D. is the most productive in-service training that I can conceive. I.D. is the best answer yet for teachers who are looking for an objective method for attacking cur- riculum problems. I.D. is a boring and uninteresting activity. I.D. is the means to reduce the gap between "what is" and "what should be." I.D. provides a means for "getting a handle" on the problems facing school districts. I.D. can be the change agent that will elevate us from the morass of problems that blind, confuse and befuddle us. I.D. is fine but I couldn't do it by myself. I.D. is right on target--there is no better way or more opportune time than to move on it right now. 1.0. enables you to get the most effeCt for the money available. I.D. has recognized and structured a systematic way to resolve problems and all educators should become committed to it. I.D. is a giant step forward. SA SA SA SA SA SA SA SA SA SA .SA SA SA SA SO SO SD SD SD SD SD SO SO SO SO SO SO SO SO SO SD 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 194 I.D. really makes one think about all aspects of the SA educational task. I.D. prov ides a method to assess the goals of an . SA instructional program realistically in terms of available resources. I.D. has taken curriculum improvement from the SA abstract to tangible evidence in dealing with edu- cational objectives. I.D. is a procedure that will result in the improve- SA ment of an instructional program. I.D. is long overdue-_ think of how many children SA we have failed and blamed them for their failure. I.D. is a "must" for every administrator who SA assumes the role of instructional leader. I.D. helps teachers who have had little training SA on how to plan systematically. I.D. and the resulting more systematic instruction SA has become essential since the educational process has become so complex. I.D. is not an end in itself, but simply a means SA that educators can and must use to update schools. I.D. is the best alternative we have to accomplish SA the task at hand. I.D. seems to be the way to go. SA 1.0. is essential to get the support so often re- SA fused because we're always dealing with generalities. I.D. is what we have been needing for years. SA I.D. will succeed because it places primary empha- SA sis on the learner and learning. I.D. is the nearest thing we have to a panacea in SA education. SO SO SO SO SD SD SO SO SD SD SO SO SD SD SD Idaho Falls Attitude Tests Chemawa Attitude Tests Baton Rouge Attitude Tests Albuquerque Attitude Tests Albuquerque #2 Attitude Tests Bucknell Attitude Tests East Greenbush Attitude Tests 195 (August 9-14, 1971) n = 57 M = 188.2 R = 102 +244 S.D. = 3.4 (August 23-28, 1971) n = 68 M = 190.14 R = 74 +234 SOD. = 3003 (August 5-10, 1971) n = 51 M = 174.26 R - 103 ->234 S.D. = 4.84 (February 8-16, 1972) n = 39 M = 201.35 R = 142 +250 S.D. = 3.48 (February 29, March 1-3, 6-8, 1972) n = 37 M = 186.7 R= 135+239 S.D. = 4.28 (January 7-9 and 14-16, 1972) n = 31 M = 201.5 R= 147 +240 S.D. = 4.04 (November 12 and 13, 16-20, 1971) n = 32 M = 199.87 R= 167—>244 S.D. = 3.5 Jacksonville #1 Attitude Tests Jacksonville #2 Attitude Tests Mt. Edgecumbe Attitude Tests Plattsburgh Attitude Tests Richmond Attitude Tests Pocatello Attitude Tests Attitude Tests 196 (November 8013, 1971) n = 24 M = 203 R =153 +237 S.D. = 4.44 (February 15-18 and 21-23, 1972) n = 34 M = 214.2 R = 166 +243 S.D. = 2.74 (November 10-12 and 15-18, 1971) n = 35 M = 198.9 R = 135 +234 S.D. = 4.19 (October 25-30, 1971) n = 35 M = 207 R -169-)'242 S.D. = 2.43 (November 15-19 and 22-23, 1971) n = 27 M = 210.6 R = 183 +246 S.D. = 3.66 (August 16-21, 1971) n = 45 M = 188.9 R =154 +226 S.D. = 2.52 (April 25-29, 1972) 20 202.0 171 +241 3.95 ° 303:3 II II II II S.D Gallup Attitude Tests San Jose Attitude Tests Statesboro Attitude Tests Toledo Attitude Tests Union Endicott Attitude Tests 197 (April 24-29, 1972) n = 44 M = 198.7 R = 148 —+241 S.D. = 2.98 (March 23, 24, 27-30, 1972) n = 21 M = 199.2 R= 93-*232 5.0. = 6.57 (February 17-18, 21-25, 1972) n = 27 M = 213.19 R =158—t-244 S.D. = 4.8 (January 24-28, 31 and February l, 1972) n = 30 M = 190.5 R = 151 +209 S.D. = 2.94 (January lO-15, 1972) n = 43 M = 208.5 R =172 +240 S.D. = 1.57 APPENDIX E CLIENT'S OPINION TOWARD INSTRUCTIONAL DEVELOPMENT RATING SCALE APPENDIX E Client's Opinions Reggrding_InStrUctiona1 Development DIRECTIONS: Please indicate your opinion of your level of understanding and/or use of the instructional development process at the time of your entry (before October, 1971), and at the present time June, 1972) by writing the appropriate number (see the numbers given below) in the proper spaces following each item listed below. The meanings of the numbers are: 1. 2. EXAMPLE: You knew (or know) its meaning and/or how to execute this step of the instructional development process. You knew (or know) its meaning and/or how to execute this step of the instructional development process to a considerable degree. You knew (or know) its meaning and/or how to execute this step of the instructional development process only moderately well. You knew (or know) its meaning and/or how to execute this step of the instructional development process only partially. You did not (or do not) know its meaning and/or how to execute this step of the instructional development process. Your Opinion of Your Performance Level at As a result of the experience(s) I have had with instructional development, I feel I know (or knew): Entry Exit 0. That instructional development is a process which takes a great deal of time. 5 1 How to identify and/or write broad instructional goals. How to identify useful sources of data for decision-making on instructional problems, i.e., staff reports, staff and learner interviews, monitoring in- struction using audio and/or video tapes, statistical analysis of tests, examining course materials, etc. 198 9. 10. 11. 12. 13. 14. 15. 16. 199 What is meant by the term organize the management. Who the individuals were (or are) who comprised the management during the instructional development with the BOTP. . How to distinguish between symptoms of a problem and the problem itself. How to analyze the discrepancies be- tween what is and what should be when identifying the problem. What is meant by preassessment of entry skills. How to preassess the entry skills of learners using a terminal behavior test. What a terminal behavior pretest is. What is meant by a behavioral objective. How to write behavioral objectives which describe what the learner will be doing at the end of instruction, the condi- tions under which he will do them, and the criteria of successful performance by the student. What is meant by task analysis. How to do a task analysis. What is meant by a task description. How to write a task description. What the essential questions to ask when doing a task analysis. Your Opinion of Your Performance Level at Entry Exit 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 200 Your Opinion of Your Performance Level at Entry Exit What is meant by a technical and com- munications review for instructional materials. What factors to consider when doing a technical and communications review during instructional development. What an enabling objective is. What the difference between enabling and terminal objectives is. What is meant by analyzing the instruc- tional setting, i.e., learner character- istics, physical facilities, instruc- tional materials available, staff and support personnel. What is meant by a prototype test. How to construct a prototype test. How to select the media form to use with the type of instruction planned. What support services have to be checked out prior to instruction, i.e., schedu- ling of equipment and materials, avail- ability of paraprofessional support personnel, instructional materials available. How to evaluate the achievement of the instructional program.- That the instructional development process is non-linear. What is meant by recycling in the instructional development process. APPENDIX F CLIENT'S POST-ATTITUDE QUESTIONNAIRE APPENDIX F QUESTIONNAIRE DIRECTIONS: Please use your experiences with the instructional deveJOP- ment process as the basis for responding to the following. questions. If you need more space than is provided, ’ continue your response(s) on the back side of the page. 1. What instructional development plans, if any, do you have for modify- ing or changing the Breathalyzer Operator Training Program. 2. Have you attempted to convince others of the value of the instruc- tional development process? Yes No Explain briefly who and why. 3. What were the most effective instructional development changes made in the Breathalyzer program during the past several months? Explain‘ghy. 4. What were the least effective instructional development changes made in the Breathalyzer program during the past several months? Explain why. 201 202 QUESTIONNAIRE (cont'd) 5. 6. 10. What other instructional development, if any, do you plan to get involved with as part of your responsibilities with the Highway Traffic Safety Center? What aspects of the instructional development process have been the most difficult for you to understand and/or to execute effectively and why? What are your present impressions of the instructional development process? What do you see as the main value, if any, of the instructional development process? With which, if any, of the instructional develOpment process steps do you have reservations and why? Would you say that your present attitude toward the instructional development process is: For the most part positive For the most part neutral For the most part negative What reasons do you have for feeling this way about the instructional development process? BIBLIOGRAPHY BIBLIOGRAPHY Abedor, Allan J. "Development and Validation of a Model Explicating the Formative Evaluation Process for Multi—Media SelfeInstructional Learning Systems." A Thesis submitted at Michigan State University. College of Education. 1971. Abedor, Allan J. and Gustafson, Kent L. "Evaluating Instructional Development Programs: Two Sets of Criteria." Audiovisual Instruc- tion, Vol. 16, No. 10, December, 1971. Alexander, Lawrence T. and Yelon, Stephen L. "The Use of a Common Expriential Reference in Instructional Systems Design.” The Educa- tional Technology Review Series: InstrUCtional Systems.. Englewood Cliffs, New Jersey. April, 1969. Allen, David and Bowers, William K. “Multi-Media," Part I, Educational Screen and Audiovisual Guide, December, 1969, p. 27. Barnlund, Dean C. "A Transactional Model of Communication." From Johnnye Akin, Alvin Goldberg, Gail Myers and Joseph Stewart, eds. Language Behavior: A Bogk_of Readings. The Hague, The Netherlands. Mouton and Company, n.v. Publishers,in press. Barson, John. "Instructional Systems Development: A Demonstration Project: Final Report." Michigan State University, East Lansing. Report Number BR 5-1411. Contract OEC 5-16-025. June 1, 1967. Benathy, Bela H. "A Systems Analysis of Systems Education." Educational Technology. Volume XII. Number 2. Englewood Cliffs, New Jersey. February, 1972. . ‘Instructional Systems. Palo Alto, California. Fearon Publishers. 1968. 1 Bertalanffy, Ludwig von. "General Systems Theory.“ General Systems Theory Yearbook, I. 1956. Beynon, Robert. "The Total Systems Concept: Research Implications." Instructional Technolo .: A Book of Readin 5. Edited by Frederick GT Knirk and—John W. 551135. New YorE: Holt, Rinehart and Winston, Inc., 1968. Borich, Gary D. (Book Review). "Handbook on Formative and Summative Evalu- ation of Student Learning," by Benjamin S. Bloom, J. Thomas Hastings, and George F. Madeus. AV Communications Review, Volume XIX, Number 3, Fall, 1971. 203 204 Boulding, Kenneth E. "General Systems Theory: The Skeleton of Science." Management Science, 2. 1956. Briggs, Leslie J. Handbook of Procedures for the Design of Instruction.' Pittsburg: American Institute for Research, 1970. . "A Procedure for the Design of Multimedia Instruction." Instructional Technolo : A Book of Readin 5. Edited by Frederick GT Knirk and Jfihn W. Ca1las. New York: Holt, Rinehart and Winston, Inc., 1970. Brooks, Carl N. "Training Systems Evaluation Using Mathematical Models." Educational Technology,’VblumeIX, Number 6, Englewood Cliffs. New Jersey, June, 19 . Carter, Launor F. "The Systems Approach to Education: Mystique and Reality," Educational Technology, Vol. IX, Number 4. Englewood Cliffs, New Jersey, April, 1969. Childs, John W. "A Set of Procedures for the Planning of Instruction." The Educational Technology Reviews Series: Instructional Systems. EfiglewoodCliffs, New Jersey, no date. Churchman, C. West. The Systems Approach.' New York: A Delta Book, 1968. Cummings, Roy J. "Air Traffic Controller Training at the FAA Academy,“ Educational Technology, Vol. XII, Number 1, Englewood Cliffs, New Jersey, January, 1970. Cunningham, Donald J. (Book Reviewer). "Perspectives of Curriculum Evaluation," by Ralph M. Tyler, Robert M. Gagné} and Michael Scriven. AV Communications Review, Volume XVIII, Number 3, Fall, 1970. Cyrs, Thomas E. and Lowenthal, Rita. "A Model for Curriculum Design Using the Systems Approach," Audiovisual Instruction, Vol. XV, No. DeCecco, John P. "The Psychology of Learning and Instruction: Educa- ‘tional Psychology.” Englewood Cliffs, New Jersey, 1968. Deterline, William A. "Applied Accountability," Educational Technolo , Vol. XI, No. l, Englewood Cliffs, New Jersey, January, 1971. - Deutsch, Karl W. "Toward A Cybernetic Model of Man and Society," From some notes on Research on the Role of Models in the Natural and Social Sciences. Synthese: 7 (48'-49). 506 33. D. Reidel Pub- lishing Company. Diamond, Robert M.) "Instructional Development--Fact or Fiction," Audiovjsgal Instruction, Vol. 16, No. 10, December, 1971. 205 Douglas, Harlan L. "Instructional Development in Three Phases," Audiovisual InstruCtion, Vol. 16, No. 10, December, 1071. Gage, Gerald. "Accountability in the American College.” A Paper De- livered at the Instructional Development Luncheon." Michigan State University, July 14, 1971. Gagne, Robert H. 'The Conditions of Learning." 2nd Ed. New York: Holt, Rinehart and Winston, Inc., 1970. ‘“~.M Gagné, Robert M. and Melton, Arthur W. ~Psychological_Princip1es in Systems Development. New York: Holt, Rinehart and Winston, Inc., 962. Garvue, Robert J. "Accountability: Comments and Questions." Educa- tional Technology, Vol. XI, No. l, Englewood Cliffs, New Jersey, January, 1971. Gerlach, Vernon S. and Ely, Donald P. Teaching_and Media: A Systematic Approach. Englewood Cliffs, New Jersey: PrentiEe-Hall, Inc., 1971. Glaser Robert. ‘Teachin Research and Education.* New York: John Wiley and Sons, 1965. Gooler, Dennis D. and Groteluescher, Arden. "Process Accountability in Curriculum Development." University of Illinois, Urbana-Champaign. Center for Institutional Research and Curriculum Evaluation, no date. Gordon, Jerome J. "A Systems Model for Civil Defense Training and Educa- tion," ducational Technology, Vol. IX, No. 6, Englewood Cliifs, New Jersey, June, 1969. Guilford, J. P. Fundamental Statistics in Psycholggywand Education, New York: McGraw-Hill Company, 1965. Gustafson, Kent L. "Toward A Definition of Instructional Development." A Paper Presented to the Instructional Development Division.: Association of Educational Communications and Technology, Philadelphia, March, 1971. Hamreus, Dale G. "The Systems Approach to Instructional Development." Monmouth, Oregon: Teaching Research Division of the Oregon State System of Higher Education, April, 1968. Haney, John B., Lange, Phil C., and Barson, John. "The Heuristic Dimen- sion of Instructional Development," AV Communications Review, Vol. 16, No. 4, Winter, 1968. Heinich, Robert. "What Is Instructional Development?" Audiovisual Instruction, Vol. 12, No. 4, March, 1968. 206 Heinich, Robert. "The Teacher in An Instructional System," Instructional ' Technology: A Book of Readings. Edited by Frederick G. Knirk and—— thnFW. ilds. New York: Holt, Rinehart and Winston, Inc., 1968. Hoban, Charles F. "From Theory to PolicyeDecision," AV Communications Review, Vol. 13, No. 2, Summer, 1965. Hyer, Anna L. "From Gold Stars to Green Stamps." Audiovisual Instruction, Vol. 16, No. 5, May, 1971. Kaufman, Roger A. "Accountability, A Systems Approach and the Quantita- tive Improvement of Education: An Attempted Integration," Educational Technology, Vol. 16, No. 1, January, 1971. . (Book Reviewer). "Every Kid a Winner: Accountability in Education," AV Communications Review, Vol. 19, No. 3, Fall, 1971. . "A System Approach to Education: Deviations and Definitions," AV Communications Review, Vol. 16, No. 4, Winter, 1968. Kemp, Jerrold. Instrgctional Design: A_Plan for Unit and Cogrse Development. Belmont, California: Fearon Publishers, 1971. Kerlinger, F. N. Foundations of Behavioral Research. New York: Holt, Rinehart and Winston, 1964? Kibler, Robert J.. Barker, Larry L., and Miles, David T. Behavioral Objectives and Instruction. Boston: Allyn and Bacon, Inc., 1970. Leonard, W. Patrick. "Individualizing Air Force Instruction With Media," Audiovisual Instruction, Vol. 17, No. 6, June/July, 1972. Lessinger, Leon M. "Robbing Dr. Peter to Pay Paul: Accounting for Our Stewardship of Public Education," Educational Technology, Vol. XI, No. l. Englewood Cliffs: New Jersey, January, 1971. Merrill, M. David. "Components of a Cybernetic Instructional Model." Educational Technology Review Series: Instructional_§ystems. Englewood Cliffs, New Jersey, no date. Merrill, M. David and Goodman, R. Irwin. "Selecting Instructional Strate- gies and Media: A Place to Begin." Prepared for the National Special Media Institutes, 1971. Miller, Elwood E. "A Descriptive Study, Evaluation and Analysis of In- structional Systems Development Activities in Selected Departments at Michigan State University During the Period 1960 to 1963.“ A Thesis submitted at Michigan State University, College of Education, 1965. 207 Miller, Elwood E. -"Directions for Instructional Development." A Paper. Presented to the Instructional Development Symposium at Michigan State University, April, 1971. Morrisett, Irving, Stevens, W. W., Jr., and Woodley, Celeste P. "A Model for Analyzing Curriculum.Materials and.Classroom.Transactions.“ Social Studies Curriculum Development:. Prospects and Problgmg, 39th Yearbook. National Council for the Social Studies, 1968. National Special Media Institutes. "What Is IDI?" The Instructional Development Institute Program for School Districts, 1971 (Brochure). Nelson, Frank G. "Models for Evaluation." Monmouth, Oregon: Teaching Research Division of the Oregon State System of Higher Education, 1970. Nord, James R. "A Search for Meaning," Audiovisual Instruction, Vol. 16, No. 10, DecembEr, 1971. Paulson, CaSper F., Jr. "A Strategy for Evaluation Design." Monmouth, Oregon: Teaching Research Division of the Oregon State System of Higher Education, April, 1970. Popham, James W. Establishing Instructional Goals. Englewood Cliffs, New Jersey: PrentiEe-Hall, Inc., 1970?— Randall, Ronald K. "Perspectives on the Instructiona1.System," Educa- tional Technology Review Series: .Instructional Systems. 'Englewood Cliffs, New Jersey. no date. Schaur, Clarence. "A Vice-President.Looks at Instructional Development." Audiovisual Instruction, Vol. 16, No. 10, December, 1971. Silverman, Robert E., contributing editor. "Theories and Models and Their Utility," Educational Technology Review Series. 'Englewood Cliffs, New Jersey. no date. Smith, R. 6., Jr. "Controlling the Quality of Training." Technical Report 65-6. Human Resources Research Office, June, 1965. Stowe, Richard A. "The Critical Issue in Instructional Development." Audiovisual Instructidn, Vol. 16, No. 10, December, 1971. . "Instructional Development's the Thing," Audiovisual Instruc- tion, Vol. 16, No. 4, April, 1971. Taba, Hilda. Curriculum Development: Theory and Practice.. Neerork: Harcourt, Brace and World, Inc., 1962. 208 Trzbiatowski, Gregory Louis. "An Evaluation of the Instructional Sys- tems Approach in Higher Education." A Thesis submitted at Michigan State University, College of Education, 1967. ' Urbach, Floyd. "Instructional Development: An Overview.", (An Informal Paper.) Monmouth, Oregon: Teaching Research Division of the Oregon State System of Higher Education, 1970. Witt, Paul W. F. "Instructional Development: What? Why? How? Who?" A Paper written for the exclusive use of participants in the Symposium on Instructional Development, Michigan State University, May 3 and 7, 1971. HICHIGRN STATE UNIV. LIBRARIES 111111111|Hill“HIN!llHIIU11111111111 31293100657174