AN ANALYSIS OF POTENTIAL. GOALS FOR INETRUCTEONAL TECHNOLOGY 3N HIGHER EDuCATon, 1972-1992, AS A BASES 2:052 RECOMMENDATIONS FOR PROGRAMS OF PREPARATXON FOR st-rmsmmNAL TECHNOLOGISTS Pi? _ E}. MICHKGAN STATE UN‘EVERSETY HARRY LAWRENQE ACKERMAN. JR. @9374 uliuuluuzllyilnllwlluuuww _ L E'BRA R Y 9 Mix higan Stan, University 4" I. This is to certify that the thesis entitled An Analysis of Potential Goals for Instructional Technology in Higher Education! 1972-1992, as a Basis for Recommendations for Programs of Preparation for Instructional Technologists presented by Har ry L. Acke rman Q has been accepted towards fulfillment of the requirements for Ph. D. degree in Instructional Development and Technology Major professor Date November 7L 1974 0-7839 ABSTRACT AN ANALYSIS OF POTENTIAL GOALS FOR. INSTRUCTIONAL TECHNOLOGY IN HIGHER EDUCATION, 1972—1992, AS A BASIS FOR'RECOMMENDATIONS FOR PROGRAMS OF PREPARATION FOR INSTRUCTIONAL TECHNOLOGISTS BY Harry Lawrence Ackerman, Jr. The challenge to instructional technology in higher educa- tion is to exploit the benefits of technology, yet avoid over- mechanizing or over-standardizing teaching and learning methods. Instructional technology offers exciting opportunities for both teaching and learning, yet inept or unskilled use can vitiate its effectiveness. Educators must think about technology and the future in terms of how to utilize it for optimum human benefits. The need to stimulate such thinking led this investigator to gather opinions from authorities in the field as indicators for determining future instructional strategies and appr0priate ap- plications for technology in the future roles to be played by higher education. A concept and a process served as a base for soliciting goals and gaining a consensus of their value from acknowledged leaders in curriculum and instruction in higher education. The concept is goal-setting as a fUnction of leadership; the process is the Delphi technique. The persons selected as participants were Harry Lawrence Ackerman, Jr. acknowledged leaders in curriculum and instruction. They were .selected by their peers as persons qualified to originate and make value judgments on goals and directions for instructional technology as well as for other aspects of higher education. The Delph technique is recognized as an effective method for achieving a consensus opinion from thinkers widely separated geographically. It appeared to be a legitimate means of sampling and distilling the opinions and perspectives of these leaders concerning goals and the values ascribed to them. A group of H2 recognized leaders was identified and invited to participate; 27 accepted and proposed a list of 73 goals for instructional technology for the next 20 years. These goals were categorized and submitted to the participants for a value rating on a five point scale. The goals and ratings were resubmitted to the par- ticipants for final review and change or comment. In the final tabulation, mean and standard deviation scores were computed for each goal to obtain both the group's consensus of value and the dispersion of ratings for each goal. The.list of 73 goals for instructional technology in higher education in the next 20 years was rank ordered according the group's consensus of value. The Delphi technique proved to be an effective instrument for gaining consensus both on goals and their relative values; the participants manifested enthusiasm for the project; and 23 of the 27 accepting invitations completed the study. Based on the list of goals established, several recom- mendations were made for preparing future instructional technolo~ gists. AN ANALYSIS OP POTENTIAL GOALS FOR INSTRUCTIONAL TECHNOLOGY IN HIGHER EDUCATION, 1972-1992, As A BASIS POR RECOMMENDATIONS FOR PROGRAMS OP PREPARATION FOR INSTRUCTIONAL TECHNOLOGISTS By Harry Lawrence Ackerman, Jr. A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Secondary Education and Curriculum 1979 C) Copyright by Harry Lawrence Ackerman, Jr. 197” ii DEDICATION My Children Larry Jim Jon Jennifer Who willingly and cheerfully adjusted to great changes in their lives in order for me to study at Michigan State University. My Mother and Father Doris Harry Whose love and generosity has always helped so much; who first instilled in me the desire to know and to accept challenges. My Sisters Virginia, Shirley, Sandra Who made me try harder. Aunt Myrtle Whose spirit is an example to follow. . . . and, most importantly . . . Fame I love you, Not only for what you are, But for what I am when I am with you. Roy Croft iii ACKNOWLEDGEMENTS I have not reached this point alone. There are many who have inspired, encouraged, and supported me along the way, some consciously, some never knowing. For the seeds of this study were planted long ago and it was some time before I realized what was growing; only now, as I am look- ing backward, can I accurately identify the roles these many peOple played along the way. And, I want them to be here, too. Therefore, these acknowledgements are for this study as well as for experiences before it. First, to my outstanding committee. Dr. Charles F. Schuller, chairman, whose enthusiasm for this study eclipsed my own and repeatedly carried me from problem to solution. It has been a high privilege for me to have had his guidance in my work. From one who has en- cOuraged so many to make changes in a world resenting change, I have gained in spirit, illegitimi ggg_carborundum! Dr. Walter Hapkiewiez has made this study so much easier with his suggestions for clarifying statistical procedures. H13 balance of cool rational thinking and warm good humor haxre made me think of him as a friend, rather than an advison iv Dr. James Nord- His knowledge of futures forecasting techniques was a great resource for me. His sound advice throughout the study strengthened it and my will to continue. Dr. Thomas U. Foster - As a committee member, he gave more of himself to this study and to me than I deserved or can describe. I will always be grateful to him for his advice, encouragement and patience. Dr. Paul W.F. Witt - An unofficial member of the com- mittee and ghg reason why I went to Michigan State University in the first place. This man personifies the highest stan- dards of moral, ethical and professional behavior. But, beyond that he is one of the most caring and concerned people I have met. I am a larger person for having known him. Faculty and staff members of the Instructional Media Center at Michigan State supported and encouraged me as a graduate student. ' Special thanks go to Dr. Elwood E. Miller who sparked my imagination for this study in the beginning; Dr. Kent Gustafson, a model Instructional Development specialist, who guided my initial thoughts and plans in drafting the proposal for the study. And, to Ruth Murphy for keeping records of my program, for warm words, big smiles and en- couragement; to Vivian Herr for processing my final papers. My colleagues in the EPDA-SD Institute, 1970-72, were people from whom ngained so much. My memory is rich with so many good times past. The thousands of students and many teacherer worked with before going to Michigan State University - students, you made me realize there had to be a better way; teachers, by your desire to improve yourselves - you helped me make a deCision. Dr. David Adler and Mr. Thomas Taylor - the former taught me my administrative responsibility for changing things; the latter, a fox, how to get people to change. Special thanks, also, to Mrs. Ann Fenner and Miss Marilyn Furtick for typing numerous drafts and the final copy. They cared so much, I knew I had to finish. There were others who will not be named here, but who will be remembered by me. My education has been an ac- cumulation of experiences with or caused by people, re- sulting in changes in myself. I'm grateful for that and hope it continues. Very Special thanks go to Tom and Ginny Foster for the rare good friends they are to me and my family. Their sus- tained support to my completing this study has been greatly appreciated. Finally, special and deep thanks to my wife Fame. So many sacrifices, so much strength and love - without her, it- never would have happened. vi No one knows what will happen next, Such portents fill the days and nights; Years prophetical! the space ahead as I walk, As I vainly try to pierce it, is full of phantoms, Unborn deeds, things soon to be, Project their shapes around me, This incredible rush and heat, This strange ecstatic fever of dreams 0 years! Your dreams 0 years, how they penetrate through me! (I know not whether I sleep or wake;) The perform'd America and Europe grow dim, Retiring in shadow behind me, The unperform'd, more gigantic than ever, Advance, advance, upon me. From "Years of Moderns," Leaves g£_Grass by Walt Whitman. vii. TABLE OF CONTENTS CHAPTER I. INTRODUCTION . . . . . . . . . . Need for the Study. . . . . . Rationale . . . . . . . . Goal Setting As a Function of The Delphi Technique. Limitations . . . . . Definition of Terms . Overview. . . . . . . II. REVIEW OF THE LITERATURE . . . . Introduction. . . . . . Delphi Studies in Education . Exploratory Studies Using the Technique. . . . . . . Leadership. Delphi Normative Studies Using the Delphi Technique. . . . . . . . . Studies of Educational Programs for Instructional Technologists. Summary . . . . . . . . . . . III. DESIGN OF THE STUDY. . . . . . . Introduction. . . . . . Selection of Participants in the Study. . . . Procedure . . . . Analysis of Data. . Summary . . . . . . IV. RESULTS. . . . . . . . . . . . . Method of.Analysis of Data. . Presentation of Goals by Categories : Rank Order Presentation of Goals. . . smw O O l O I O O O O O 0 viii Page l-‘CDNCDU‘IFH I" P‘ ta LU PHH UJLAJ 1H 16 19 21 29 29 2” 26 28 29 32 32 35 H3 51 TABLE OP CONTENTS--Continued CHAPTER Page V. SUMMARY AND CONCLUSIONS . . . . . . . . . . . . . 53 Introduction . . . . . . . . . . . . . . . . . 53 Summary. . . . . . . . . . . . . . . . . . . . 53 Conclusions. . . . . . . . . . . . . . . . . . 55 Recommendations. . . . . . . . . . . . . . . . 58 Discussion . . . . . . . . . . . . . . . . . . 68 Implications . . . . . . . . . . . . . . . . . 7n LIST OF REFERENCES . . . . . . . . . . . . . . . . . . . 77 APPENDIX A. LETTER.OP SUPPORT . . . . . . . . . . . . . . . . 81 B. COVER LETTER FOR QUESTIONNAIRE I. . . . . . . . . 83 . QUESTIONNAIRE I . . . . . . . . . . . . . . . . . 86 . QUESTIONNAIRE II. . . . . . . . . . . . . . . . . 88 . COVER LETTER FOR QUESTIONNAIRE II . . . . . . . . 102 C D E F. QUESTIONNAIRE III . . . . . . . . . . . . . . . . 10” G. COVER LETTER FOR QUESTIONNAIRE III. . . . . . . . 117 H GOALS AND MINORITY OPINIONS . . . . . . . . . . . 120 ix CHAPTER I INTRODUCTION TO THE STUDY The purpose of this study is threefold; (l) to solicit goals for instructional technology in higher education for the next 20 years from acknowledged leaders in curriculum and instruc- tion; (2) to ascertain from the participants the relative value of each goal; and (3) in terms of those goals, to recommend appro- priate learning experiences for future instructional technologists. NEED FOR THE STUDY The challenge to instructional technology in higher ed; ucation is to eXploit the benefits of technology, yet avoid over- mechanizing or over-standardizing teaching and learning methods. Instructional technology offers exciting opportunities for both teaching and learning, yet inept or unskilled use can vitiate its effectiveness. Educators must think about technology and the future in terms of how to utilize it for optimum human benefits. The need to stimulate such thinking led this investigator to gather opinions from authorities in the field as indicators for determining future instructional strategies and appropriate appli- cations for technology in the fUture roles to be played by higher education. New instructional strategies have emerged in the past 20 years as a result of changes in technology. Developments such as miniaturization of equipment and materials, electronic information retrieval systems, learning resource centers and individualization l 2 of learning, offer new techniques for teaching (Eurich, 1963:U6). The adoption of these techniques by colleges and universities has been swift, making prOSpects good for more rapid and com- prehensive changes in the future (Eurich, 1963:96). Saettler (1968:359) feels innovations in instructional technology are the result of efforts to c0pe with the so-called "knowledge explosion" and are rapidly accelerating the rate of change in teaching methods. New knowledge is thus being generated and Toffler (1970:31) says this is cyclical: "Knowledge is change and accelerating knowledge acquisition, fueling the great engine of technology, means accelerating change." Thus, the speed of change is clearly increasing, but there is little evidence of needed evaluation of which changes are worthy ones. Watson (1968) feels that benefits of technology are being gained, but emphasizes that responsibility for the direction and use of it rests with professional educators, who must decide technology's role in the field. There is evidence that this reSponsibility, if recognized, has not been taken seriously to this point. Brown and Thornton (1968) surveyed a large number of institutions of higher learning to determine what innovational media practices exist in higher education. One of their signifi- cant findings was that applications Of instructional technology to 'higher education appear to be far more adaptive than creative. One result seen by Brown and Thornton, is that "educational ob— jectives are often subordinated to the needs and adaptability of media instructional equipment." It appears that designers of 3 instructional methods have more often placed a higher priority on "fitting" course content to media equipment than to the needs of learners. The same criticism has been made of instructional technologists (Carnegie Foundation for the Advancement of Teaching, 1972, p. 15) who have stressed the "potentials of the new media and systems" and failed to define "reasonable objectives for the deveIOpment of technology itself." Determining techno- logy's develOpment in higher education is important for univer- sities and colleges as they reSpond to pressure for a new role in society. The stereotyped image of universities as groups of cloistered scholars going about their studies in an ivory tower atmOSphere, oblivious of the world and its woes, has long been diSpelled; they are shifting to involvement in community affairs and away from isolation from them (A Report to the President of Michigan State University from the Committee on the Future of the University, 1990, p. 213). More direct action in community affairs is expected of universities. Many faculty and adminis- trators are actively involving their institutions in solving the problems of society, in "building a two-way street between society and the university." Green (1972:29-26) describes how a uni- versity may become involved in the urbanization of America. He calls for "problem—oriented and multidisciplinary" urban and racial studies that transcend internal university organizational structures and send students beyond campus, into "urban affairs" in action-oriented programs. Such changes cause controversy on and off university campuses. Opposing constituent groups within ll universities press for and against continuation of these alters native instructional practices. Leaders of universities are challenged to develOp simultaneously new and more effective pro— grams and to sustain traditional values and functions of the university. Imaginative leadership for fUture reforms must be developed to establish direction for innovations in higher ed— uCation (Brown & Thornton, 1963:176). Instructional technology appears to be one of the promising areas for change. If we can accept McLuhan's (1966:viii) assumption that "any technology gradually creates a totally new human environment," there are serious challenges ahead for instructional technologists and other agents of educational change both in terms of their Own innovative practices and in how they relate to the forces within and beyond the academic world. In preparing future instruc- tional technologists it is essential to identify these challenges as best we can and to determine the more promising directions in which they may influence change. It is possible to identify changing goals for higher education, arrive at some reasoned con- sensus of their relative values and to plan future courses of action to attain them (Helmer and Rescher, 1959:90). RATIONALE A concept and a process serve as a base for soliciting goals and gaining a consensus on their value from acknowledged leaders in curriculum and instruction in higher education. The concept is goal-setting as a function of leadership; the process is the Delphi Technique. 5 GOAL SETTING AS A FUNCTION OF LEADERSHIP Goal-setting is acknowledged as a key function of leader- ship. Lippitt (1955:556-557) listed "giving information or ad- vice on setting directions of growth," and "indicating plans," as two major functions of leadership. In an analysis of skills aiding in the identification of leadership, Kissinger (1959:30) agreed. He claimed the ability to "infuse and occasionally to transcend routine with purpose," is found in men rising to leader- ship positions from specialized functions of a bureaucracy. Skertchly (1968) stated that leaders must make goal decisions relating to both quantity and quality in their organizations by aSking: "What do I want the enterprise to become: ...Having decided upon reasonable and at- tainable objectives, the process through which these goals can be reached will then determine the product or service range... (these may be) quantitative goals...(or) qualitative goals..." Qp.61). Hollander (l969:8) reviewed the results of four leadership studies which indicated that peer recognition of a person's capacities to point-the-way is evidence of leadership. Taking the form of hierarchical status in organizations and professional grOUps, recognition includes acknowledgment of a leader's accurate percep- tion of the aims the group seeks or should seek to achieve. Leadership exists on all organizational levels. Acknowl— edgment of leadership capability involves more than recognition of the individual as an innovator. Sartain and Baker (1965:237) state that the person must be a "thinker... who almost without exception, ...(will) be found to be an authority in his field, an experienced 6 'doer’ who has exhibited operational capabilities for being entrusted with topélevel planning decisions." The present study undertook to use the above concept of leadership as a point of reference for identification of probable long-range goals for instructional technology in higher education. The persons selected as participants are acknowledged leaders in curriculum and instruction. They were selected by their peers as persons qualified to originate and make value judgments on goals and directiOns for instructional technology as well as for other aspects of higher education. These leaders are located in various parts of the United States and it would have been impracti— cal to try to bring them together. Even if this could have been done, face-to-face consultation on this project could have dis- torted the results. It was therefore decided to apply the Delphi technique as the instrument to solicit and gain value judgments on goals for instructional technology in higher education. \ THE DELPHI TECHNIQUE The Delphi technique, developed at the Rand Corporation in the 1950's under the guidance of Olaf Helmer, is recognized as one of the most effective methods for achieving a consensus Opinion from thinkers widely separated geographically (Pfeiffer, 1968:152). Conference-type meetings are frequently used as an aid to planning to obtain advice and assess opinions of individuals who work in close or relative proximity. It has been known for some time, however, that direct conferring of this nature has smmm inherent weaknesses (Kelly and Thibaut, 1959). Asch (1953) 7 substantiates this with the following listing, among others: (1) group Opinion is strongly influenced by dominant, talkative individuals; (2) much discussion in group situations, though appearing to be problem-oriented, is often meaningless or pre- judicial to progress toward problem solution; (3) group pressure to conform can distort individual judgment. In an experiment by Dalkey (1969), it was found that "after face-to-face discussion, more often than not the group response is less accurate than a simple median of individual estimates without discussion." Gaining a consensus by the Delphi technique is normally accomplished by having consultants complete a series of question- naires. Among the advantages of the Delphi technique are savings in time and money (Uhl, 1971), but accuracy of opinions or esti- mations of the consultants is the more important goal. The method seeks to reduce "influence of certain psychological factors, such as Specious persuasion, unwillingness to abandon publicly expressed Opinions, and the bandwagon effect of majority opinion" (Helmer and Rescher, 1959:90). The experimenter controls inter- action among participants to aid them to think independently and to provide time for them to form rational opinions. The re5pon- dents remain anonymous throughout the study. The first step in a Delphi study is to provide selected subjects with a questionnaire that asks them to list their opinions on a particular topic. A second questionnaire randomly lists all responses from all participants in the first round and asks each S"bject to rate or evaluate each item by some criterion such as imPOrtance, probability of success, and so on. In the third 8 round, the list and a summary of responses to the items are sent to the participants. Each may revise his opinion, if in the minority, or indicate his reasons for remaining outside the con- sensus of the group. Finally, the fourth questionnaire includes list, ratings, the consensus, and minority opinions. This gives each reSpondent a final chance to revise his opinions. The procedure usually succeeds in gaining convergence of opinion and provides a clearly defended minority opinion (Pfeiffer, 1968: 152-153). LIMITATIONS Naturally, there are potential limitations in the Delphi technique. Dennis Weaver (1972), for example, saw a "serious sterility in the process of summarizing mass information into numerous narrowly terse statements" in his critical review of the Delphi technique. Despite its limitations, however, the Delphi technique has proved particularly useful and suitable for studies concerned with opinions and estimates of future develop- ments. Uhl (1972), Cyphert and Gant (1971), Anderson (1970), Norton (1970), and JUdd (1970), had to cope with Delphi's limita- tions. Each declared that useful knowledge was gained for deter- mining future directions Of growth for the institutions involved in their studies. Data gained by the Delphi technique made aspirations of constituent groups common knowledge to all. Priorities were achieved for each institution and constituent grouns in all were pulled together in focusing on a common direc- tion of growth. 9 There are few alternative methods to the Delphi technique for gaining a consensus of opinions. As previously indicated, face—to-face conference meetings also have limitations. It would be particularly difficult if not impossible to assemble the participants in the present study due to wide geographical sep- aration and the normal difficulties in scheduling such a meeting among leading educators. Useful knowledge will be gained of the opinions of leaders in curriculum and instruction in higher ed- ucation regarding desirable directions for the future development of instructional technology. It is also anticipated that attention will be focused on the need for other studies to provide infor- mation on alternative goals and directions so that increased em- phasis will be focused on planning for the future. In summary, there appears to be a real need for the es- tablishment of new goals for instructional technology in higher education. Recognized leaders in curriculum and instruction in higher education are the logical sources for identification of such goals and the relative merits of each - by virtue of the goal setting re6ponsibilities inherent in their leadership positions. The Delphi technique appears to be a legitimate means of sampling and distilling the opinions and perspectives of these leaders concerning goals and the relative values ascribed to them. There— fore, it appears that useful knowledge may be acquired by applying the Delphi technique to gaining a consensus on values held for goals in instructional technology by acknowledged leaders in curriculum and instruction in higher education, in the United States. 10 DEFINITIONS OF TERMS Several terms used in the study are of importance in interpreting it. .Aeknowledged Leader As used in this study, "acknowledged leader" includes three classifications of persons all of whom share generally recognized high reputations for demonstrated interest and skills in dealing with problems of instruction and curriculum in higher education. These classifications include individuals who: (1) hold or have formerly held a college or university professional appointment of high administrative reSponsibility; (2) hold or have formerly held a position of leadership 'in a national foundation or professional organization devoted to improving curriculum and/or instruction in higher education; (3) are recognized for scholarship and knowledge of instructional and curriculum problems in higher educa- tion through publications, participation in pertinent national studies and/or through the recommendations of other nationally known scholars. 9231 A goal is a statement, submitted by a participant in the Study, of a condition or action which he considers desirable and Worthy of achieving for instructional technology in higher ed— ucation. ll Instructional Technology (I.T.) The definition of Instructional Technology (I.T.) used by the Commission on Instructional Technology (1971:S) is used in this study because it is sufficiently comprehensive and clearly stated. "Instructional technology...is a systematic way of designing, carrying out, and evaluating the total process of learning and teaching in terms of Specific objectives, based on research in hu— man learning and communication, and employing a combination of human and non—human resources to bring about more effective instruction." Higher Education Higher Education refers to colleges and universities in the United States that Offer baccalaureate degrees. Delphi Techniqge The Delphi Technique is a method of technological fore- casting that can be used to achieve a consensus of opinions and/or value judgments of consultants without bringing them together, by having them complete a sequence of questionnaires concerning de- sirable or needed future developments.* OVERVIEW A need and framework for the study have been attempted in this chapter. A literature review of uses of the Delphi technique in education and of current professional training pro- grams of instructional technologists is contained in Chapter II. *Delphi is also used for prediction and conflict resolution purposes; neither purpose was used in this study. 12 Chapter III contains a discussion of the subjects and procedures of the study and the method of analysis employed. The results of the study and an analysis of data are reported in Chapter IV. A summary of the study, conclusions, recommendations, discussion, and implications for further research are found in Chapter V. CHAPTER 11' REVIEW OF THE LITERATURE INTRODUCTION In this study, acknowledged leaders in higher education originate and place relative values on goals for instructional technology during the next 20 years. The consensus of their considered judgments is used as a basis for recommendations of programs to prepare instructional technologists for the future. The present chapter contains a review of the literature of studies in education that have employed the Delphi technique for comparable forecasting; it also contains a review of studies and reports on current professional programs for the preparation of instructional technologists. DELPHI STUDIES IN EDUCATION Researchers generally have utilized two types of Delphi studies - exploratory and normative - to obtain a perSpective of the future. Exploratory studies are concerned with "what may occur and when" (Cetron and Monahan, 1968, p.165). Exploratory studies are analyées of "data in terms of short run as well as long-range potentialities, and relevant environment, in orderto establish a pattern of events and dates for the future" (p.165). Normative studies are concerned with what "should be," with determining desirable goals for the future. Normative studies involve "goals, needs, objectives, or desires ... specified ... and rated... for 13 1” an overall figure of merit" (p.165). Some educators have ex- pressed a need for more normative studies, i.e., goal setting, as prerequisites to setting priorities for scarce resources and for selecting strategies for future growth. Exploratory studies have been criticized for failing to serve as guideposts in plan- ning, since they only point out possible events and dates that may occur (Dressel, 1972). Both exploratory and normative studies employing the Delphi technique are reviewed in this chapter. EXPLORATORY STUDIES USING THE DELPHI TECHNIQUE Two studies found the Delphi technique effective in forecasting future events for educational institutions. The purposes of these exploratory studies were slightly different, i.c., one sought to predict dates events would occur, the other sought to identify needs from a perSpective of future events. The method of each, however, was similar. Participants in the study generated both the events and the possible dates of their occurrence. Berghofer (1970) in a study of general education in post- secondary institutions, systematically selected student and faculty subjects to explore possible dates for future events. Berghofer discovered the subjects were moving dates further into the future as the study progressed. He felt this was a result of two characteristics of the Delphi technique: (1) opportunities for individual subjects to think soberly without interference of 15 others' opinions, and (2) feedback of opinions from fellow participants. The advancing of the dates was considered by Berghofer to be productive. There were differences in the pre- dictions of young and old panelists. The older group seemed to take a more absolute position on the events that were to occur. The younger grOUp appeared to agree less on what events would take place. However, Berghofer synthesized the following con- clusions from respondents opinions in ten problem areas identified in the study: "...(both young and old) reSpondents looked for- ward to a society in which equality of opportunity is emphasized; quality of life is placed above quantity in life; leisure is used creatively; com— munication skills are stressed; concern is shown for major human problems, and a philOSOphic basis is sought for social, cultural, economic and medical changes" (p.19). Berghofer did not-estimate the value of the data gathered to planning for the fUture. The ideas expressed in the above quotation, though desirable, do not provide sufficiently Specific goals for achievement. Seeking to predict vocational educational needs for New York State, HudSpeth (1970) used the Delphi technique in an ex- ploratory study involving professionals selected from the field. In addition to generating events and projecting dates of occurence, the group also rated each event for its value to them personally and to society and in addition identified sources of power and strategies that could encourage or discourage the occurrence of each event. Convergence of opinion on dates of occurrence was achieved for most events and most were seen as more beneficial to 16 society than to the subjects themselves. The subjects agreed on power groups that would encourage or discourage the occurrence of specific events, but disagreed on appropriate strategies to achieve either circumstance. Hudspeth felt an identification of the needs of vocational education in New York was achieved by interpreting the events identifed by participants. No value rating of the needs was undertaken. Though supporting power groups were identified, no direction was gained as to which needs should receive highest priorities. Hudspeth did not discuss how the data he obtained would aid future planning of vocational ed- ucation in New York. Exploratory studies using the Delphi technique have suc- ceeded in identifying possible events and dates of occurrence. Determining what may occur and when, has some value in providing productive information for planning, even though that data is indefinite. Data from exploratory studies provides less positive direction than data from normative studies. Researchers favor- ing normative studies have rejected the idea of forecasting the fhture; rather they have set out to determine alternative worth- while futures in the opinions of groups of people. Researchers in such studies tend to believe that their work.will be more productive than exploratory studies in guiding efforts toward goals generally agreed to be worthy of the necessary expenditures of time, effort and/Or money to achieve. NORMATIVE STUDIES USING THE DELPHI TECHNIQUE In a massive normative study involving thousands of 17 participants across the country, Helmer sought to determine long—range goals for educational innovation in America (Helmer, 1966). Hoping to explore potential applications of the Delphi technique to educational planning, Helmer wanted also to gain a priority list of innovations for expenditures of national re- sources. A long list of innovations was developed and rank ordered. Helmer also found the Delphi technique to be effective in gaining original ideas and a consensus of opinions from edu- cators on goals for their future work. The participants in the study were favorably impressed with the methodology and were eager to apply it to other problems in the future. Also applying the Delphi technique, Clark and Coutts (1971) found that a group of 198 of today's teachers agreed that future teachers need to be skilled not only in the use of technology, but also in individual and group instruction techniques, team teaching and learning principles. The subjects disagreed, however, on the amount of control necessary in teacher education programs and in standards for certification. Cyphert and Gant (1970) used the Delphi technique to survey and clarify opinions on goals from selected members of constituent groups, i.e., alumni, students, faculty, business people, state legislators, of the School of Education of the University of Virginia. Goals were solicited from the subjects and returned to them for value ratings. Cyphert and Gant con- cluded that data generated were useful in establishing a persPec- tive of the institution's priorities for planning. The authors felt this to be more productive for educational planning than 18 previous techniques they had encountered, e.g., face-to—face group sessions. Following the study, many participants expressed a change in attitude from indifference to positive concern for the future of the college. Cyphert and Gant recommended elim- inating the fourth round of questionnaires in the method due to insignificance of data, desirability of getting-on with the work and for convenience of the subjects. Anderson (1970), applied the Delphi technique to gain a perspective of preferred services for a public school district. Participants in the study were selected from groups within the school systems. Results provided the staff with directions for developing resources and made them aware of the complexity of their work. Anderson found that the work of conducting a Delphi study becomes increasingly burdensome as the number of subjects increases. He recommended that teams of researchers be involved in value studies where more than 25 consultants are used. Uhl (1971) conducted a study of five colleges for the Education Testing Service (E.T.S.), to determine goal preferences of students, faculty, administrators, trustees, alumni, parents of students and community leaders. The subjects were arranged into two groups, on-campus and off—campus, and asked to respond to questionnaires on goal items originated by E.T.S. Respondents were instructed to rate the goals for their actual importance at their own institutions as well as to how important each should be. Convergence of Opinion occurred on goal preferences. Uhl felt the Delphi technique to be a valuable aid to planning in higher education because it provides a method for assessing how 19 an institution's various publics feel about its goals. Since it is often not possible to assemble these groups, the Delphi study is a useful method for communicating with them, Uhl concluded. STUDIES OF EDUCATIONAL PROGRAMS FOR INSTRUCTIONAL TECHNOLOGISTS Several recent studies and reports have been made to provide useful information for designing programs of study for students of instructional technology. The findings of some provide a base from which one is able to view and assess needs for programs to prepare instructional technologists of tomorrow. In their extensive survey of colleges and universities, Brown and Thornton (1968) found educational needs being subor- dinated to the technical demands of media equipment. Brown's and Thornton's work apparently pointed up the need to provide media professionals with a broader based training in other fields as well as in audiovisual technology. Brown and Thronton stimu- lated others to investigate the problem and recommend improve- ments in educating instructional technologists. Recognizing the need fer a new type of educator, the "instructional developer," Clark and Hopkins recommended pro- viding such professionals with knowledge to enable them to bridge the gap between research-based educational theory and actual classroom practice. The program of studies included training in: (1) research, (2) educational media, (3) communi- cations, (9) evaluation of strategies of instruction. In addition, Clark and Hopkins believe instructional developers need field experiences in the real world of education so they 2“ may better recognize educational problems, appreciate their rami— fications and learn to communicate more effectively with faculty. Apparently, the instructional developer is more concerned with learning, instruction and message design, than with the demands of media equipment (Hornet, 1970:61-67). Larson (1970) agrees with Clark and Hopkins regarding the need for persons prepared in research and development in instruction. He criticizes most educational media programs for preparing professionals for traditional instructor-centered learning environments, and placing instructional materials in a peripheral role. He found that only a few institutions were preparing students to serve in systems-oriented, student-centered learning environments that integrate instructional materials in planning strategies for teaching (Larson, 1970). He, too, ex- pressed a desire for amatecreative educational experiences for instructional develOpers so as to develop and demonstrate unique methods of instruction. In a manpower study, Fleury, Cappelluzzo and Wolf (1970) criticized the traditional training of educational researchers. They indicate that innovation research demonstrates a need for development and diffusion personnel as well as for researchers in order to bring about changes in educational programs. York (1968) agreed, concluding with more Specific criticisms, "skills in which the least training is being provided are (1) needs assess— ment, (2) long-range planning, and (3) systematic analysis of present" (p. 9-10). Exploring job classifications and competencies in 21 _ instructional media, Wallington and Hyer (1970) recommended that instructional technologists be trained to generalize solutions to instructional problems from one application to another. They also wish them to be able,to design instructional systems that meet the needs of learners and not to consistently focus on producing mediated instructional materials as standard solutions to problems. Beilby, Miller and Murphy (1972) list several conclusions drawn from a conference on curriculum for preparing instructional technologists. A few recommendations by them bear directly on program content. Stating that students of instructional technol- ogy feel insulated from the werld of experience, the authors ad- vodate field experiences as a means of instruction. Since instructional technologists are to be involved in planning, Beilby, Miller and Marphy feel they need experiences in future forecasting. There is also a need to recognize continuing educa- tion as a reality requiring technologists to devise ways of reaching adult learners. Most Significant among their conclusions, however, is that instructional technologists be proficient in solving instructional problems with their clients. SUMMARY Two types of Delphi studies have been employed in educa- tion: (1) exploratory studies that seek to establish a pattern of events and dates for the future and (2) normative studies that seek to Specify goals, needs, objectives, or desires and determine their relative values. Several studies have employed 22 each of the types with success, ‘Exploratory studies in public education, teacher training and vocational education have yielded a convergence of the opinions of subjects regarding probable events and dates of the future. However, exploratory studies have been criticized for failing to provide a valid basis for planning. Such criticism has led to wider use of the Delphi technique in normative studies, resulting in several educational institutiOns having gained directions for planning future growth and development. Normative studies have been employed to assess opinions from Subjects for desirable innovative changes in ed- ucation and to determine preferred goals for some colleges and. universities. The results of normative Delphi studies have pro- duced usefu1 information for planning. Studies of educational programs of instructional technol- ogists, along with reports on the state-of-the-art of instruc- tional technology, have produced information and recommendations useful for the preparation of instructional technologists. It has been found that innovational media practices may, at times, have subordinated educational objectives to the demands of media equipment. Recommendations have been made to provide instructional technologists with (l) backgrounds in communications and learning theory; (2) field experiences in the real world of education; (3) training in educational media; and (9) training in evalua- tion of strategies of instruction. Other findings have revealed that many future instructional technologists are being prepared to serve in traditional instructor-centered, rather than systems- oriented, student-centered learning environments. It has been 23 recommended that proficiency in transferring solutions to instruc- tional problems from one application to another, he somehow ac- quired by neophyte instructional technologists. The ability of instructional technologists to solve instructional problems is, believed to be necessary for designing effective instructional systems. CHAPTER III DESIGN OF THE STUDY INTRODUCTION This study attempts to gain a consensus from acknowledged leaders in higher education on goals for instructional technology over the next 20 years in order to make recommendations for im— proved programs of preparation for instructional technologists. A group of H2 recognized leaders was identified by a pre-selection process described in this chapter and invited to participate. From the 27 who accepted, a list of 73 goals for instructional technology for the next 20 years was obtained. These goals were categorized and submitted to the participants for a value rating on a five—point scale. The goals and ratings were resubmitted to the participants for final review and change or comment. This procedure, known as the Delphi technique, and the manner in which the results were dealt with are further described in subsequent sections of this chapter. SELECTION OF PARTICIPANTS IN THE STUDY Participants in this study were persons with generally recognized high reputations for interest and skills in dealing with.problems of curriculum and instruction in higher education in the United States. 2” 25 An initial selection of 75 persons was made from profes- Sional publications and directories“: and submitted independently to three professors familiar with national leadership in higher education and instructionaltechnology. Three criteria were established for selecting participants who: (1) hold or have formerly held a college or uni- versity professional appointment of high ad- ministrative reSponsibility; (2) hold or have formerly held a position of leadership in a national fbundation or pro- fessional organization devoted to improving curriculum and/or instruction in higher edu- cation; (3) are recognized for scholarship and knowledge of instructional and curriculum problems in higher education through publications, parti- cipation in pertinent national studies and/or through the recommendations of other nation- ally known scholars. From the initial group of 75, 92 persons were agreed upon by two or more of the above professors to be leaders in curriculum and instruction in higher education and were invited to participate *Whg's Who ig,Ame;igan Edugatigg, ngger§,ig,fidugatiog, Directogy g£_the Association for Higher Education, Directory pf the Association for Educational Communications and Technology. 26 in the study. Though leadership in curriculum and instruction in higher education was a characteristic common to the group, they held a variety of professional career positions. Several were college and university administrators, others were scholars and writers in curriculum and instruction, others were associated with na- tional educational foundations or professional organizations. The table below indicates their distribution: Administrators in Scholars & Writers Officers of Higher Education in Curriculum and Foundations and Instruction Professional Organizations 16 13 13 PROCEDURE Two letters accompanied the first questionnaire. One was a statement of support for the project from Dr. Charles F. Schuller (see Appendix A). A second letter from the investigator explained the purpose and procedure of the study, the technique to be employed, and invited recipients' participation (see Appendix B). Participants were asked to list the goals on Questionnaire I (see Appendix C) which they believed would be most important for instructional technologists to achieve in higher education during the years 1972 to 1992. Two weeks were alloted fer completion and return of Questionnaire I. The invitation to participate was accepted by 27 persons. They submitted 199 possible goals. It was necessary to categorize each goal, to avoid duplication of similar ideas. These categories were established after the first round of questionnaires and were 27 as fellows: 1. Definition and Refinement of the Role of Instruc— tional Technology. 2. Economic Support. 3. Diffusion of Instructional Technology. A. Evaluation and Measurement. 5. Development of Non-Human Resources. 6 Development of Human Resources 7. Research. 8. Organizational and Administrative Changes. A naive collator, one formally trained in the field of instruc- tional technology and its nomenclature, but unfamiliar with the categorization system, was employed to confirm the validity of the categories by arranging the 14” goal statements in them. The naive collator performed his task with 83% agreement with the investigator's judgment. Goal statements that were similar in content, but expressed in different terms, were thus identified and a generic goal statement was written. This process yielded 73 goals. These goals were again presented to a second naive col- lator, who categorized them with 86% agreement with the first collator. Questionnaire II (see Appendix D), consisted of a random listing of the 73 goal statements. In a cover letter (see Appendix E), participants in the study were aSked to rate each goal for its importance according to the fOllowing scale: 1 - of extremely high importance, 2 - of’high importance, 3 - of medium importance, ll - of low importance, 5 - of no importance. Two weeks were 28 alloted the reSpondents to complete and return the form. Twenty- three reSponscS were received, from which goal ratings were tab- ulated for convenience and accuracy of computing the mode score. The 23 persons Who completed all questionnaires in the study in- cluded 9 higher education administrators, 8 scholars and writers in curriculum and instruction, and 6 officers of foundations and professional associations. On Questionnaire II (see Appendix F), each subject's previous rating for each goal was indicated on the form. The value rating most chosen by members of the group was also indi- cated for each goal. Participants were requested to review each goal, their ratings and the group's ratings. If they wished, ‘participants could change their ratings, (see Appendix G). Twenty-three sabjects returned Questionnaire III. Each questionnaire was reviewed and the value rating for each goal statement was recorded. ANALYSIS OF DATA The mean score of the value ratings of each goal was computed. This score represented the group’s consensus on the value of each goal statement. The standard deviation of the value rating of each goal was also computed; this score repre- sented the degree to which the value rating of each goal fluc- tuated from the mean score. In addition to standard deviations which clearly Showed a high degree of reliability among most ratings, inter-rater reliability measures, i.e., an analysis of variance of each goal rating, were obtained as a cross-check. 29 In the final tabulation, mean and standard deviation scores were computed for each goal to obtain both the group's consensus of value and the dispersion of ratings for each goal. Statements submitted by each participant to support value ratings he recorded differently from the group’s consensus rating were also recorded. These statements provided minority Opinions on each goal (see Appendix H). Since majority Opinions are re- flected in the consensus rating, separate SUpporting statements were not solicited from participants. Recommendations of learning experiences to meet the future needs of instructional technologists were made on the basis of the goals secured. Goal statements were rank ordered.within categories in order to provide a useful base from.which to make recommendations for the future preparation of instructional technologists. SUMMARY The population of this study consisted of persons with generally recognized high reputations for interest and skills in dealing with problems of curriculum and instruction in higher education in the United States. From an initial list of 75 persons, three consultants selected 42 persons who were invited to participate in the study. Three criteria were established for selecting participants who: (1) hold or have formerly held a college or uni- versity professional appointment of high ad- ministrative reSponsibility; 30 (2) hold or have fermerly held a position of leadership in a national foundation or pro— fessional organization devoted to improving curriculum and/Or instruction in higher edu- cation; (3) are recognized for scholarship and knowledge of instructional and curriculum problems in higher education through publications, parti- cipation in pertinent national studies and/or through the recommendations of other nationally known scholars. Letters explaining the study and inviting participation accompanied the first questionnaire sent to prOSpective partici- pants. Each was asked to submit several goals for instructional technology in higher education over the next 20 years. The invitation was accepted by 27 persons who submitted a total of 199 goals. ‘These were reduced to 73 goal statements by employing a classification system to identify duplicate goals and writing generic statements for them. I The second questionnaire contained the 73 goal state- ments. Each partiCipant was asked to rate each goal on a five- point likert scale. Within two weeks 23 subjects returned the completed questionnaire. The ratings on the questionnaires were recorded and the mode score of the ratings of each goal was computed. The 23 persons who completed all questionnaires in the study included 9 higher education administrators, 8 Scholars and writers in curriculum and instruction, and 6 officers of 31 foundations and professional associations. In tho third round of questionnaires, each participant was asked to review the mode score of each goal and to compare it with his own rating. Participants could either change or de- fend their ratings. All 23 participants returned their question- naires. In the final tabulation, mean and standard deviation scores were computed for each goal to obtain both the group's consensus of value and the dispersion of ratings for each goal. The statements submitted by each participant to support value ratings he recorded differently from the group's consensus rating were recorded. These statements provided minority opinions of each goal. Statements in support of majority opinions were not solicited. Reconmendations for learning eXperiences for future instructional technologists were made from rank ordered listings of the goal statements within categories. CHAPTER IV RESULTS This study undertakes to gain a consensus from acknowl- edged leaders in curriculum and instruction in higher education on goals for instructional technology over the next 20 years in order to make recommendations for programs of preparation of future instructional technologists. This chapter contains a de- scription of the method employed to analyze the data as well as a list of the goals and their value ratings. METHOD OF ANALYSIS OF DATA In analyzing the data secured in this study, the investi- gator was faced with several problems. First, the study is not a sampling study. It is an attempt to elicit statements of signif- icant goals from a small and highly select group of persons. The purpose of the study is to identify what leaders in the field consider important for future development in order to guide de— signers of programs to prepare instructional technologists. Be- cause of the eminent position of the participating leaders in higher education, certain inferences were made from the data in order to recommend learning experiences that will meet the future needs of instructional technologists. In summarizing the data it is necessary to have some measure of central tendency and some measure of disPersion. In classical statistics, the arithmetic mean, the median and the mode are used to determine central tendency; the average deviation, 32 33 standard deviation or point measures of variability (such as quartile deviation, inter-quartile range and percentile points) as measures of diSpersion. Judging from the experiences of other researchers using the Delphi Technique in normative studies, it seemed wise to choose a measure of central tendency that would precisely pin- point group consensus because the ratings of goals have varied relatively little among participants in other studies. Thus, the median score was rejected as too crude for the data and the mode score was also rejected to avoid obtaining bi- or tri-modal scores. Such scores by reason of their range could appear to embrace al- ternative points of view and thus make the data appear ambiguous. Ambiguous results would, of course, reduce their value for de— cision—making purposes. In selecting the mean score it was as- sumed that the intervals between ratings on the scale are constant, that is, the interval between first and second is the same as be- tween second and third, third and fourth, and fourth and fifth. Standard deviation was selected to measure the diSpersion of ratings. Such a measure is necessary in order to establish the range of ratings for each goal and to reveal how closely the group agreed on each value rating. Since it is quite possible for individual ratings at ex- tremes of the scale to strongly influence both the mean and standard deviation scores, inter-rater reliability measures were also made as a cross—check. Inter-rater reliability estimates were obtained by comparing each rating with all the other ratings in the study for each goal statement. These data indicated the 3n degree to which the consensus of value for each goal was influenced by every individual rating. A very high degree of reliability was found in the ratings, indicating that the mean and standard de- viation scores are reasonably accurate measures of the group's reSponses in all cases. The above statistical measures of consensus have been used successfully by numerous researchers who have employed the Delphi technique. This investigator found consistent successful uses of the mean and standard deviation scores as measures of conversion and dispersion in all Delphi studies excepting those involving large numbers of participants. In these cases the standard de- viation score was often replaced by an inter-quartile range score. Statistical treatment is normally used to describe a set of measurements or relations between sets of variables when too large a number of cases is involved for first-hand observation. When the number of cases is small as in the present study, it is possible to present the data in tabular form as well as statis- tically. More importantly it is necessary here to present the data in 923g tabular and statistical form because the goals are an es- sential part of the results since they were originated by the par- ticipants. Because the tabular presentation is somewhat lengthy and difficult to interpret, the goals are presented first by clas- sifications and second by the relative values of the participants on each goal in relation to all the others. In the latter case all 73 goals are arranged in rank order by mean scores. It is important to remember that the purpose of this study 35 was to solicit goals from leaders in curriculum and instruction in higher education in order to recommend learning experiences for future instructional technologists. In the presentation by categories, goals are arranged in rank order of importance with- in each category. The reader is cautioned to keep in mind that the categories in the presentation do not represent greater or lesser importance to the participants. The categories are presented in random order. Subjects of the study did not review or judge the value of goals by categories; indeed, they were never informed that the goals were to be categorized. The categories were set up to facilitate use of the data for the ultimate purpose of the study and to identify and eliminate duplicate goal statements so— licited in the first questionnaire. They have been used in this chapter as an aid to the reader in synthesizing the data. They are also used in Chapter V as an aid in synthesizing recommendations for programs of preparation for instructional technologists. ggesentation 3f goals by Categories In the following presentation, the 73 goals are arranged in rank order of importance within each category. This presenta— tion is intended to aid the reader in synthesizing the data. Evaluation and Measurement STANDARD GOAL MEAN DEVIATION 1. To learn how to use 1.1 0.3% technology to increase, learning significantly. 10. 36 GOAL To develop methods of diagnosis and evaluation of learning problems To develop more suitable methods of evaluation. To revise instructional systems on the basis of field tests and valida- tion. To develop cost-effective measures based on student learning. To formulate, implement and periodically revise quantitative standards in instructional techno— logy. To improve techniques for teaching minority groups. To develop new procedures for cost studies of instruc- tional strategies. To reduce the unit cost of instruction. To use P.P.B.S. as a means of resource control and account— ability. Development 9£_Non-Human Resources 1. To develop alternative self-instructional units fer most students' educational programs. To increase accessibility of instructional materials and equipment. MEAN 1.3 l.” 1.6 2.0 2.0 2.1 2.3 2.7 1.8 STANDARD DBVIATION 0.57 0.65 0.66 0.85 0.60 0.87 0.99 1.03 1.00 0.67 0.76 GOAL 3. To develop skills in users of technology for goal- directed behavior H. To create hardware that is simple and inexpensive to use. 5. To influence manufacturers of technological devices to meet curriculum designer applications with products. 6. To set standards for educa- tional equipment and materials. Development 9f Human Resources 1. To involve students in the design of their own educational programs. 2. To increase numbers of persons in instructional technology for work in institutions on all levels of education. 3. To establish the team concept for instruc- tional technology work on campuses. H. To set national certification of training programs for technologists in higher education. Research 1. To adapt learning Opportunities to different learning styles. 37 2.1 2.2 2.0 2.3 2.7 STANDARD DEVIATION 0.72 0.97 0.96 0.60 0.92 1.01 0.92 1.25 0.62 38 GOAL MEAN STANDARD DEVIATION 2. To conduct research 1.7 0.96 to determine the effectiveness of instructional techno- logy in various physical settings. 3. To conduct research to 1.8 . 0.62 identify appropriate settings for various types of learning. u. To disseminate results 2.0 1.23 of research establishing the instructional value of educational media. 5. To conduct studies to 2.1 1.23 establish the validity of instructional techno- logy. 6. To develop systems to 2.5 0.95 locate and reproduce research materials swiftly, at low costs. Organizational and.Administrative Chggges 1. To develop and refine 1.5 0.85 processes for management of technology. 2. To encourage instruc- 1.8 0.73 tional improvements within the traditional system of higher educa- tion. 3. To utilize instructional 1.9 0.96 systems in academic planning. 4. To assign reSponsibility 1.9 1.09 for instructional techno- logy to the highest possible academic officers. 10. 11. 12. 13. 19. 39 COAL To set accreditation standards for learners instead of institutions. To form multi-state consortia of colleges and universities fer joint efforts in Instructional Development and technology. To change faculty to measure institutional out- puts in terms of student achievement. To direct education toward a role of deliberate change agent rather than an ac- cidental change agent. To specify architectural and environmental condi- tions necessary for instruc- tional technology. To design institutional management systems. To restructure higher education fiscal ap- paratus to eliminate P.T.E. as basis for fund- ing. To provide expertise to enable universities to establish external degree programs. To develop macro-systems to provide any learner with materials to achieve any behavior he desires. To refocus higher educa- tion from technological innovation to social in- novation. MEAN 2.0 2.0 2.1 2.1 2.1 2.9 2.5 2.5 2.6 2.6 STANDARD DEVIATION 0.91 1.09 0.83 0.89 1.09 1.63 0.99 0.93 0.93 15. 17. 18. 19. 20. 21. 22. 23. ”0 GOAL To develop synergistic systems of public school, college and university personnel for joint goal setting and planning. To devise full interna- tional access systems of all instructional materials. To establish independent credit granting office within institutions with power to invest technolo- gically based instruction with as much authority as is vested in faculty. To establish credit and degree granting authorities independent of institutions. To deve10p a statement of student/faculty rights to ensure that technology serves them, rather than the reverse. To adjust laws to protect creative endeavor. To provide up to 80% of instruction in "Open university" type systems. To promote the establishment of a National Institute of Instructional Technology. To orient college education away from occupational competence toward a liberal educa- tion. MEAN 2.7 2.7 2.7 3.0 3.1 3.1 3.1 3.2 3.6 STANDARD DEVIATION 0.89 0.89 1.11 1.09 0.99 0.81 0.99 'l.21 0.96 24. 25. ”1 GOAL To employ instructional technology to enable individuals to acquire a baccalaureate degree by age 18. ' To recommend the use of a voucher system for higher education. Definition and Refinement 9f the Role Q: Instructional Technology 1. To establish human values as paramount in the ap- plication of instructional technology. To conduct research to determine technology ’ s effectiveness in all types of learning. To determine how instruc- tional technology can be applied to enhance learn- ing in the affective do— main. To define the area of instructional technology . To adopt a code of performance and performance standards by a professional national organization for professionals in instruc- tional technology. Economic Support 1. To provide fUnding of local projects of instructional technology. 9.0 1.7 2.9 2.1 STANDARD DEVIATION 0.96 0.72 0.39 0.65 0.87 1.01 0.8u 0.96 u2 2. Tu influence higher educa- 2.3 tional administration to increase budget figures for instructional techno- logy. 3. To establish federal 2.9 support for instructional technology at 1% of the total national expenditure for higher education. Diffusion gf_Instructional Technology 1. To assist in the design of 1.” curriculum and instructional strategies. 2. To design competency-based 1.4 instructional programs. 3. To develop in-service 1.5 programs to train faculty to understand and utilize instructional technology. u. To incorporate a wider 1.7 range of disciplinary resources in instructional technology. 5. To change faculty to 1.8 measure learning ac- cording to achievement. 6. To develop strategies 1.9 to gain faculty adoption of instructional techno- logy. ’ 7. To provide pre-service 2.0 instructors with knowledge of the potential of modern technology for improving learning. STANDARD DEVIATION 1.11 0.89 0.84 1.15 0.62 0.76 10. ll. 12. 13. 14. GOAL To create more awareness in the U.S.0.E. of the value of instructional technology in higher education. To extend instructional technology to public schools. To develop a close bond with those concerned with the expressive and af- fective side of students' growth and development. To integrate instructional technology with library re- sources. To cooperate with scholars abroad toward mutual development of instruc- tional technology. To develop a higher ed- 43 MEAN 2.2 2.3 2.3 2.3 2.6 3.0 ucation division in A.E.C.T. to advance instructional technology. To prepare instructional technologists for work in foreign countries. 3.2 RANK ORDER PRESENTATION Q£_GOALS in rank order of importance according to mean scores. STANDARD DEVIATION 1.13 0.95 1.01 0.65 1.13 0.83 In the following presentation, the 73 goals are arranged This presentation provides the reader with the value each goal held to the study's participants relative to all the goals stated by them. 10. 11. GOAL To learn how to use technology to increase learning significantly. To develop methods of diagnosis and evalua- tion of learning problems. To develop more suitable methods of evaluation. To assist in the design of curriculum and instruc- tional strategies. To establish human values as paramount in the application of instruc- tional technology. To design competency- based instructional programs. To revise instructional systems on the basis of field tests and valida- tion. To develop and refine processes for management of technology. To develop in-service programs to train faculty to understand and utilize instructional technology. To deve10p cost-effective measures based on student learning. To conduct research to determine technology's effectiveness in all types of learning. an MEAN 1.1 1.3 1.4 1.4 1.5 1.6 STANDARD DEVIATION 0.34 0.57 0.65 0.89 0.34 0.79 0.66 0.85 0.84 0.85 0.65 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. GOAL To incorporate a wider range of disciplinary resources in instruc- tional technology. To determine how instruc— tional technology can be applied to enhance learn- ing in the affective domain. To adapt learning op- portunities to different learning styles. To conduct research to determine the effective— ness of instructional technology in various physical settings. To conduct research to identify appropriate settings for various types of learning. To develop alternative self-instructional units for most of students' educational programs. To encourage instruc- tional improvements with- in the traditional system of higher education. To change faculty to measure learning ae- eording to achievement. To utilize instructional systems in academic_ planning. ‘ To develop strategies to gain faculty adoption of instructional techno- logy. us 1.7 1.7 1.8 1.8 1.8 1.9 STANDARD DEVIATION 0.84 0.87 0.62 0.96 0.62 0.67 0.73 1.15 0.46 0.62 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. GOAL To define the area of instructional techno- logy. To increase accessibility of instructional materials and equipment. To assign responsibility for instructional tech- nology to the highest possible academic officers. To develop skills in users of technology for goal— directed behavior. To create hardware that is simple and inexpensive to use. To provide pre—service instructors with know- ledge of the potential of modern technology for im- proving learning. To involve students in the design of their own educational programs. To set accreditation standards for learners instead of institutions. To form multi-state consortia of colleges and universities for joint efforts in instructional development and technology. To formulate , imple- ment and periodically revise quantitative standards in instruc- tional technology. us MEAN 1.9 2.0 2.0 2.0 2.0 2.0 2.0 2.0 STANDARD DEVIATION 1.01 0.76 1.04 0.72 0.47 0.76 0.92 0.91 0.73 0.60 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 1:7 GOAL To disseminate results of research establish- ing the instructional value of educational media. To improve techniques for teaching minority groups. To change faculty to measure institutional outputs in tems of student achievement. To direct education toward a role of de- liberate change agent rather than an ac- cidental change agent. To develop new procedures for cost studies of instructional strategies. To specify architectural and environmental condi- tions necessary for instruc- tional technology. To conduct studies to es- tablish the validity of instructional technology. To influence manufacturers of technological devices to meet curriculum designer applications with products. To provide funding of local projects of instruc- tional technology. To create more awareness in the U.S.0.E. of the value of instructional technology in higher ed- ucation. MEAN 2.0 2.0 2.1 2.1 2.1 2.1 2.1 2.1 2.2 2.2 STANDARD DEVIATION 0.60 0.87 1.04 0.83 0.99 0.84 1.23 0.46 0.72 1.13 42. 43. 1m. 45. 46. 47. 48. 49. 50. 51. GOAL To set standards for educational equipment and materials. To extend instructional technology to public schools. To develop a close bond with those concerned with the expressive and affective side of students' growth and development. To increase numbers of persons in instructional technology for work in institutions on all levels of education. To integrate instruc- tional technology with library resources. To reduce the unit cost of instruction. To influence higher education administration to increase budget fig- ures for instructional technology. To establish the team concept for instruc- tional technology work on campuses. To design institutional management systems. To adopt a code of performance and performance standards by a professional national organization for professionals in instructional techno- logy. 48 MEAN 2.2 2.3 "d 0 U.) 2.3 2.3 2.3 2.3 2.3 2.4 2.4 STANDARD DEVIATION 0.60 0.60 1.01 1.01 1.33 1.03 1.11 0.92 1.09 0.84 52. 53. 54. 55. 56. 57. 58. 59. 60. GOAL To establish federal support for instruc- tional technology at 1% of the total national expenditure for higher education. To restructure higher education fiscal ap- paratus to eliminate F.T.E. as basis for funding. To develop systems to locate and reproduce research materials swiftly, at low costs. To provide expertise to enable universities to establish external degree programs. To develop macro-systems to provide any learner with materials to ac- hieve any behavior he desires. To refocus higher education from technolo- gical innovation to social innovation. To develop synergistic systems of public school, college and university personnel for joint goal setting and planning. To cooperate with scholars abroad toward mutual development of instructional technology. To devise full inter- national access systems of all instructional materials. 40 2.4 2.5 2.5 2.5 2.6 2.6 2.6 2.6 2.7 STANDARD DEVIATION 1.01 0.95 0.99 0.93 1.12 1.01 0.65 0.89 01. 62. 63. 64. 65. 00. 67. 68. 69. 70. 50 GOAL To set national certifi- cation of training pro- grams for technologists in higher education. To use P.P.B.S. as a means of resource control and accountability. To establish independent credit granting office within institutions with power to invest technolo- gically based instruction with as much authority as is vested in faculty. To establish credit and degree granting authori- ties independent of insti- tutions. To deve10p a higher ed- ucation division in A.E.C.T. to advance instructional technology. To deve10p a statement of student/faculty rights to ensure that technology serves them, rather than the reverse. To adjust laws to protect creative endeavor. To provide up to 80% of instruction in "open university" type systems. To prepare instruc- tional technologists for work in foreign countries. To promote the estab— lishment of a National Institute of Instructional Technology. 2.7 2.7 3.0 3.0 3.1 3.1 3.1 3.2 3.2 STANDARD DEVIATION 1.25 1.00 1.11 1.04 1.13 0.99 0.81 0.99 0.83 1.21 51 STANDARD GOAL MEAN DEVIATION 71. To orient college educa- 3.6 0.96 tion away from occupa- tional competence toward a liberal education. 72. To employ instructional 3.8 0.96 technology to enable individuals to acquire a baccalaureate degree by age 18. 73. To recommend the use of 4.0 0.72 a voucher system for higher education. SUMMARY Since both goals and value ratings of them were obtained from the subjects of this study, the data are presented in tabular form. These data include mean and standard deviation scores to display both points of consensus and range of diSpersion for each goal. An analysis of variance of each goal rating indicates a high degree of reliability among ratings. These procedures for the analysis of data are widely used by researchers using the Delphi technique. The goals were first presented by category to aid the reader to identify the various goals in a classification system and to more easily read and synthesize them into coherent group— ings. In the second presentation, all 73 goals are arranged in rank order by mean score; this tabulation illustrates how the participants valued each goal in relation to all the others. Minority statements to support ratings different from the con- sensus rating appear in Appendix G. 52 llllllllllllllllllllllllllllllLJllllll 2.236.535 .309 on nw ow on On n¢ 0? an on ON ON 0. O. n o _ bxb — p _ — — _ p F _ p — — — p p — — _ b — — p — _ p F _ P p p b p _ _ Phb P p p — — — — — _ P p u _ — _ _ _ b p — P b PIP _ p P F _ _ p _ p b l ITIIITTTIIIIIIIYII[[TTIIIIIIIIIIII[IIII q q _ — q q — — _ _ _ u _ — d A u 4 fl — — q q A 4 u d _ — _\4 d d — q 1\4\4‘— 4 - — — — 4 — u - — A 4\A — — q u a — — q _ fi - fl D d 4‘: _ a q — _ Oh mm cm on on n? o¢ an on 0N ON a. o. n 0 £58. 8 .083: USP to .80.th0 g g :50: 034‘. E43454) whogmam >0 3400 ...o mozzhm w34<>uomzoiwo 34024.5 93 mmmoom z of your time to that effort. I vvould not request your c00peration for an ordinary survey, bu}: Mr. Ackerman is a highly capable and responsible in- diwridual and I think the results of his survey may be worth- while to us all. Yonlr assistance will accordingly be appreciated, if you can glare it. The results will be sent to you if you so desire. Cordially, Charles F. Schuller Professor of Education Director, Instructional Media Center CFS/cs 82 APPENDIX B COVER LETTER FOR QUESTIONNAIRE I 83 MICHIGAN STATE UNIVERSITY IN\IHII( ‘IIUNAI. MICHIA (IIINI'I‘R EAST LANSING ' MICHIGAN ' 48821 April 20, 1972 Gentlemen: A new attitude toward future-planning has developed in several sectors of American society, an attitude which indicates that future-planning based on informed, intui- tive judgements about the future may enable man to exert considerable influence over future events, and perhaps to give them direction. Instructional technology faces a challenging future in higher education. Rapid changes on college and univer- sity campuses have placed heavy responsibilities on pro- fessionals in the field. Indications are that more com- prehensive changes will occur at an accelerated rate. What do these developments imply for instructional tech- nology? Where and how should professionals in the field attempt to guide change? How can we best prepare persons to assume positions of responsibility on the campuses of tomorrow? A study is currently underway at Michigan State University to attempt to gain expert agreement on goals for instruc- tional technology in higher education over the next twenty years. The method to be used for this study is the Delphi Technique. Developed in the early 1950's by Olaf Helmer and his colleagues at the Rand Corporation, the procedure obtains a consensus of expert opinion on a given tapic by means of a series of questionnaires, interspersed with information and opinion feedback, rather than face-to-face meetings of such experts. The anonymity of participant response contributes to the effectiveness of the technique. During a preliminary survey to identify persons who could provide significant contributions to this study, one or more colleagues recommended that you be invited to parti- cipate. 84 85 Page 2 The procedure for the study will be as follows: 1. The first questionnaire, which is included with this letter, asks each participant to list those goals he feels are most important for instructional technology to achieve in higher education by 1992. 2. A second questionnaire will randomly list all of the responses of all of the participants and will ask each to rate each item on a 1 to 5 scale, according to his judge- ment of the value of the particular goal. 3. A third questionnaire will report the individual's prior rating and the mode score of the group for each item. Each participant will be asked to review the data and to re-rate any item he wishes to change in the light of the information received. You will receive a duplicate copy of each questionnaire for your files. Please return the enclosed questionnaire by May S, 1972. Subsequent questionnaires will follow at approximately two-week intervals. A final report will be provided to each participant who desires one. We feel that leaders in higher education should provide direction for their profession as well as their institu- tions. Establishing a consensus of desirable goals for instructional technology is a first step in that direction. We believe that our study will provide this information. For this reason we hope you will be willing to participate; we need your expertise to accomplish the task. Thank you very much for your cooperation. Sincerely, Barry L. Ackerman HLAzaf Encl. APPENDIX C QUESTIONNAIRE I 86 'Instructional Media Center MICHIGAN STATE UNIVERSITY Delphi Questionnaire I *Name: Position: *(Names will not be used in published tabulations) Your position of responsibility and recognized professional accomplishments make you uniquely qualified to recOmmend desirable goals for instructional technology in higher edu- cation. Please use the space provided below to list those developments you believe most important for instructional technologists to achieve in higher education within the next twenty years. Possible examples of items you might list would be: (1) Apply concerted pressure on administra- tion to cause development of cost-efficiency studies. (2) Support instructional programs that involve higher edu- cation in social change. These two examples are provided as examples only; no value Judgement is implied. If you wish to make additional suggestions, you may use the reverse side of this sheet. Please return this questionnaire by May 5, 1972, in the envelope provided. We appreciate your cooperation in this study. 1. 87 APPENDIX D QUESTIONNAIRE II 88 n q n N H n q n N H m c m N H m Q m N H n q n N H muaoaaoo OocmuuodaH monouuonaH ouamuuoan oonmuuonaH moamuuoan san on no 30H mo asHvOa mo anus mo mHoaouuxo mo .Ouanaoo on mounaHa me mHoumaonummm oxau vHsono ouHoEEOHumosa use SOHnoua Ono OuEOHvaH OomOHm .vovuos mH uH mm aouH do on maHvaommou .wchumOH OmmouodH >Huamu IHwHamHm ou %onoc£oou on: ou 30: EHOOH OH .hw0Hoanoou HmaoHuoauum naH mo «nounoun HuooH mo waansm OVH>oua oH .uo>mov:o 0>Huuouo uouu noun ou mBoH umafivo OH .hm0Hodnoou mo uaoa Iowoama you mommaooua OaHmOu van QOHO>Ov OH .EEOHusuHunaH mo unavaonovaw OOHuH luonuao maHuaauw ounwov vac uHcouo anHnoumo os mHmn so» CH Amaofluuaaaau cuamaanaa a“ was: up non Haas moamzva "EOHuHmom HH OHHOEEOHumoso HnnHon MHHmMm>HZD madam zumm Immooa mcowunaoo Hmu uaoacouH>aO was Hausa Iouanuum hmwoonm OH H .mcHauoOH mo monk» HHm EH omOco>Huoommo m.hmo IHoanoou uaHauouov ou. summons“ uoavaou 09 H .NNOHonnoou HmaoHuonuumaH no name IQOHO>ov Hmsuna vuma Iou vmouno muuHonow :uHB Oumuoaooo 09 H .ucomm omamso Hmuaov IHoom no coca nosumu uaowm omamno oumumn IHHov mo OHOH m uuwsou aoHumoavu uooqu OH H .EOHuuonvO uoswan mom ousquaonxo HwGOHu Ima_Hmuou man «O NH um NNOHocsuou HmEOHu IoouumoH you unoQQSm Hmuovow anHnmumm OH H .moouoomou mumunHH zuHs hmoHoanoou HEEOHu IosuumnH oumuwouaH OH muaoaaoo moamuuomaH moamuuomflH moamuuodaa.uonmuuonaH monouuomaH nan mHHEE OHAOEO Ou OmHuuunxO OOH>OEE OH .meHquO mEOHuE> EH thHOEEOOu HOEOHanuumEH mo OOOE IO>HuOOmmo any OEHauOuOv Ou EOHOOOOH EOEOEOO OH .EOH>OEOA OOuOOEHOIHmow now HwOHOEEOOu mo anon: EH OHHHJO EOHO>OO OH .OOHwOuOuum HOEOHu loouumEH VEE aEHsOHquO mo EwHOOO onu EH umHmmO OH .HmOHOE Inoou HOEOHuoEHuOEH mo HOHOHHE> can anHnmumo Ou OOHOEOO uOEvEOO OH .mwOHOE Inoou HOEOHuosuumEH ONHH IHuE OEO onumuOOEs Ou huHEOOm EHOuu Ou mamumoum OOH>EOOIEH EOH0>OO OH deou 92 muEOaaOo OOEmuuomaH ooEOqume OOEOOEONHH ooEmqumaH OoEouuomEH nan HHoaouuxo HO asflnaa mo swan no .HmOHOEEOOu HOEOHuonuum IEH ooEO>OO ou .H.o.m.< EH EOHOH>HO EOHuOOEOO EOEwHE o EOHO>OO OH .Oamum Iona HOEOHuOOsOO E3O “Honu mo EmeOO Ono EH muEOOEuO o>HO>EH OH .EHOaOv u>HOOOmmO Ono EH mEHEuOOH OoEOE IEO Ou OOHHEEO on EEO HwOHoEEOOu HOEOHuOEuum IEH to: OEHauOuOO OH .Oamuwoum HOEOHuOEquEH momma thoEOuOano EmHuov OH .wEHEEOHE OEO mEHqun Huom uEHOn you HoEEOOuOE huHouo>HEs OEO OwOHHOO .HOOEOO OHHAEE mo maoumhu OHumHmuOEhm EOHO>OO ofi .momamamo Eo xuoa thHoEEOOu HO IEOHuOoquEH now uEOoEOO anon asp EOHHnOuOO OH deou 93 wdEOaaoo OoEOquEEH ooEmuuome ooEEuummaH uoEEuuomEH OoEOquEaH EMHE OE HO 30H mo EEHEOE no an“: we mHoEOEuEO mo .OmE Ou O>Hm IEOEonH OEE OHEBHO OH umnu ouosvums muoouo OH .OEOOHH two OHEOEOOO OHEHOOOE umOann Ono Ou HwOHOE Inoou HOEOHuOEEumEH mom huHHHnHOEOEmOH EmHmmE OH .mEHEuOOH wEH>OunaH you HwOHOEEOOu Euovoa mo HOHuEOuOE onu mo uvaH IaoEx nuHs muOuOEquEH OOH>EOOIOEE OOH>OEE OH .EOHuOo IEOO EOEwHE mo auumhm HEEOHEHOEuu onu EHEEHB muEOaO>OEEEH HOEOHu IuEquEH OwOuEOOEO OH .mEHEEOHE OHEOEOOO EH maoumhm HOEOHuonuquH ONHHHEE OH .maoumhm mama :huHmuo> IHEE EOEO: EH EOHuoEuumEH mo now ou a: OOH>OEE OH mHHu IOuHHOEv EEO O>HuOuHuEOEU OOH>OH hHHOOHEOHuOE EEO uEOaOHEEH .OuOHEaEOm OH .uEOaEHEUO EEO OHOHHOan HOEOHuoaquEH mo huHHHn IHOOOOOO OOOOuoEH OH .OOu0>OE OEu EOE» EOEEOE .aOEu OO>EOO HwOHOEEOOu EOEu OEEOEO Ou OuEwHu HuHEOOm\uEOEEuO mo uEOaOuOuO O EOHo>OE OH .HmOHOEEOOH HOEOHuOsquEH mo OuEuHuO IEH HOEOHqu O HO uEOa IEOHHnOuOO Onu Ouoaoum OH .OaOuOHO uEOanOEOa HOEOHuEuHuOEH EwHOOE OH .EOHuO>OEEH HOHOOO ou EOHuO>OEEH HOO IHwOHOEEOOu Eoum EOHuOO IEEO EOEwHE OEOOHOE OH .OEOHAOEE mEHEuOOH HO EOHuOEHO>O EEO OHOOEwOHE mo OEOEan EOHO>OE OH madoo n O m N H n O m N H m E m N H m e m N H 5 9 m c m N H m E m N H m c m N H OuEOaaoo OuEOuEQNEH OuEOuEOmaH OoEOuuonaH OoEOquEBH OOEOEEOEEH suH: OE mo 30H mo EEHEOa mo EwHE mo HHOEOEENO mo .OuOOO 30H uO .HHumHam OHOHuOan EOEOOO IOE OoEEOEEOu EEO OuOOOH Ou OEOOOHO EOHO>OE OH .OHOOaOO OHHEEO Ou HwOHoEEOOu HOEOHuOEquEH EEOOEO OH .EOHEOOEEO HOEOnHH O Eanou OoEOuOEEOO HOEOHu IOEEOOO aoum HOSO EOHu IOOEEO ONOHHOO uEOHuO OH .OHOHEOEOE EEO uEOaEHEuO HOEOHuOo IEEO pom OEEOEEOOO uOO OH .HuHHHnOuEEOOOO EEO HOEu IEOO OOEEOOOH mo OEOOE O OO .m.m.m.m OOE OH .HmOHOE IEOOu HOEOHuOEquEH you OOEEme quEEn OOOouoEH On EOHuOEumHEHaEO EOHuOO IEEO HOENHE OoEOEHmEH OH .wEHEuOOH mo OOOHu OEOHHO> pom OmEHquO OuOHuEOEEEO HHHuEOEH Ou EOEOOOOE OOEEEOO OH mHOEE8H OH H .OEOOE HOEOHuOOaEO .OuEOE IEuO mo uOOa mom OuHEE HOEOHuosquEHImHOO O>HuOEEOuHO OOHo>OE OH H .EOHuO IEHO>O mo OEOEEOE OHnO IuHEO ouoa OOH0>OE OH H .EOHu IOOEEO EOENHE EH Hmo IHOEEOOu HOEOHuoEqu IEH mo OEHO> Ono mo .m.o.m.= OEu EH OOOE IOEOSO Ouoa OOOOEO OH H .EOHuOOEEO mo OHO>OH HHO EO OEOHuEuHuOEH EH xuoa Mom HmOHOEEOOu HO IEOHuoEquEH EH OEOOEOE mo OuopaEE OOOOEOEH OH OuEOaaoo OOEOEEOOEH OoEOquOEH OOEOqumEH OoEOquEaH OuEOuuodaH Ean mHEOO OE mo 30H mo aEHEOE mo OOH: mo HHOEOEOEO mo H .HOOHOEEOOO EH uEOa IEOHO>OE HOEOHuOEqu IEH EH Ouuome uEHOw you OOHEHOEO>HEE EEO OOwOHHOO mo OHquOEOO ouOuOIHuHEa auow OH H .EOHuOOEEO EOEwHE you aOuOHO HO509O> O mo OOE OEu EEOEEOOOE OH H .OOHquEOO EwHOuOm EH xuoa you OuOHmOHOEEOOu HOEOHu IoEnuOEH OEOEOEE OH H .Oafiuasm now OHOOn OO .m.H.m OuOEHaHHO Ou OEuOuOE IEO HOOOHm EOHuOoEEO uoEwHE OHEOOEEEOOH OH H .EOHuoEquEH mo uOOO uHEE OEE OOEEOE OH H .HwOHOEEOOu HOEOHu IoEquEH EH OHOEOHOOON Iona How EOHuOOHEOmuO HOEOHuOE HOEOHOOONOHE O Ha OEEOEEOOO OOEOa luomuom EEO ooEOEuom IEOE mo OEOO O EEOEO OH OuEOaaoo OoEOqumaH OOEOuEOdaH OOEOuuQNEH OOEOOEONEH OOOOEHONEH EMHE mqOHEOO uEOEEuO mo OauOu EH OuEEuEO HOEOHuEuHuOEH OHEOOOE Ou HuHEOOm OwEOEO OH .HwOHOEEOOu HOEOHuOEquEH mo OOHO Onu OEHHOE OH .uEan>OHEOO Ou wEH IEEOOOO wEHEuOOH OEEOOOE Ou HuHEuOm OmEOnO OH .OOHwO IuOqu HOEOHEOEHuOEH mo OOHEEuO uOOO How OOEEE IOOOHE 3OE EOHO>OE OH .uEOaEOHO>OE EEO Eusoum .OEEOEEEO mo OEHO O>Hu IOOHHO EEO O>HOOOHEEO Onu EuHs EOEumoEOO OOOEu EuHa EEOE OOOHO O EOHO>OE OH .OuOEEOEE EEHE OEOHu IOOHHHOOEO EOEwHOOE EEH IEOHEEEO OOOa Ou OOOH>OE HOOHwOHOEEOOO mo OEOEEE IOOwEEOE OuEOEHmEH OH mHOE OH .OHEOE HOEOHuOOEEO mo OEHO> HOEOHuoEquEH OEu NEH IEOHHEOEOO EOEOOOOE mo OuHEOOu OuOEHEOOOHE OH .wEHEuOOH uEOEEuO EO EOOOE OOEEOOOE O>Hu IOOmmOIuOOO EOHO>OE OH .EOHuOO IEEO EOEmHE EH OuOHwO IHOEEOOE now OaOuwoun wEHEHOuu mo EOHuOOHm aHquO HOEOHuOE uOO OH .OOHHuO mEHEuOOH EEO IEOHHHE Ou OOHuHEEquE IOO mEHEuOOH EEOEO OH .mH OwO Hp OOquE OuOOuEOHOOOOn O OEHEvOO ou OHOEEH>HE IEH OHpOEO Ou HmOHOEEOOu HOEOHuOOquEH HOHEEO OH HEOU 100 .OOEHOOE OE EOH>OEOE HEO O>OHEOO Ou OHOHEOEOE EuHs HOEHOOH HEO OEH>OEE Ou OaOuOHOIOHOOE OOHo>OE OH .HuHEOOm EH EOuOO> OH OO HuHuozuEO EOEE OO EuH3 EOHuOEquEH EOOOA HHHOOHmOHOEEOOu umo>EH Ou uoaom EOHB OEOHuEuHuOEH EHEuHa OOHH two wEHuEOuw uHEOHO uEOE IEOEOEEH :OHHnOuOO OH .EOHu IOEHHO> EEO OuOOu EHOHm mo OHOOE Osu EO OEOOOHO HOEOHuOsquEH OOH>OE OH .HEOHOEEOOE HOEOHu losquEH EH OOOEEOOOE HEOEHHEHOOHE mo OwEOE EOEHa O OEOEOEEOOEH OH .HwOHOEEOOu HOEOHuoEqu IEH mo EOHuOOHHEEO OE» EH uEEOaOuOE OO OOEHO> EOEEE EOHHnOuOO OH OuEOeaoo OoEOuuomEH ooEOquEaH ooEOuEOEEH OoEOqumEH OoEOquEaH EuHE madou OE mo 3OH mo BEHEOE mo EOHE mo HHOaOuuxO mo 101 m e m N H .maoHuauHumaH mo OOOOOEH OEOEEOOH mom OEEOEEOEO COHUNUfiUUHUUQ ...-0m OH. n O m N H .OHOHEOuOE HOEOHuOEquEH HHO mo OEOuOHO OOOOOO HOEOHu uOEuOuEH HHEH OOH>OE OH OuEOaaoo OuEOquNEH NOEOOEOEEH OoEOqumEH OoEOqumaH MMEOOEQNEH EMHE mqOun OH n O m N H .uo>OOEEO O>HuOOuO uoou noun Ou O3OH uOEfiEO OH m w m N H nwOHOEEOOu mo uEOaOmOEOa mo OOOOOOOEn OEHMOH EEO nOHo>OE OH n O m N H .OEOHuEuHuO IEH mo uEOEEOnOEEH OOHuH luosuEO wEHuEOum OOENOE EEO uHEOuO EOHHEOEOO OH OuEOaaoo OOEOqunEH OoEOuuonaH ooEOuuomaH ooEOuuoan OoEOquan EMHE mHOn Eon «H AOEOHuOHEnOu EOEOHHnEn EH EOOE on uOE HHHa OOaOzVa "EOHuHOon “OaOza HHH OEHOEEOHEOOEO HEnHOn HHHmmu>HZD HHEO EEO HOuEu IOOuHEOEO HHHOOnO OH .mEHEuOOH mo OOnnu HHO EH OOOEO>HuOOmmO O.anHOEEOOu OEHauOuOE Ou EOEOOOOH uOEEEOO OH EnuOHOE IEOOu HOEOHuoEquEH no quanOHO>OE HOEuEa Eanou EOounO OHOHOnOO EuHa OuOuonOOo OH .quuO OuEOEO HOEEOEHOOO EO EOEu EOEOOE quuO OOEOEO OuO IEOEHHOE mo OHOE O EEO3 Ion EOHuOOOEO uOOHHE OH .EoHu IOOEEO EOENHE you OEEu IHEEonNO HOEOHuOE HOuOu Onu «O NH uO nOOHOEEOOu HOEOHuoaquEH now upon InEO HOEOEOH EOHHAOOOO OH JOOOHEOOOE nuOun IHH EuHs HOOHOEEOOO HO IEOHuOEquEH OuOumOuEH OH mado .OaOuwoun OOumOE HOEEOuxO EOHHEOOOO Ou OOHEHOEO>HEE OHnOEO Ou OOHuuonxO OEH>Oun OH m E m N H .OwEHquO OEOHEO> EH HmOHOEEOOu HOEOHuOEquEH mo OOOE |O>Huoowmu Onu OEHEEOEOE Ou EOHOOOOE uoEEEOO OH n O m N H .uOH>OEOA EOOOOHHEIHOOw you anHOEEOOu mo Oman: EH OHHHxO nOHO>OE OH 107 m E m N H .OOHwOuOqu HOEOHuOEEuO IEH EEO EEHEOHHEEO mo EwHOOE OEu EH uOHOOO OH m E m N H .anHOE IEOOu HOEOHuodquEH mo HOHEHHO> an“ :mHHnmumo Ou OOHEEEO uoEEEOO OH m E m N H .anHOEEOOu HOEOHu IOEquEH OOHHHEE EEO EEOuOuOEEE Ou nuHEOOw EHOuu ou OaOumoun OOH>EOOIEH nOHO>OE OH OuEOaaou OOEOqumaH ooEOuuonaH OoEOqunEH OoEOquNEH OoEOuuonaH EMHE mHEO Ou .H.o.m.< EH EOHOH>HE EOHuOO nEEO uOEmHE O nOHo>OE OH .OaOuwoun HOEOHuOOEEO EBO uHOEE mo EwHOOE Onu EH OuEOEEuO O>Ho>EH OH .EHOEOE O>HuOOHMO OEu EH wEH nEuOOH OOEOEEO Ou EOHHnnO on EOO nwOHOEEOOu HOEOHu nOEquEH BOE OEHEuOuOE OH .OaOuw noun HOEOHuOEquEH EOOOn nnoEOuOnaOO EwHOOE OH .wEHEEOHn EEO mEHquO HOom uEHOh uom HOEEOOuOn nuHOuO>HEE EEO ONOHHOO .HOOEOO OHHnEn mo OaOuOHO OHuOkuOEnO nOHO>OE OH .OOOEnEOO EO xuoa nwOHOEEOOu HOEOHu noEquEH uOm unOOEOO EOOu Osu EOHHnOuOO OH mHO EEO OHOOEmOHE mo OEOEEOE nOHO>OE OH .O>HO nEonxOEH EEO OHnaHO OH uOEu OuOsEuOE OuOouo OH .OuOOHw nwo OHEOEOOO OHAHOOon uOOENHE OEu Ou nwOHOE nEOOu HOEOHOOEquEH uom nuHHHnHOEonmou EmHOOO OH .wEHEuOOH wEH>OunEH uom anHOEEOOu EuOEOa mo HOHuEOuon onu mo ONEOH nEOEx EuHs OuOuOEquEH OOH>uOOnoun OEH>Oun OH .EOHuOoEEO uosmHE mo EOE nOnO HOEOHuHEOuu Onu EH nEuHa OuEOaO>OunaH HOEOHu noEquEH owOuaooEO OH .wEHEEOHn OHEOEOOO EH OaOuOHO HOEOHuOEquEH OOHHHEE OH .OEOEOHO Onnu :nuHOuO> nHEE Eono: EH EOHququEH mo now on n: OEH>Oun OH mHH» nO»HHOEu EEO O>H»O»H»EOEO OOH>Ou HHHOOHEOHuOn EEO »EOaOHnaH .O»OHEauOm OH .»EOEnHEOO EEO OHOHuO» nOa HOEOHuonu»OEH mo nuHHHEHOOOOOO OOOOuoEH OH .OOuO>Ou OE» EOE» uOE»Ou .EOE» OO>uOO anHoEEOO» »OE» OuEOEO o» OuEwHu HuHEOOm\»EOEE»O mo »EOEO»O»O O nOHO>OE OH .anHOEEOOH HOEOHuoau»OEH mo O»E»H»O nEH HOEOHqu O «O uEOE nEOHHEOunO OE» O»anun OH .OEOumnO »EOanOEOa HOEOH»E»H»OEH EmHOOE OH .EOH»O>OE nEH HOHOOO O» EOH»O>OEEH HOOHmOHOEEOO» Eoum EOH» nOOEEO uOEmHE OEOOmOu OH mHOE OH .OHOOEOO OHHEEn O» nwOHOEEOO» HOEOH»OEu»OEH EEOuxO OH .EOH»OOEEO HOuOEHH O EuOso» OOEO»OnaOO HOEOH» nOnEOOO aoum HO3O EOH» nOanO OmOHHOO »EOHuO OH .OHOHuOan EEO »EOanHEvO HOEOH»OOEEO uou OEuOEEO»O »OO OH .HuHHHEOuEEOOOO EEO Hou» nEOO OouEOOOu mo OEOOE O OO .m.m.n.n Om: OH .anHOE nEOO» HOEOHuosuumEH uom OOuEme »OwEEE OOOOuoEH O» EOHuOuumHEHaEO EOHuOO nEEO uOEmHE OoEOEHmEH OH .wEHEuOOH mo OOnn» OEOHuO> uom OwEH» n»OO O»OHunounnO HHHuEOEH O» EouOOOOu uonEEoo OH mHOunaH OH .OaOuwoun HOEOH»OOOEO .O»EOEE»O mo »Ooa uom O»HEE HOEOH»OEuumEHnMHOO O>H»OEuO»HO nOHO>OE OH .EOH»OEHO>O mo OEOE»OE OHEO»HEO Ouoa nOHO>OE OH .EOH»OO nEEO uoEmHE EH nwOHOEEOO» HOEOH»OEu»OEH mo OEHO> OE» mo .m.o.m.= OE» EH OOOEOuO3O Ouoa O»OOuO OH .EOHuOOEEO HO OHO>OHA HHO EO OEOH»E»H»OEH EH Euoa uom nwOHOEEOO» HOEOH»OEuumEH EH OEOOuOn mo OuOEaEE OOOOuoEH OH mEEOO 113 O»EOaaoo OoEO»uOnaH OuEO»uonaH OOEO»uOmaH OoEO»uonaH OOEO»uonaH EMHE OE mo 30H mo BEHEOE NO Emu: Oo HHOaOu»xO mo .»EOEnOHO>OE EEO Eusoum O.»EOEE»O mo OEHO O>H» nOOmmO EEO O>HOOOunxO OE» E»H3 EOEuOoEOO OOoE» E»H3 EEOE OOOHO O nOHO>OE OH .O»OEEoun E»H3 OEOH» nOOHmHOOnO uOEwHOOE BEH nEOHquO »OOa O» OOOH>OE HOOHwOHOEEOO» mo OuOuE» nOOmEEOE OoEOEHmEH OH .nwOHOEEOO» EH »EOanOHO>OE HOEOH»OEu»OEH EH O»uONMO »EHOn uom OOH»HOuO>HEE EEO OOmOHHOO mo OH»uOO nEOO O»O»OnH»HEE auom OH .EOH»OOEEO uOEwHE uom aO»OnO uOEOEO> O HO OOE OE» EEOaaOOOu OH .OOHu»EEOO EwHOuOm EH EuoE uow O»OHwOHOEEOO» HOEOHuonu»OEH OuOnOun OH .wEHEEEm uom OHOOE OO .m.H.n O»OEHEHHO O» OE» nOuOnnO HOOOHm EOH»OOEEO uOEmHE OuE»OEu»OOu OH mHHE nEH OHEOEO O» nwOHOEEOO» HOEOH»OEuumEH HOHnaO OH .»EOEO>OHEOO »EOEE»O mo OEuO» EH O»En»EO HOEOH»E»H»OEH OuEOOOa O» H»HEOOM OwEOEO OH 114 In E m N .... m E m N H .nmo nHOEEOO» HOEOH»OEu»OEH mo OOuO OE» OEHwOE OH n e m N H .»EOEO>OHEOO O» mEH nEuOOOO NEHEuOOH OuEOOOa O» H»HEOOm ONEOEO OH m E m N H .OOHmOuOu»O HOEOH»Onu»O nEH mo OOHEE»O »OOO uom OOuEEOooun 3OE nOHO>OE OH OuEOaaoo OOEO»uonaH OoEO»uonaH OoEO»uONEH OuEOuuonaH OoEO»uonaH EuHE mH EEO O»OO» EHOHm mo OHOOE OE» EO OaO»OnO HOEOH»OEu»OEH OOH>Ou OH m E m N H .nwOHOEEOO» HOEOH»OEu»OEH EH OOouEOO nOu nuOEHHnHOOHE mo OmEOu uOEHs O O»OuonuOOEH OH n O m N H .nwOHOEEOO» HOEOH» nusu»OEH mo EOH»OOHHn nnO OE» EH »EEOaOuOn OO OOEHO> EOEEE EOHHEO»OO OH 115 n O m N H .nwOHOEEOO» HOEOH»Osu»OEH mo EOH»nOEO H»HEOOH EHOw O» OOHmOuOu»O nOHO>OE OH n O m N H .OHEOE HOEOH»OOEEO mo OEHO> HOEOH»OEuumEH OE» wEHEOHHEO»OO EOuOOOOu mo O»HEOOu O»OEHEOOOHE OH m E m N H .wEHEuOOH »EOEE»O EO EOOOE OOuEOOOE O>H»OOmmOn»OOO nOHO>OE OH O»EOaaoo OoEO»uOnEH OoEO»uonaH OoEO»uOdaH OoEOuuonaH OuEO»uonaH EuHE mHOOU OE mo 30H mo EEHEOE mo EmHE.mO HHOaOu»xO wo 116 O»EOEEOU OuEO»uonaH OuEOquNaH OoEO»uOnaH OoEOuuonaH OoEO»uonWH EMHE OE HO 30H mo EEHEOE mo Emu: Oo HHOaOuuxO mo .OEOH»E»H»OEH mo EOOuOEH OuOEuOOH uow OEuOEEO»O EOH»O»HEOuOOO »OO OH .OHOHuO» nOa HOEOH»OEquEH HHO mo OEO»OHO OOOOOO HOEOH» nOEuO»EH HHEm OOH>OE OH .OOuHOOE OE uOH>OEOE HEO O>OHEOO O» OHOHuOan E»H3 uOEuOOH HEO OEH>Oun O» OEO»OnOnouOOE nOHO>OE OH .H»HEOOH EH EO»OO> OH OO H»HuoE»EO EOEE OO E»H3 EOH»OEu»O nEH EOOOE HHHOOHwOHOE nEOO» »OO>EH O» anon E»H3 OEOH»E»H»OEH EHE»H3 OOHHHO mEH»EOuw »HEOuO »EOEEOnOEEH EOHHEO»OO OH mH