“hi ‘5”: WIN W \i \ i\\\\\\'\\\\\\ This is to certify that the dissertation entitled CONTRIBUTIONS TO ADULT LEARNING BY COMBINING EXPERT SYSTEMS AND OPTICAL DATA STORAGE TECHNOLOGIES IN COMPUTER-ASSISTED INSTRUCTION presented by Timothy McLaughlin has been accepted towards fulfillment of the requirements for Ph.D. degreein EducationaTSystems Development Major professor [hue November 1992 MS U i: an Affirmative Action/Equal Opportunity Institution 042771 r _~— \ LUMZRY Michigan State University \ 1‘ PLACE iN RETURN BOX to remove this checkout from your record. TO AVOID FINES return on or before date due. I DATE DUE DATE DUE DATE DUE l MSU ie An Affirmative Action/Sous] Opportunity Institution chS-DJ CONTRIBUTIONS TO ADULT LEARNING BY COMBINING EXPERT SYSTEMS AND OPTICAL DATA STORAGE TECHNOLOGIES IN COMPUTER-ASSISTED INSTRUCTION BY Timothy McLaughlin A DISSERTATION Submitted to Michigan State University in partial fUlfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY College of Education 1992 ABSTRACT CONTRIBUTIONS TO ADULT LEARNING BY COMBINING EXPERT SYSTEMS AND OPTICAL DATA STORAGE TECHNOLOGIES IN COMPUTER-ASSISTED INSTRUCTION BY Timothy McLaughlin The purpose of this research was to determine expert judgements about possible technological combinations of expert systems and optical data storage systems that might occur within the next five years (1991 - 1996) and to determine the impact such combinations could have on adult learners if applied in the education and the training fields. This research sought to raise the awareness of hardware and software developers, trainers and educators as to the opportunities that exist for adult learners if instructional systems based on an expert system and an optical data storage medium were introduced into adult education settings. Three rounds of the Delphi technique were conducted with 74, 68 and 67 participants in each respective round with 49%, 49% and 60% return rates. The participants were composed of instructional technology professionals, trainers, expert system researchers and developers, and optical data storage system developers. The major findings from this research were as follows: Participants predicted that it will be possible to combine an expert system with an optical data storage technology (eg. CD-ROM, interactive video, CD—I) in CAI within the next five years. Such an instructional system could benefit adult learners by: 1. providing instruction that is individualized for each learner, creating performance support systems at the work site, increasing the portability of computer assisted instruction systems to the degree that learning would not have to occur in the formal computer laboratory setting. However, it was also noted that the effective application of these combined technologies in an instructional system for adult learners may be hindered by: 1. 2. production costs, the technical and production difficulties of creating such systems, the technical and production difficulties of updating such systems, the lack of cost/benefit analyses which demonstrate the value of these instructional systems. Also noted in this research was the broad understanding across the professional areas of the contributions that could be made to adult learners though combining these technologies. 0f the 64 statements generated from the 153 initial ideas in Round One, only three statements were unique to any one professional area. © Copyright by Timothy McLaughlin 1992 Dedicated to My Mother, Eileen McLaughlin and My son Wesley McLaughlin, as he looks to the future. ‘1 “s D e ACKNOWLEDGMENTS This dissertation is the culmination of the efforts of many people. I would like to acknowledge and thank each of them. First, the members of my doctoral committee: To Dr. Charles Blackman, a special thanks for seeing in me an ability I had not recognized in myself and Providing me with the vision to pursue a higher degree in education. Also appreciated were the gentle nudges to keep 111 cving forward . The hours that I have spent with Dr. Castelle Gentry, my chairperson, in preparation for and the writing of this <3 issertation have been both enjoyable and educational. I <2 annot express enough my appreciation to Dr. Gentry for his “t: :i_me, expertise, guidance and support in achieving this goal in my life. To both Dr. Gentry and Dr. Blackman, thank you for =531::aying with me well beyond the call of duty. Enjoy your 31‘ etirements . Dr. John Vinsonhaler's contributions to my academic SEJTrowth have primarily been through discussions of new T‘Zbechnologies and theories as applied to education. Thank ili’ou for all of the ideas, your support, and more gentle ‘liludges to stay on track. Dr. Lawrence Redd, thank you for joining my committee iSit a crucial time and for your guidance and support. vi I gratefully acknowledge the personnel at the Michigan center for Career and Technical Education for their support and understanding as I progressed toward the completion of this degree. The encouragement from Dr. Gloria Kielbaso and Dr. John MacKenzie helped to draw me back on task. To my mother, Eileen McLaughlin, your faith in me has always been a driving force. Thank you. Finally, to my wife, Yeon Hong Min, thank you for your support and being a single parent over these last few months. Without your love, support, and understanding this project would never have been completed. vii I TABLE OF CONTENTS ISIST OF FIGURES AND TABLES . . . . . . . . . . . . . . x (:flHAPTER I. INTRODUCTION . . . . . . . . . . . . . . 1 Statement of the Problem . . . . . . . . . . . . . 2 Purpose of this Research . . . . . . . . . . . . . 9 Definition of Terms . . . . . . . . . . . . . . . 10 Importance of this Research . . . . 15 Value of this Research to Educational Technology . 19 Theoretical Base for this Study . . . . . . . . 21 Research Questions . . . . . . . . . . . . . 25 Limitations of this Research . . . . . . . . . . 25 Summary and Overview of this Study . . . . . . . . 27 CI-aAPTER II. REVIEW OF THE LITERATURE . . . . .1 . . . 30 Introduction . . . . . . . . . . 30 Adult Learning Theory and Models . . . . . . . . . 30 Malcolm Knowles (Andragogy) . . . . . . . . . . . 31 K. Patricia Cross (CAL) . . . . . . . . . . . . . 33 Howard McClusky (PLM) . . . . . . . . . . . . . . 36 Alan B. Knox (Proficiency) . . . . . 38 Jack Mezirow (Perspective Transformation) . . . . 39 Paulo Freire (Conscientization) . . . . 41 Stephen D. Brookfield (Transactional Dialog) . . 43 Research Supporting the Common Components of Adult Learning . . . . . . . . . . . . . . . . . 45 Expert Systems . . . . . . . . . . . . . . 53 Expert Systems in Education . . . . . 59 The Technological State of Optical Data Storage . . 64 Optical Data Storage Systems in Education . . . . . 72 Computer Assisted Instruction and Adult Learners . 76 The Delphi Research Method . . . . . . . . . . . 79 Summary and Overview of this Study . . . . . . . . 83 QI‘IAPTER III. RESEARCH METHODOLOGY . . . . . . . . . . 86 Introduction . . . . . . . . . . . . . . . . . . . 86 Research Questions . . . . . . . . . . . . . . . . 86 Population of the Study . . . . . . . . . 88 Criteria for Selection Of the Sample . . . . . . . 89 Description of the Sample. . . . . . . . . 91 Development of the Delphi Exercise . . . . . . . . 93 Analyzing the Data . . . . . . . . . . . . . . . . 104 Data Collection Schedule . . . . . . . . . . . . . 106 Summary . . . . . . . . . . . . . . . . . . . . . . 107 viii P’I' Wu 9‘” 1‘ LP! ’1’ D 'l! l K D (:HAPTER IV. FINDINGS . . . . . . . . . . . . . . . . 109 Introduction . . . . . . . . . . 109 Response Characteristics of the Participants . . . 109 Results of Round One . . . . . . . . . . . . . . . 113 Results of Round Two . . . . . . . . . . . . . . . 123 Results of Round Three . . . . . . . 129 Comparison of the HRD & DEV Group Judgements . . . 134 Termination of the Delphi . . . . . . . . . . . 136 Summary . . . . . . . . . . . . . . . . . . . . . . 143 (:ZlfiAPTER V. CONCLUSIONS, OBSERVATIONS AND RECOMMENDATIONS . . . . 145 Introduction . . . . . . . . . . . . . . . . 145 Purpose of the Research . . . . . . . . . . . . . 145 Research Summary . . . . . . 146 Research Questions, Findings, Conclusions and Recommendations . . . . . . . . . . . . . . . 149 Question 1 . . . . . . . . . . . . . . . . . . . 150 Question 2 . . . . . . . . . . . . . . . . . . . 154 Question 3 . . . . . . . . . . . . . . . . . . . 162 Question 4 . . . . . . . . . . . . . . . . . . . 167 Question 5 . . . . . . . . . . . . . . . . . . . 171 Summary . . . . . . . . . . . . . . . . . . . . 177 Final Thoughts . . . . . . . . . . . . . . . . . . 179 APZPENDICES A. Telephone Script . . . . . . . . . . . . . . . 184 B. Round 1 Cover Letter . . . . . . . . . . . . . 188 C. Round 1 Questionnaire . . . . . . . . . . . . 189 D. Cover Letter to Reviewers . . . 193 E. Round 1 Suggestions Offered by Participants . 194 F. Round 2 Cover Letter for Participants Who Signed the Round 1 Questionnaire . . . . . 218 G. Round 2 Cover Letter for Participants Who Did Not Sign the Round 1 Questionnaire . . . 219 H. Round 2 Collated Suggestions . . . . . . . . . 220 I. Round 2 Questionnaire . . . . . 229 J. Ranking the Judgements for the SuggestiOns . . 236 K. Round 3 Cover Letter for Participants Who Signed the Round 2 Questionnaire . . . . . 246 L. Round 3 Cover Letter for Participants Who Did Not Sign the Round 2 Questionnaire . . . 247 M. Round 2 Comments Supporting Judgements . . . . 248 N. Round 3 Questionnaire . . . . . . . . . . . . 258 0. Request for Final Report . . . . . . . . . . . 274 P. Round 3 Judgements . . . . . . 275 Q. Round 3 Comments Supporting Judgements . . . . 285 ElisI.IO<3mn.Pmr.....................293 ix LIST OF FIGURES Cross' Characteristics of Adults as Learners LIST OF TABLES Classification of Optical Disk Technologies Participants' Average Years of Experience by Field . . . . . . . . . . . . . . Response Rate for Questionnaires Response Rates for the Delphi Subgroups Estimated Individual Participation in the Delphi Rounds . . . . . Example of Insights Provided From Round One Suggestions Contributing to Final Statements Distribution of the Participants' Suggestions to the Round Two Statements by Participant Subgroup . . . . . . . . . . . . . . . Distribution of the Participants' Suggestions to the Round Two Groups by HRD and DEV . Examples of Non-grouped Suggestions Judgements In Response to Round Two . . Judgements In Response to Round Three Mann-Whitney Comparison Of the DEV and HRD Groups' Judgements to Each Statement in Round Three . . . . . . . . . . . . . . Stability of Each Statement After Round Three Number of Statements Predicting the Combination Of an Expert System with Optical Data Technologies . . . . . . Comparison of Performance Support System Statements . . . . . . . . . . . . . 33 66 92 110 111 112 114 117 120 122 123 126 131 134 139 153 170 CHAPTER I Introduction When Dustin Heuston was the CEO of WICAT Systems, a major software company, he commented that if a process could improve the productivity Of the 25 students in a classroom rather than being focused on the improvement of one teacher, it would be making an extraordinary contribution (Van Horn, 1991). The implication of this st atement is that, for too long, the focus Of change in educational and training has been on teaching rather than 13 arning. Van Horn (1991) echoed this thought by saying: ...we must concentrate our reform efforts on learners and not on teachers. ... When we invest in new technology, we must invest in new learning systems, not in new teaching systems. The purpose of this research was to determine expert j udgements about possible technological combinations Of e)(pert systems and optical data storage systems that might o':‘—Cur within the next five years (1991 - 1996) and to determine the impact such combinations could have on adult learners if applied in the education and the training EZlelds. In addition, it was hoped that the participants in the study, a number being software developers, as well as §\Jblic educators and corporate training personnel, would be thouraged to develop and/or utilize software products antaining the linkages identified by the study. This chapter addresses the problem under consideration éand the rationale for this study. The theoretical base for 2 this study is developed and limitations of the study are (iiscussed. In addition, definitions of key terms are The chapter concludes with an examination of how given. the study was organized. The Problem Education and training systems have always faced the problem Of matching the instruction to the student. Most obvious, in this match, is identifying what a student knows L1 pen entering instruction and then customizing the learning Less experience to focus on what needs to be learned. obvious is the matching of the instruction to the learning style of the student. Skipper (1985) points out that the mo st effective instruction matches the learner's style. AS 't (1988) supports this by stating: It is reasonable to assume that the learning Of all students would be enhanced if they were taught in a manner conducive to their individual learning style(s). While the earlier research Of Scerba (1979) and MacNeil (1980) indicated that there was no significant increase in learning when learning style and teaching style were matched, the more recent research of Dunn and Griggs ( 1988), Garlinger and Frank (1986), Shelby (1985) and a Eaper presented by Ast (1988) support the matching of learning and teaching styles in order to increase learning I‘Iowever, learning style is composed of elements from Qcagnitive, affective, and physiological aspects, that can combine in infinite patterns. Thus, it follows that an individual's learning style is unique (Ast, 1988). How, ;hen, can even an aware teacher match instruction to each of the students in the classroom? Another issue plaguing education and training is identifying the pace at which the learning is to occur. Too rapid a pace can cause unreasonable pressure and frustration for the learner. Excessive amounts of remediation can destroy the motivational effects Of a s teady challenge (Daniel & Cox, 1989) . Computer-Assisted Instruction (CAI) has, for years, 0 ffered the promise of greater capabilities in being able to analyze the learner's instructional needs, learning Styles, and pacing in an attempt to match the instruction to the learner. Some stand alone computer programs have addressed these areas of analyzing the learner's instructional needs, learning styles, and pacing. Two coI'nputer Managed Instruction (CMI) programs that have been <3eveloped to assist with the assessment of students needs are The Computerized Test of Reading Comprehension and the QQl'nputerized Test of Spelling Errors (Chadwick and Watson, 1 $86) . Other software programs, such as the Learning Styles I«twentory and HILI (How I Learn Inventory), are designed to identify a student's preferred learning style. Cosky ( 3.1980) points out that once a student's learning style is 3~(3entified, instructional strategies, activities, media, 4 23nd assessment used in computer—assisted instruction should rnatch that style. However, a study by Clariana and Smith (1988) indicates that there is a shift in student learning sstyle preference in CAI environments. Consequently, a computer-based program designed for students originally assessed with a particular learning style may not provide <52 Physiological/Aging > > Sociocultural/Life Phases > —-————> Psychological/Developmental Stages Situational Characteristics Full-Time Learning Versus Part-Time Learning Compulsory Learning Versus Voluntary Learning Cross describes her model as follows. Personal characteristics are the variables which describe the learner. The Physiological/Aging continuum describes 34 factors related to age. Along this continuum Piaget's theory of cognitive development might be placed, as well as the decline in vision usually associated with middle age. The Sociocultural/Life Phases continuum is characterized by transitions, such as graduation and moving into the workforce, marriage, and parenting. The Psychological/Developmental Stages continuum is characterized by transitions in the formation of the concept of "I" or "me". This continuum begins with the infant exploring the surrounding environment and establishing what is "me" and "not me" and continues through life as the organization of perceptions defining the "self" (Combs, et al., 1980, p. 8). Cross continues with the situational characteristics, the conditions under which the learning takes place. (These variables) are usually treated as dichotomous because they differentiate adult education from education for children more sharply than other variables, and they provide much of the flavor and distinction of adult education (p. 241). Cross elaborates on this statement by recognizing that neither of the variables under the situational characteristics are true dichotomies. Cross justifies discussing the variable of part-time versus full—time learners as a dichotomy based on the assumption that children and adolescents have a major "full time" responsibility of "going to school" whereas adults have primary commitments to a job and family and schooling is often a secondary commitment. In a similar manner 35 compulsory versus voluntary learning is justified. School is compulsory for children and voluntary for adults. In situations where the adult is a full time learner and/or is under much compulsion to learn, Cross indicates that the situational variable(s) no longer distinguish the uniqueness of adults as learners and should be disregarded in the model. The personal characteristics, in such cases, should still be given full consideration in determining how to treat the adult learner. Thus, the adult learner would always be treated differently from the schoolchild (p. 242). Merriam's components of adult learning can be incorporated into the CAL model continua. First, the components of the adult coming to the learning situation with a bank of experience can be incorporated into the physiological/aging continuum. Cattell (1963) postulated the existence of two intelligences in the individual that illustrate this continuum, fluid and crystallized (Shouksmith, 1970 p. 68). Fluid intelligence is characterized as the ability for new conceptual learning and problem solving, while crystallized intelligence represents acquired knowledge and developed intellectual skills (Eysenck, 1971 p. 54, Knox, 1977, p. 421). As the individual ages, the fluid intelligence of one year contributes to the crystallized intelligence of the next year (Shouksmith, 1970 p. 69). Goleman (1984) supports this idea of an accumulation of crystallized intelligence 36 by stating that research shows a decline in fluid intelligence with age, while crystallized intelligence continues to rise over the life span. Merriam's component of the immediate application of the learning is contained in the transition points along the sociocultural/life phases continuum. The arrival of a baby for new parents might illustrate this point. With this transition to parenthood comes the need to adjust many aspects of the individual's life. Self evaluation determines when the needs no longer exist. Finally, the development of the self-concept in the psychological/developmental stages continuum accounts for the self-directedness of the learner as suggested by Merriam. In 1959 Howard McClusky introduced his theory of Power Load Margin (PLM), also known as the Theory of Margin, with the intention that it would be used for studying adults in order to develop realistic educational programs for adults (Weiman, 1984, p. 1). The basic concept of this model is that in order for adults to participate in education they must possess a margin of resources that can be devoted to the educational process. This margin is determined as the ratio between an individual's total "load" or obligations and the "power" or resources that the individual has to deal with the load. An individual's load is both external and internal. The external factors of the load are the factors imposed on the individual by civic and social 37 obligations, family and work, as examples. The internal factors are the expectations and demands set by one's self, based on values, attitudes, needs and goals. Power is composed of the strengths available to bear the load. As with the load, an individual's power is both external and internal. External power may be composed of positions, allies, financial situation, social standing and other tangible factors. Internal power consists of physical ability, mental ability and the life skills accumulated as one matures (Day a James, 1984, p. 5, MacLean, 1985, p. 46- 47, Weiman, 1984, p. 2). Therefore, the greater the power or the lesser the load the greater the margin of resources which the individual can devote to learning. The life-span studies of Buhler, Henry, and Peck (McClusky, 1963, p. 10), which lead McClusky to develop the theory of Power Load Margin, were concerned with the identification of stages in human development as related to life-span. In identifying these stages it was observed that they marked major changes in life direction for the individual. These major changes in life provide the natural occurrences, for the need to know, the motivation to learn, and the immediate application of the learning. Having the basis for the theory of Power Load Margin (PLM) in life—span studies also relates to Cross' (1981) Physiological/ Aging continuum and Goleman's (1984) concept of crystallized intelligence and, consequently, the acquisition of experience in adults. 38 Finally, McClusky (1963) contends that a person has margin who has the autonomy to choose relevant resources from a broad range of resources in dealing with their personal loads (p. 17). This autonomy and the ability to >choose implies that the theory of PLM recognizes the adult as a self-directed individual, given the necessary resources. In choosing resources the individual would also be evaluating the effectiveness of what is learned from these resources and if others are needed in order to deal with the personal loads. Alan B. Knox introduced his Proficiency Theory in 1980 as another model of understanding adult learners. At the core of this theory is the notion that the individual notices a discrepancy between the current level of some performance and the desired level of that performance, or the level of proficiency (Knox, 1980). In his 1980 article, Knox defined "proficiency" as the capacity of an individual to perform satisfactorily if given the opportunity (p. 378). In 1986 he elaborated on this definition, stating that proficiency emphasizes optimal standards of performance as related to adult life roles. Knox believes this is the distinguishing feature of proficiency-oriented education when compared to competency— based education, which emphasizes only the minimum standards of performance in a task (Knox, 1986, p. 16). Knox recognized that in progressing toward optimal proficiencies, the adult will be building on the current 39 proficiencies (Knox, 1986, p. 16). He also identified several factors that may affect a person's striving to enhance proficiency. These include the individual's physical ability, educational background, values, interests, and self-concept (Knox, 1986, p. 17). Once again, the reader will find that the components of adult learning as mentioned by Merriam are contained within the Proficiency Theory. The basic premise of the theory, that there is a discrepancy recognized by the individual between what is and what is desired, indicates the recognition for a need to learn that is intrinsically motivated and is problem-/life-centered, hence, there is a need for immediate application of the learning. The existence of the experience which an adult brings to an educational setting is acknowledged in the theory as new proficiencies being built upon current ones. And self- concept, which impacts self—directedness and the ability to conduct self-evaluation, is recognized as a factor that can directly affect the success of the individual attaining new proficiencies. When developing the Perspective Transformation theory for adult learning, Mezirow drew on the works of the German philosopher Jurgen Habermas who stated the conditions for free and full participation in reflective discourse. Mezirow believed these conditions were also ideal conditions for adult learning (Mezirow, 1991). As a result, the Perspective Transformation theory for adult 40 learning postulates that learning is a process of, first, critical inward reflection on how and why past experiences have created the assumptions and premises by which a person derives meaning in life (Mezirow, 1981, pp. 6-7). These past experiences may have created limited, distorted and arbitrarily selective modes of perception with regard to 1) the development of the adult self-concept, 2) relationships to others and 3) the way new experiences are given meaning for the individual. This reflection on the assumptions and premises becomes transformative, changing the assumptions and premises, whenever the assumptions or premises are found to be distorting, inauthentic, or otherwise invalid (Mezirow, 1991, p. 6). As a result of the transformation of the assumption or premises, the final step in the Transformation Theory is some action on the part of the individual based on the new perspective (Mezirow, 1981, pp. 6—7). Merriam (1987) describes how the perspective transformation process might typically come about: The process of perspective transformation begins with a "disorienting dilemma" to which one's old patterns of response are ineffective. This situation precipitates a self-examination and assessment of assumption and beliefs. A movement begins whereby one revises specific assumptions about oneself and others until the very structure of [the] assumptions becomes transformed...and results in a new agenda for action (p. 194). The adult learner who is the product of this process should be one who is aware of the assumptions and premises 41 which dictate the way personal meaning is assigned to new experiences, capable of evaluating these assumptions and premises as a result of new experiences and willing to modify these assumptions and premises, when necessary, based on new experiences. In relating Perspective Transformation theory to the four components of adult learning, it is noted that the process is initiated by a personal "disorienting dilemma," which in itself, creates an immediate application for the learning. As it is a process of critical inward reflection on past experiences the components concerning self— evaluation of the learning and the relationship to life experiences are accounted for. And, though Nowak (1981) suggests that the perspective transformation process can be facilitated, it remains largely self-directed, being primarily a mental exercise on the part of the individual. Paulo Freire's commitment to adult learning rests on his conviction that any individual, however ignorant, is capable of critically looking at their life situation (Mashayekh, 1974). The embodiment of this commitment is in the concept of conscientization which refers to the process as one "in which [individuals], not as recipients, but as knowing subjects, achieve a deepening awareness both of the socio-cultural reality which shapes their lives and of their capacity to transform that reality (Freire, 1970, p. 27)." Fundamental to the concept of conscientization is the assumption that humans exist "in" and "with" the world. 42 That humans have the ability to objectify and act upon their world to cause change, not just react to it. That they have the ability to stand apart and analyze not only the present but the past, which has led to the present, and project into the future (Freire, 1970, pp. 28-32). Being able to critically reflect upon the experiences that have shaped one's existence leads to a recognition of the obstacles that have prevented a clear perception of reality (Freire, 1970, p. 51) and allows individuals to understand and change their situation in life. This increased awareness of one's situation results in the individual moving from a level of consciousness where there is no understanding of how forces shape one's life to a level of critical consciousness (Merriam, 1987). Lloyd (1972) stated that Freire's conscientization method had been widely used in a number of Latin American countries. The most notable success being observed in Chile where the illiteracy rate dropped from 15-30% of the population in 1968 to just 5% in six years (p. 4). In looking critically at one's life, which is the mark of conscientization, discrepancies between what is and what is desired will manifest themselves. As with the discussion of Mezirow, above, this type of self examination ultimately leads to learning which incorporates self- directedness and life experiences, has immediate application and is self-evaluated. 43 Brookfield's own research (1980, 1981), supported by that of Thiel (1984) and Pratt (1984) (Brookfield, 1986, pp. 42-43), has resulted in his view of adult learning as a transactional encounter or transactional dialogue "in which learners and teachers are engaged in a continual process of negotiation of priorities, methods, and evaluative criteria" (Brookfield, 1986 p. 20). In this context the responsibility for determining the content of the curricula, instructional methods and how to evaluate achievement is a process shared by both the instructor and the adult learner. Brookfield believes that dialogue must occur to prevent the educational process from becoming entirely instructor or learner driven. In a totally instructor driven setting learners merely regurgitate what has been delivered from the instructor's academic repository. In a purely self-directed setting the instructor is reduced to a role of administrator, publicist and budget specialist. The instructor's insights, views, experience and knowledge are lost in the educational process (Bloomfield, 1986, p. 21). A learning process that is totally self-directed also has the disadvantage of not challenging the learner's paradigms. Bloomfield (1986) suggests that often the most significant learning adults experience is as the result of some anxiety—producing event which causes an uncomfortable reassessment of some aspect of the personal, occupational 44 or social life. Through transactional dialog the function of the instructor can become one of a facilitator who will challenge learners with alternative ways of interpreting their experience and to present to them ideas and behaviors that cause them to critically examine their values, ways of acting, and the assumptions by which they live (pp. 22-23). In relating the four components of adult learning to transactional dialogue it is noted that the adult learner has only partial control or self-direction over the content of the learning. As a result, this may influence the degree to which the learning has an immediate application in the learner's life. The transactional dialogue also affects the self-evaluation of the learning. The adult's perception of the effectiveness of the learning may be moderated by the input from the facilitator. Even though transactional dialogue does not directly address previous life experiences, these life experiences cannot be discounted in the learning process. As discussed in the preceding theories and models, previous life experiences will always act as foundations and filters for new material that is learned. So too, with transactional dialog. 45 Research Supporting the Common Components of Adult Learning The common ground for these adult learning theorists lies in the areas of self-directed learning, incorporation of life experiences into the learning process, self- evaluation and immediate application of the learning. The first of these, self—directed learning, is considered by Merriam and Caffarella (1991) to be "where the learners have the primary responsibility for planning, carrying out, and evaluating their own learning experiences (p. 41)." The importance of self—directed learning in adult education has been noted in several studies. In 1970 Allen Tough conducted a survey to determine how common and important the undertaking of new learning was to adults. Of the 66 individuals interviewed, 98% had taken on at least one learning project within the last year. The survey results indicated that the typical person undertakes about eight learning projects in one year investing between 700 and 800 hours in the learning. Of particular interest was the degree of control that the interviewees exerted in deciding and planning for the learning. A full two-thirds of the learning projects were self-planned (self—directed), with 95% of the interviewees self-planning at least one learning project within the last year. Merriam and Caffarella (1991) note that Tough's 1970 study became the basis for other studies which also verified the existence of self-directed learning among 46 adults. These studies were conducted by Coolican (1973), Bayha (1983), Brookfield (1984), Richards (1986), and Caffarella and O'Donnell (1987, 1988) on a wide variety of populations ranging from mothers with small children to farmers and physicians. Leean's study (1981) could also be added to this list. Cross' observation, made in 1981, that even though the percentage of the adults may vary between studies, the participation of adults in self-directed learning is almost universal (p. 63), still appears valid. Further support for self-directed learning came in a 1989 study conducted by The Personal Adult Learning Lab, part of the Georgia Center for Continuing Education, at the University of Georgia's residential conference center. Data were collected from their clients over a two year period about the nature of the educational interaction between the individual and the computer assisted instructional materials available at the lab. These data were from both staff observations of the clients' interactions with the CAI and survey results from the clients. Analysis revealed that the adult learners regularly wanted to pick and choose their subject from a larger content of materials available. Furthermore, within their subject the learners wanted to be able to adjust the speed and sequence of the presentation, and skip around within the material as their interests dictated. "When the instructional technology did not permit these learning strategies, adult learners became frustrated with the 47 interaction, and the quality of the experience was diminished (DeJoy and Mills, 1989)." Studies such as these have indicated that many, though not all, adults are capable of and prefer self-directed learning experiences. Instructional designers need to keep this variability and preference in mind as new adult education programs are developed. The impact that life experience has on the learning situation is the second common thread linking these theorists together. The importance of combining life (practical) experience with academic studies has been voiced numerous times in the literature (Faure, 1972, Smith and McCormick, 1992, Dewey, 1916, Davies, 1981, Lindeman, 1926. Lawler, 1991). Dewey (1940) stated: We are familiar only with things which specifically enter into our lives and with which we steadily reckon and deal. All concepts, theories, general ideas are thin, meager and ineffectual in the degree in which they are not reflective expressions of acts and events already embodied, achieved in experience (p. 151). Three studies conducted in the 1980's examining the role of previous life experiences in learning, stand out in the literature. Gibbons, et al. (1980) identified several findings with regard to life experience and how it affected the lives of nineteen acknowledged experts without formal training beyond high school and one with only one year of college. Initially, some primary life experience focused these individuals' attention and interest on the particular field of their expertise. Thereafter, all other random 48 experiences were related to this field. Formal education either played an insignificant or negative role in these individuals' lives. Consequently, they developed their expertise through self-directed, experiential, situational and often challenging means. Generalizing the effect of life experience on self-directed learners, Gibbons, et al. (1980) concluded that practical experience is an important factor in self-directed learning from the perspectives of: 1) choosing what is to be learned, 2) implementing a personal learning style, and 3) gaining additional knowledge from the practical application of what was learned. Taylor (1981) was able to identify a common pattern of integrating new learning with previous experience among adult learners. The model developed was cyclic in nature and identified four phases in this process; detachment, divergence, engagement and convergence. Detachment represents a new experience which fits within an existing pattern of experiences or frame of reference. Divergence represents the point at which new learning begins. It is at this phase when an experience does not fit, or diverges from, the existing frame of reference resulting in cognitive dissonance (Festinger, 1964) and discomfort with the new experience. During the engagement phase the learner accepts that the new experience does not fit an existing frame of reference and begins to examine the new experience in greater detail. Gradual insights lead to the 49 convergence phase which is characterized by a major insight or a new understanding to be incorporated into the individual's frame of reference. In summary, experience, as a frame of reference, interacts with new learning to create varying degrees of discomfort in the adult learner. The adult will then progress through a series of phases in order to resolve this discomfort and bring the new learning into their frame of reference. Jarvis' research (1987) on meaningful and meaningless experience acknowledges previous life experiences as part of the individual's biographical history. As new experiences are confirmed within this biographical history the individual's growth is in the form of reinforcing knowledge and beliefs. New experiences which cannot be confirmed within the biographical history cause the individual to reflect on the new experiences and engage in active experimentation with regard to the new experience. Provided the biographical history is adequate, the individual makes an evaluation on the new experience (assigns meaning to it), internalizes it into the biographical history and becomes a more experienced person. The failure to associate the new experience with an adequate biographical history results in the experience being meaningless and results in a failure to learn. Each of these researchers has identified previous life experience as the foundation for new learning. Developing learning strategies which recognize the importance of 50 experience in adult education programs may lead to a more effective use of the personal body of knowledge which adults bring to the learning situation. The third common thread linking these theorists together is the importance of establishing self directed approaches to the evaluation of learner outcomes. As Knowles (1970) explains, there is no greater incongruity in self-directed learning than having an instructor grade a student. Such an act reduces the adult to the level of a child and for the adult it is the ultimate sign of disrespect and dependency. Agee's research (1991) indicates that lifelong adult education programs have been successfully relying on self- assessment for years. These programs have discovered that honoring the experience and maturity of their students also meant being partners in the evaluation process. The resulting dialog between the student and the teacher in establishing the assessment criteria compromised neither the academic standards nor learner identity. It was also noted that during such exchanges teachers often observe how the content and process of learning provides students with the raw materials they need to construct meaning through ‘their individual identities and that students became more aware of how the content and processes of learning extends 'their experience and helps them explore their identities in different contexts . 51 Kopp (1987) points out that "evaluation strategies for adults are most effective when traditional authority roles are de-emphasized, and the learner's role as an autonomous, responsible adult is emphasized" (p. 50). These roles are established through collaboration, or a dialog, between the instructor and adult learner to identify objectives, evaluation techniques and criteria. Kopp reports that the results of constructing evaluation in this fashion benefit adult learners by: 1) identifying current skills and knowledge, which is necessary in order to determine the learning objectives and evaluation techniques, and 2) providing adult learners with a valuable understanding of how they learn. The similar findings of Agee and Kopp stress that during the process of developing self-evaluation learners benefit first by gaining insight into the act of learning. This insight involves the clarification of the difference between their entry level and the desired exit level with regard to what is learned. Second, learners gain insight into their own personal preferences in such areas as learning style or biases in different situations. Strategies in adult learning which recognize the importance of developing self-evaluation may promote the acquisition of these benefits in learners. Finally, each of these theorists believe that the ability to apply new learning to practical life situations is a major factor in adult learning. Several studies, 52 mostly conducted in the 1960's and 1970's, support this position. One of the most recent, a survey conducted by Selz (1979), revealed that being able to apply learning is of prime importance to adult learners. In an earlier study involving 35 adults to determine the major reasons for beginning and continuing a self— learning project, Tough (1968) concluded that, ”The single most common and important reason for adult learning is the desire to use or apply the knowledge or skill (p. 52).” In the same report Tough sites the work of Johnstone and Rivera (1965) in a national survey of adult learners in the United States. This survey concluded that a major emphasis in adult learning was placed on subject matter that was practical, applied and directly useful in performing everyday tasks and obligations. Also cited, was a study conducted by Robinson (1965) which concluded that the majority of interest for learning in adults comes from the personal, practical needs of everyday life rather than curiosity about an academic body of knowledge. One additional study worth note was conducted in France by Dumnazedier (1967) and was aimed at "discovering the attitudes of the self-educated (p. 204)." This study reported that the adult learner's attitude toward topics was utilitarian in nature. The preferred subject matter for these adult learners dealt with matters concerning everyday life. 53 These studies indicate a preference among adults engaged in self-directed study for learning which has a practical application in real life situations. Recognition of this preference in adults may influence how adult learning programs are designed to interact with the learner. Expert Systems As this research is concerned with combining the technologies of expert systems with optical data storage, this section will review the literature which broadly describes the nature of expert systems in order to give the reader a general feeling for their potential abilities. In most fields, human experts could be used much more frequently than they currently are to solve problems and offer advice. But due to the great demand for their services human experts are usually in short supply (Mishkoff, 1985, p. 53, Frenzel, 1987, p. 1) or may be too expensive to engage in solving a problem (Parsaye and Chignell, 1988, p. 1). Forsyth (1986) points out yet another problem with human experts: when they die they take their expertise with them (p. 186). One potential solution to these problems is the expert system, an artificial intelligence computer program specially designed to represent human expertise in a particular subject area or domain (Mishkoff, 1985, p. 54). 54 Storrie-Lombardi (1986) trace the historical roots of expert systems back to the 1940's and 50's with McCulloch, Pitts, and Weiner's work on cybernetics, the use of opposing mechanisms to achieve a range of stability within a system (Parsaye and Chignell, 1988). Cybernetics, in turn, sparked attempts at trying to develop computer systems which would simulate the way the real human neural net works, the beginnings of the research and work on Artificial Intelligence (AI). These AI neural nets would not process data step by step as most computers handled data, but in parallel streams. Developers of these systems foresaw computers capable of dealing with the unforeseen and of synthesizing knowledge from random data with little or no human assistance (Galagan, 1987, p. 76). The complexities of this effort gave way in the 1960's to attempts at building computer systems around symbol- manipulation. In this arena Newell and Simon at Carnegie- Mellon developed a program called the General Problem Solver (GPL) which functioned well within the limits of a set paradigm, but had trouble in expanding beyond those boundaries. Some ten years later Feigenbaum's team at Stanford turned this apparent weakness of GPL into a strength by focusing programming efforts on very specific areas of expertise (Storrie-Lombardi, 1986). Parsaye and Chignell (1988) characterize this developmental sequence as one of the main lessons to be learned from the history of artificial intelligence. That 55 is, general purpose problem solving strategies often failed to achieve an acceptable level of performance. However, by narrowing the scope of the problem, successful "intelligent" systems could be developed (p. 4). These successful "intelligent" systems became known as expert systems. Shoval (1985) defines an expert system as a computer system that: ...performs the job of an expert or consultant in some area, and supports making decisions in unstructured problem situations. A more rigorous definition was offered by Parsaye and Chignell (1988): An expert system is a program that relies on a body of knowledge to perform a somewhat difficult task usually performed only by a human expert. The principal power of an expert system is derived from the knowledge the system embodies rather than from search algorithms and specific reasoning methods. An expert system successfully deals with problems for which clear algorithmic solutions do not exist (p. 1) To accomplish its task, an expert system must be equipped with the same resources as the human expert, and it must be able to follow the same method of reasoning as the human would to reach its conclusion (Hawkins, 1987, p. 95). While the human expert's resources are the accumulated facts, structure and rules in the expert's domain, the expert system's resources are stored in its knowledge/rule base (Frenzel, 1987). Two types of knowledge comprise the knowledge/rule base. The first is 56 factual knowledge or information that can be documented. The second is heuristic knowledge which captures the rule— of—thumb experiences of the human expert (Hofmeister and Lubke, 1986). The items within the knowledge/rule base are related to each other in a definite logical and structured way. However, there may be many different paths through the knowledge/rule base (Hawkins, 1987, p. 95). For the human expert to be able to solve a problem, a method of reasoning must be employed that can extract relevant information from the domain—specific resources. In the same manner, the part of the expert system that conducts a method of reasoning to extract relevant information from the knowledge base is called the inference engine (Parsaye and Chignell, 1988, p. 32). The inference engine is the program that drives an expert system. It establishes the connections among the rules stored in the knowledge/rule base in order to develop a recommendation or conclusion (Hawkins, 1987, p. 95 and Frenzel, 1987, p. 101). To briefly summarize the relationship between the knowledge/rule base and the inference engine in the expert system; the knowledge base contains domain specific content, the inference engine contains the general problem solving knowledge that is applied to the knowledge/rule base in the expert system. As the inference engine is general in nature, it is possible to use the same inference 57 engine with different knowledge/rule bases in constructing expert systems (Parsaye and Chignell, 1988, p. 33). Just as human experts need a way of communicating with their sources of information and their clients so, too, do expert systems need communication channels between their knowledge/rules base and their users. This communication channel is referred to as the user interface. The user interface typically gathers input data from the computer keyboard as the user responds to questions displayed on the monitor by choosing from a list of options or responding in full sentences (Frenzel, 1987, p. 101). Another means of human—machine communication can be through a graphic user interface (Parsaye and Chignell, 1988, p. 33) , where the user inputs data by chosing from icons available on the monitor. It is also possible that the user interface does not link directly to a human at all. The expert system may be embedded within larger application programs. In these configurations the expert system may be invoked as a component of the larger system when a binary request is passed across the interface to the expert system, which in turn produces a binary response that is passed back across the interface to the larger system (Walters and Nielsen, 1988, p. 6, Parsaye and Chignell, 1988, p. 33) Mishkoff (1985) states that there is no such thing as a "standard" expert system. The problem the expert system is designed to solve and how the knowledge engineer approaches the problem impact the design of the final 58 product. However, most authors recognize the three components discussed above as common to all expert systems: knowledge/rule base, inference engine and user interface (Mishkoff, 1985, Frenzel, 1987, Parsaye & Chignell, 1988, Hawkins, 1987). The researcher believes that one additional component should be included in the discussion of expert systems, as Forsyth (1986) calls it, the explanatory interface. Just as human experts may need to explain their recommendation or decisions to a client, expert systems need to be able to justify their actions if requested to do so by the user. Having the ability to demonstrate to the user how it arrived at the recommendations or decisions made on a problem can have the effect of reassuring the user as to the appropriateness of the expert system's performance (Parsaye and Chignell, 1988. PP. 33-34). The explanatory interface may also function as an instructional tool. As users study the expert system's rules for arriving at solutions in the knowledge domain, they may begin to understand the human expert's reasoning process that was used in developing the expert system. 'With.enough explanations and practice, users may become experts (Frenzel, 1987). Another way this explanatory interface can be used is for the expert system developer to check on the rule construction and sequence in the knowledge/rule base (Frenzel, 1987) and evaluate the effectiveness of an expert 59 system by following through its operations (Parsaye and Chignell, 1988, p. 34). Though similar to a database program in its ability to retrieve information, the above discussion differentiates an expert system from a database on one significant point. A database contains only declarative knowledge and so users are expected to draw their own conclusions, based on the information retrieved from the database. An expert system contains not only declarative knowledge but also procedural knowledge and, therefore, is capable of drawing its own conclusions and presenting them to the user (Mishkoff, 1985, p. 54). Expert Systems in Education Little research has been conducted on the advantages of applying CAI with an expert system to adult learning. Specifically, what is missing are the benefits adult learners would gain from such systems in the areas of self direction, incorporating personal experiences into the learning, self evaluation and being able to apply the learning immediately to real-life situations. Oravec (1988) contends that one of the great advantages for adult learners using CAI with expert systems is the ability to receive interactive advice which is tailored to a specific problem (p. 109). This advice may take the form of suggestions on the curriculum content the learner should cover, or when additional review or skipping 60 material may be appropriate. Oravec continues by stating a second advantage of many expert systems is their ability to explain why the advice was given (p. 109). For an adult learner who questions a CAI/expert system recommendation, this capability may be able to provide the rationale necessary to alleviate the learner's concerns. In a broader examination of the research on CAI combined with expert systems in education, Hartschuh (1990) sees expert systems having their most dramatic effect as intelligent tutoring systems. These systems have the capability to guide students through instruction based on the student's strengths and weaknesses (Hartschuh, 1990, "Power On!", 1988). Such a system, also known as intelligent computer-assisted instruction (ICAI), is capable of making decisions during the instructional sequence about which instructional methods to select on the basis of rules provided by the knowledge engineer. Winn (1987) reminds his readers that an instructional system capable of making decisions during the instructional sequence is not a new development. Programmed text and interactive video, for example, depend on a student's response to determine what happens next in the instructional sequence. In effect, once delivery systems were developed that could branch on the basis of student performance, they became capable of making primitive instructional decisions. What distinguishes ICAI from these forerunner systems is the ability of the ICAI to 61 adapt to relevant contingencies within the realm of the instruction and its ability to present information to the student in a variety of formats; text, graphics, and audio. This ability to adapt indicates that ICAI systems must have the capability to monitor instructional situations. Several examples of ICAI systems which demonstrate this ability to adapt, follow. The first is the Minnesota Adaptive Instructional System (MAIS) which was designed to monitor student performance in order to determine how many items a student needs to practice during instruction. The algorithm that guides this decision making process also takes into consideration the relative disadvantages of not letting a student progress even though the subject matter has been mastered, and of letting a student continue without learning the subject matter (Tennyson, Christensen & Park, 1984). The Geometry Tutor monitors how students apply geometric rules in developing proofs. The system's knowledge base contains "ideal" geometric proofs and the various errors that students are most likely to make in developing a proof. By comparing the student's work to its knowledge base the system can determine if a student has applied an inappropriate rule. If an inappropriate rule has been applied, the tutor then takes remedial action (Anderson, Boyle & Yost, 1985, "Power On!", 1988). 62 Kimball (1982) developed the self-improving tutor for symbolic integration which provides another example of the ability to adapt to contingencies. This ICAI system monitors a student's progress in solving a problem by comparison to a model procedure. However, if the student solves the problem effectively in fewer steps than the model procedure the system retains the student's solution for future use. O'Shea's (1982) self-improving tutor for quadratic equations relates different strategies to the achievement of four goals: 1) increasing the number of successful students, 2) increasing the average post-test score, 3) decreasing student time on task and 4) decreasing the amount of time the computer is used. In working with the student the system uses a deductive procedure to determine which strategies are likely to result in achieving one or more of the goals. These strategies are then implemented and the impact on the four goals is statistically monitored. If there is a significant improvement the new combination of strategies is incorporated into the system's production rules. SOPHIE (SOPHisticated Instructional Environment) monitors students in a simulated electronics laboratory as they examine faulted circuits and attempt to problem solve the situation. The system is capable of providing detailed feedback as to the logical validity of a student's proposed solution and can generate counter examples and critiques 63 when the student's hypothesis has flaws in its logic (Barr & Feigenbaum, 1982). GUIDON is an ICAI program for teaching diagnostic problem-solving for infectious diseases. A unique feature of this system is its ability to initiate "opportunistic tutoring." That is, the adaptation and presentation of material, when needed, into the diagnostic dialogue. GUIDON is sensitive to how a tutorial dialog fits together, and what kinds of interruptions and probing are reasonable and expected for the diagnosis under study (Barr & Feigenbaum, 1982, p. 275) As a last example, SCHOLAR is an ICAI system designed to tutor students about simple facts in South American geography. Barr & Feigenbaum (1982) describe SCHOLAR as a mixed-initiative tutoring system, capable of responding to questions as well as initiating a conversation with the student. Such a dialogue allows the tutor to identify the relevant material to be learned, based on the student's previous knowledge, and to identify any misconceptions that the student might have. Winn (1987) points out that ICAI systems, such as these, simulate aspects of the instructional designer's and teacher's expertise which have to do with the selection of appropriate instruction, based on the close monitoring of student interactions. Systems such as those, above, demonstrate that it is possible to have ICAI systems monitor instructional situations in order to prescribe 64 effective instructional methods. This ability to monitor the interaction of the adult learner with the computer assisted instruction will be necessary in systems that are going to be able to adapt to the unique characteristics of adult learners. Adaptations that will need to consider the degree of self-directedness and self-evaluation which produces an optimum level of learning in the adult learner, the personal experiences of the adult learner which could be incorporated into the instructional situation and real— life situations to which the instruction could be immediately applied. The Technological State of Optical Data Storage As this research is concerned with combining the technologies of expert systems with optical data storage, this section will review the literature which broadly describes the technology differentiating one type of optical data storage from another. LaserVision videodiscs introduced in 1980 and compact disks introduced in 1982 were the first commercially available optical disks that appeared on the market (Ullmer, 1989). Since that time other optical formats have appeared and still others are under development. Schamber (1988) observed that these optical disk formats go by many names, e.g. Optical Disk (OD), laser disk and just simply disc, all of which are acceptable. Though most optical disks cannot be erased and lack the access speed of hard 65 disks, these disadvantages are offset by their huge storage capacity, higher level of accuracy and greater durability (Schamber, 1988; Blanchard, Frisbie, and Tullis, 1988). The optical disk is usually either 4.72 or 12 inches in diameter. Some Write Once, Read Many (WORM) optical disks are 5.25 or 8 inches in diameter. A 14 inch diameter disk is offered by some manufacturers and a smaller 3.5 inch disk has been introduced to the market (Dykeman, 1988). Ullmer (1989) characterized the capabilities for optical disks as read only; write once, read many; and write many, read many. Ullmer (1989) also separated optical disks by their method of storing information; analog and digital. Table 1 is a brief classification of optical disk technologies based on Ullmer's classification. Table 1. 66 Classification of Optical Disk Technologies Disk Capabilities Analog Attributes Digital Attributes Peraanent Storage Read Only CAV 54,000 fraaes 30 ain aotion 30 ND LaserVision (Reflective) CLV 108,000 fraaes 1 hr notion 32,400 fraaes 18 sin notion LaserFila (Transaissive) CD-V 20 sin audio 5 sin notion ICVD 10 ain notion CD-ROM 550-850 M0 Storage life 50 yrs Capacity for: audio still fraee text graphics prograeaing inforaation 00-! Capacity for: audio still fraae text graphics prograsaing inforaation DVI 1 hr notion Capacity for; audio still fraae text graphics progransing inforaation Persenent Storage Write Once, Read Many DRAI 1 GB per side CD-IORH 500 N8 Erasable Storage Write Many, Read Many (None Available) .6 - 1 CD Storage life 25 years More than 1,000,000 rewrite cycles CD-EHO .44 - 1 DB Approxiaately 100,000 rewrite cycles Phase Change 67 Ullmer (1989) discusses three types of analog disc formats; reflective, transmissive, and Direct Read And Write (DRAW). The first two systems discussed are read only technologies, reflective and transmissive. Both technologies use a laser beam to read the encoded data on the disc. The reflective technology reflects the laser light off the surface of the disc. Information is encoded on the disc surface in a series of irregular pits. The laser light is weaker when reflecting off from a pit than it is when reflecting off the disc's normal surface. A _photodiode senses these strong and weak reflections and converts them into a standard television signal. The transmissive technology differs from the reflective technology in that it uses a transparent film containing a sequence of dots which encode the information rather than a reflective surface with pits. The varying intensity of the laser light as it passes through the disc is picked up by a photodiode and converted into a standard television signal. These two videodisc formats differ in storage capacities. The reflective technology in a constant angular velocity mode (CAV) can hold 54,000 still images or 30 minutes of real—time motion video, about 30 megabytes of data (Ullmer, 1989, Bradley, 1989). In the constant linear velocity (CLV) mode reflective technology can hold one hour 68 of real-time motion video and 108,000 still images (Helsel, 1990). The transmissive technology has a capacity of 32,400 still frames or 18 minutes of motion. The third videodisc technology Ullmer discusses, known as Direct Read And Write (DRAW), allows a user to record a playable videodisc without going through the expense of creating a "master disc" from which replicas can be made. This technology allows up to 1 Gigabyte of information to be written on each side of the videodisc only once (Schwartz, 1986). The disc cannot be altered once the information is encoded on the surface. Recorded data on the disc may be read immediately after writing. All digital disc formats share three common technical elements. First, all use a reflective laser for reading the information on the disc. Second, the audio and video information is encoded on the disc in a digital format. Third, the discs rotate with a constant angular velocity (CAV). Beyond these commonalties, the technologies vary greatly to provide for different utility in the systems. Ullmer (1989) briefly describes each of the optical disc technologies as follows. Compact Disc-Video (CD-V) offers 20 minutes of digital audio and 5 minutes of full motion analog video on a 4.72 inch disc (Ullmer, 1989). Even though the time available on this format is limited, it may prove useful in the same arena as the single concept filmloop of the 1970's and 1980's. 69 Interactive Compact Video Disc (ICVD) combines interactivity with digital audio and analog video. This format provides up to 10 minutes of video in the CAV mode and 20 minutes in the CLV mode. Three other formats may prove more useful for education and training. The first of these is Compact Disc—Read Only Memory (CD-ROM). A CD-ROM drive is a computer peripheral that functions as a mass data storage device, similar to a hard disk, but capable of storing upwards of 550-650 megabytes (Bradley, 1989, Arnold, 1991). With this amount of storage capacity and an estimated storage life of 50 years (Arnold, 1991), CD-ROM has great potential in business applications, education and libraries. Compact Disc Interactive (CD-I) is another of the CD formats that Ullmer forecasts as having potential applications for education (especially continuing education) and training. The international standards established for CD-I specify that the medium must be capable of carrying not only the information for audio, still video, text, graphics and data on the disc but programming information as well. This will give CD-I players the capability to operate as stand alone machines (Ullmer, 1989, "Power On!", 1988). This stand alone capability will remove the need for an external computer interface with the CD player, allowing it to function in a manner similar to Level II interactive video systems. 70 The third CD format that Ullmer forecasts as having potential applications for education and training is Digital Video Interactive (DVI). By using sophisticated data compression/regeneration techniques, DVI can provide more than an hour of full motion, full screen digital video from a standard CD-ROM configured disc. The DVI format also supports still video, audio, text, graphics and data. Until recently the inability of these systems to produce high quality still and real time motion video was their greatest draw back. However, the development of the Edit Level Video technology has greatly improved the real time motion and by using compression ratios of 2 and 3:1 still images are now suitable for medical training. While analog videodisc still provides the sharpest still images and the smoothest real time motion, DVI is rapidly closing the gap. Write Once, Read Many (WORM) offers the user the ability to record 500 megabytes (Arnold, 1991) of information onto an optical disc. These optical disc files can be edited or copied from a personal computer, but they cannot be erased. All versions of edited materials are saved with the most recent appearing when the user recalls the file. Previous versions of edited files are "archived" and can only be accessed through a read-only mode. (Modern Office Technology, 1989) Schamber (1988) describes the Compact Disk-Erasable Magneto Optic (CD-EMO) format as a combination of optic and magnetic technologies. Like WORM, CD-EMO requires separate 71 read and write lasers. In writing to the disc, Roth (1991) describes the process as one where an infrared laser rapidly heats selected data storage spots on a 20 nanometer thick magnetic material pressed between two polycarbonate layers. Before these spots can cool, a magnetic coil aligns their magnetic field either with the north pole up (digital 1) or the north pole down (digital 0). When light from the read laser strikes these spots it is polarized either clockwise or counter clockwise depending on which way the spot is magnetized. This polarized light is then . received by a photodetector and interpreted as binary data by the disk drive. CD-EMO disks have a storage capacity between 600 MB up to 1 Gigabyte (GB) with an expected life of the storage media that may approach twenty—five years and over one million rewrite cycles (Davis, 1991, Stone, 1991). The technology of phase change involves the changing of a reflective metal film between a highly reflective crystalline state and an amorphous state of lower reflectivity. To create an amorphous spot in the metal film a highly focused laser melts a very small area. The heat is rapidly conducted to the surrounding material and a reflective crystalline structure does not have time to form. By defocusing the laser a larger spot on the disk is heated. The center of this spot cools more slowly because of the surrounding heated material and allows a reflective crystalline surface to form (Bradley, 1989). A photodiode 72 senses the strength of the laser reflected off these spots and converts them into an electronic signal. The storage capacity for phase change technology ranges from 44 Megabytes to 1 Gigabyte (Davis, 1991). Optical Data Storage Systems in Education Optical technology has the ability to "change the way information is stored and delivered in content area disciplines (Blanchard, Frisbie and Tullis, 1988, p. 698)." Rowe and McLeod (1988) believe that only the surface potential of this powerful laser disc medium has been explored especially when optical disc technologies, such as CD-ROM and video discs, are linked together. These linked optical disc technologies in conjunction with hypertext software, which allows linkages between different but related ideas (Halsey, 1989), could allow students to identify materials and develop an easily searchable computerized package that best meets their needs. In effect, students could literally develop their own "books" on a subject. Of the optical technologies that are available to education and training, videodisc and interactive videodisc (IVD) have been available to educators and trainers since the late seventies (Blanchard, et al., 1987), a longer period of time than have the CD-ROM formats. Lookatch (1989) cites that throughout the past decade IVD has been compared to traditional classroom methods of instruction, 73 such as lecture and demonstration, and the results have consistently favored IVD. These results have shown that instruction using interactive video produces achievement in knowledge and skills equal to or greater than that achieved by traditional training methods. These results have also shown that the time needed to acquire new skills and knowledge is reduced when using interactive videodisc instruction. Longitudinal studies indicate a greater retention of the material learned from IVD when compared to traditional training methods. And, the individuals who use IVD system have shown an enthusiastic acceptance of and preference for IVD training over traditional training methods. One interactive videodisc program which has a user base of over 700 sites across the country (Rabe, 1990) is the Principle of the Alphabet Literacy System (PALS). Developed by International Business Machines (IBM), this program is designed to improve the reading and writing skills of illiterate and functionally illiterate adolescents and adults. Recent studies of this system have been conducted by the Bidwell Training Center, Inc. (Njie, 1989), Office of Educational Research and Improvement (Mann, 1989) and the Center for the Study for Adult Literacy, Georgia State University, Atlanta (Nurss, 1989). All have concluded that PALS can be an effective tool in teaching adults to read and write. Njie and Mann reported 74 reading and writing improvements of 2-3 grade levels in only 20 weeks. Bosco and Wagner (1988) conducted a study of the effectiveness of IVD training compared to classroom format using instructional videotapes. The study involved the development of two parallel programs to deliver training to UAW-GM employees on the handling of hazardous materials and was administered to 209 employees from 15 Mid—West GM plants. The results of this study showed that the subjects in the IVD group had a significantly higher achievement than those in the videotape group. With regard to the attitude of the subjects to their training, 22% more of the IVD group felt that their training would help them make safe users of the solvents while 14% more of the IVD group felt the training was interesting. The IVD group took advantage of the ability to set their own pace in moving through the instruction. Times ranged from 20 - 74 minutes with the mean being 33.87 minutes. Finally, 80% of the employees that experienced both types of instruction indicated a preference for the IVD. RAND Corporation conducted research into the effectiveness of IVD used in the advanced individual training of 764 communications electrons specialists at the U.S. Army Signal Center, Fort Gordon, Georgia (Winkler and Polich, 1990). Two studies examined two common applications of IVD in training communications electrons specialists: 1) supplementing hand-on training with IVD and 75 2) simulating hand-on training with IVD. The results concluded that supplementing hand-on training with IVD caused improvements in measures of subsequent task proficiency, specifically, radio installation time was reduced by 15 percent. Replacing hands—on equipment training with IVD training did not diminish students' ability to perform the relevant tasks. Alignment of communications systems was performed as well by the IVD trainees as those trainees who actually practiced with equipment. Compact Disks, as an optical data technology, were introduced for computer applications in 1985 by Sony and Phillips Corporations (Blanchard, et al., 1987). Their application to education and training, in their various forms, is just starting to emerge and be studied. Blanchard, et al. (1988) indicate that most of the databases used by teachers and students are now available in CD—ROM formats. These include Grolier's Encyclopedia, Dissertation Abstracts, Educational Resources Information centers (ERIC), Books in Print, Ulrich's International Periodical Directory, Reader's GUide to Periodic Literature, dictionaries, thesauri, almanacs, writing style manuals, and spelling checkers. However, the research of Gary Marchionini (1989) indicates that novices using CD-ROM databases are severely limited in their use, choosing only the features that have been made explicit to them as 76 defaults or preset options and, consequently, bypass more powerful features. In 1988, Discis Knowledge Research, Inc. began creating Discis Books, interactive computer books on CD-ROM for children. Each Discis Book contains the complete text and graphics of the original book with the enhancements of in-context information for every word and picture element. Music and sound effects are also added for the student's enjoyment. The Discis Books have an audio channel to assist with pronunciation, second language translations, and oral presentation of sentences. Initial research indicates that students who were previously considered nonreaders became more independent readers because they did not have to depend on the teacher for explanations of unknown elements in the story. An increase in story writing and reading of other books was also observed (”Discis Books", 1990). Computer Assisted Instruction and Adult Learners Because this research is concerned with combining expert system and optical data storage technologies in computer assisted instruction for adult learners, a brief review of the research relating the effectiveness of computer assisted instruction to adult learners will be undertaken. Bostock and Seifert (1986) state that from their own research, and that of colleagues, they have been able to 77 conclude that the use of computer assisted instruction with adult learners can improve the quality of adult learning in courses across the whole spectrum of subject areas (p. i). Weller (1988) echoes this belief by stating that computer- based instruction has been demonstrated to be a very powerful technology (p. 23). And, in support of these general comments, Ebner, et al., (1984) indicate that many researchers of computer-assisted instruction have established, through their work, that there is an improvement in learning efficiency that occurs relative to standard instructional methods. That is, the same level of proficiency can be achieved in less time. Galagan (1987) states that James J. L'Allier's research on CAI in training programs has led him to comment that: If you were to compare a traditional form of instruction and that same instruction on a computer, you would find no real difference in the outcome. But the amount of time spent learning on the computer would be half that spent with the traditional method (p. 73). The significance of L'Allier's statement lies in the rapidity with which knowledge becomes obsolete and the frequency of career changes. For example, an engineer's knowledge is out of date three years after graduation and individuals entering the work force today are expected to 78 make at least four career changes, each requiring training (Galagan, 1987. p. 73) Kamm (1983) compared 50 tutorial physics units that were developed to parallel a traditional lecture style physics course based on a mastery model, where students could retake exams until mastery was achieved. For the computer assisted instruction, the results showed a decrease in the number of students who left the course without finishing the 50 units. Also, for those students involved in the computer assisted instruction, the number of retakes necessary to achieve mastery on the units decreased. In a study involving nursing students, Boettcher, Alderson and Saccucci (1981) compared the outcomes of computer assisted instruction against the outcomes of more traditional programmed instruction techniques. They found that in posttest scores and longitudinal follow ups designed to measure retention of knowledge, skills and interest that there were no statistically significant differences between the effectiveness of the two methods. Their conclusion; that computer assisted instruction can be as effective as more traditional programmed instruction techniques for teaching factual knowledge and skills. Deignan and Duncan (1978) compared computer assisted instruction with programmed instructional text in a medical radiology course and to the traditional lecture method in a medical laboratory course. The results indicated a greater 79 knowledge gain for the students participating in the computer assisted instruction than those in the lecture or using the programmed text. It was also reported that time savings on completing the assigned material ranged from twelve to fourteen percent. As Hamilton (1984) suggests, studies such as these do not conclusively indicate that computer assisted instruction is superior to any other instructional methodology (p. 68). Supporting this, a conclusion drawn by Jamison, Suppes and Wells (1974), after an comprehensive study of instructional media, was that there was an overabundance of statistically no significant difference studies with regard to computer-assisted instruction. They continued by stating the belief that computer-assisted instruction had not reached its envisioned role. Perhaps the incorporation of expert system and optical data storage technologies into computer-assisted instruction will fulfill this vision. The Delphi Research Method The RAND Corporation developed and introduced the Delphi technique as a method of data collection in the early 1950's to systematically solicit the views of experts in controversial sociopolitical areas of discourse. However, it was not until the mid-1960's that it came to the attention of individuals outside the defense industry and was applied to technological forecasting (Linstone & 80 Turnoff, 1979, Spinelli, 1983). Broadly, the Delphi technique is a questionnaire method for organizing and shaping the opinion of a number of experts in a field through group feedback. The individuals participating in a Delphi response group are anonymous to one another, with a Delphi administrator collecting and compiling the participants' responses for each round. After each round the administrator returns the compiled responses to the participants along with a new question for a new round. The Delphi usually engages the participants in conjecturing about the likelihood of an event occurring at a particular time in the fUture (Weaver, 1972). The objective is to allow the group to achieve consensus about a given topic (Spinelli, 1983). The experts participate in three or more rounds of questionnaires. Each questionnaire, after the first, provides the participants with compiled data for the entire group. This use of multiple questionnaires with feedback is what distinguishes the Delphi from an ordinary polling procedure. It gives the participants the opportunity to modify or refine their judgments based upon their reaction to the collective views of the group (Mitroff & Turoff, 1975). The Delphi process is terminated after the participants have achieved consensus or stability on the topics. Consensus on a t0pic may be determined by having a certain percentage of the votes falling within a prescribed range (Scheibe, et al., 1975). The observation that people 81 tend to shift their estimates toward a group norm under conditions of iteration is a consistent and solid observation (Weaver, 1972). However, groups of strong- minded individuals may produce a bimodal or flat distribution indicating no strong convergence of opinion. This type of information in a Delphi is no less valuable then that which arrives at a consensus and should be viewed with special interest. Identifying these variations in the Delphi involves a measure not of consensus but of the stability in the participant's vote distribution curve over successive rounds (Scheibe, et al., 1975). As a data collection tool, the Delphi technique has a number of unique advantages. It can be used to forecast when events might occur as the experts arrive at consensus through the use of the repeated questionnaires (Weaver, 1972). Face-to—face contact among the participants is not required. This anonymity prevents professional status and high position from forcing judgements in certain directions --as frequently occurs when panels of experts meet (Weaver, 1972). It also avoids the possibility that personality conflicts and interpersonal politics might affect judgements (Orlich, 1978). Another advantage is that the Delphi technique allows experts to be assembled for a group decision making process that may otherwise be impossible to achieve due to personal schedules or costs if the individuals had to meet at a common Venue . 82 Delbecq, et al., (1986) discuss a further benefit of the Delphi technique. Since the time frame for the rounds [can allow participants several days, or even weeks, to respond, they have the flexibility to respond at times that are most convenient for them. This also gives the participants adequate time to think and reflect upon their responses. The Delphi technique, as with any other research methodology, also has some caveats. First, the manner in which the question(s) and/or statement(s) in each round are written may affect the priority ranking. Value-laden terms cause shifting in the rank upward or downward and result in a biased set of responses (Orlich, 1978). If value-laden terms are used, the researcher must be aware of their purpose in the research. The Delphi technique involves a great deal of data handling between rounds. This may involve the compiling of statements from an open ended question in the first round and editing supporting comments in successive rounds. In performing these tasks the researcher must be aware of the subjectivity that may enter into the editing (Orlich, 1978). Mitroff and Turoff (1975) point out that the judgements which typically survive a Delphi may not be the ”best" judgements but, rather, the compromise position. As a result, the surviving judgements may lack the 83 significance that extreme or conflicting positions may possess. Finally, in cases where there is a lack of consensus on the Delphi items, the method used in making the rankings might shift the priorities. Several techniques may be used to determine rankings, such as, 1) total weighted average, 2) medial ranking, 3) frequency ranking, 4) unweighted frequency, or rank orders. Depending on the technique used an individual item might shift upward or downward in priority (Orlich, 1978). Summary Implementation of adult education programs can benefit from the awareness that many adult learners are deeply imbued with the characteristics of being self-directed, desiring an immediate application for the learning, wanting to self-evaluate the learning and having a large reservoir of experiences which form a context for the learning. As a result, adult learning may take a form more along the lines of a dialogue between the learner and the information provider in determining the approach to the learning situation. As this research involves the integration of expert systems and optical data storage technologies into computer assisted instruction, a discussion of each of these areas was undertaken. First, expert systems are an area of Artificial Intelligence (AI) which focus on solving 84 problems or providing advice within highly defined knowledge boundaries. They are designed to perform the same functions as a human expert would in the same subject area. Several expert systems have demonstrated the ability to monitor the learner and adapt the instruction during CAI. This ability will be necessary if ICAI systems are going to be developed to meet the needs of adult learners. Second, optical data storage systems exist in a variety of formats with their main advantage being the mass storage of data. The main impact that optical data storage has had on education to date has been in the areas of videodiscs, interactive videodiscs and CD-ROM. The effectiveness of interactive videodiscs as an instructional method has proven itself over the last decade. While CD- ROM is still fairly new on the educational scene, it is primarily finding its way into education as large databases of information containing text, graphics, audio, photographs and motion video. Some interactive applications are now being developed. Third, computer assisted instruction applied to adult learning has demonstrated that it is as effective as more traditional instructional methods, such as lecturing or programmed instruction, but may be able to reduce the time needed for instruction significantly. In the last section of this chapter the Delphi research method was discussed. As a means of soliciting ideas from a group of participants about a specified future 85 event and then bringing that group to a consensus on the ranking of those ideas, the Delphi method is a powerful tool. CHAPTER III RESEARCH METHODOLOGY Introduction The purpose of this research was to determine expert judgements about possible technological combinations of expert systems and optical data storage systems that might occur within the next five years (1991 - 1996) and to determine the impact such combinations could have on adult learners if applied in the education and the training fields. The Delphi technique of arriving at consensus on this forecast was chosen since the experts most qualified to address the topic were widely separated geographically and/or employed in separate companies. Being so widely separated it was not economically nor temporally practical for this reseacher to bring these individuals together for a face-to-face group process. Research Qpestions This study attempted to answer five research questions that evolved through three rounds of the Delphi Process. The research questions were: 1. Which combinations of the expert system and optical data storage technologies are most likely to be achieved in the next five years? 2. What contribution(s) to adult learning could the combinations of these technologies be expected to produce? 87 3. Which insights were unique and originated solely from those participants involved in human resource development (HRD), as a group, and those participants involved in the development of the hardware and software technologies (DEV), as another group? 4. Is there a significant difference between the group judgements on the HRD and DEV participants for any of the statements? 5. What are the factors that may negatively impact the effective application of these combined technologies as an instructional system for adult learners? To begin the Delphi process (described later in this . chapter) an initial question was put to the participants as follows: ”Over the next five years, what specific contributions to adult learning can be expected by the combining of expert system and optical data storage technologies in computer-assisted instruction?” Participants were asked to answer this question in two parts (Appendix C). First, by indicating which of the optical data storage technologies will be combined with an expert system; for example, an expert system combined with CD-ROM. Second, the impact that this combination will have on adult learning. Responses to this question formed the basis for the design of the second round questionnaire which asked participants to offer a judgement for each statement on the questionnaire (Appendix I). Participants were asked to: 1. Judge if the statement described or implied some advantageous impact on the adult learner. 88 2. Judge if the combination of technologies described would be combined in the next five years. 3. Offer supporting comments for their judgements. Based on the results of the second round, a third round asked participants to re-evaluate their judgements for each statement after considering each of the following: 1. Whether the statement described or implied some advantageous impact on the adult learner. 2. The possibilities that the technologies described in the statement would be combined in the next five years? 3. Their own original judgement of the statement. 4. The group judgement value for the statement. 5. The supporting comments offered by the group for the statement. Population of the Study The population for this study was composed of computer scientists researching artificial intelligence/expert systems, optical data storage developers, expert systems developers, industrial training professionals, and instructional technology professionals. Only individuals who had at least one year's experience with.expert systems, optical data storage systems, and adult learners were included as participants. This limit was imposed as it was assumed that only participants with a year's experience in each of these areas would have the understanding of the systems and concepts necessary to respond to the survey. 89 Criteria for Selection of the Sample A research sample of 74 subjects was selected. The total sample, though heterogeneous, was made up of five homogeneous groups: Expert Systems developers; optical data storage developers; professionals involved with the designing, planning and managing of training programs; instructional technology professionals; and researchers in the field of artificial intelligence/expert systems. Delbecq, Van deVen and Gustafson (1986) suggest that if participants within these homogeneous groups are well- chosen, 10 - 15 participants would be enough to generate a full range of ideas, with few new ideas being added once the group size exceeds thirty. The computer scientists were selected from the Expert Systems membership list of the American Association for Artificial Intelligence (AAAI). The optical data storage and expert system developers were selected from companies identified using the Training Marketplace Directory. This included companies such as Allen Communication, BCD Associates, Paperback Software, WICAT Systems, Inc., and National Educational Consulting, Inc. The industrial training professionals were drawn from the Industrial Training and Education Division (ITED) of the 1990 membership list of the Association for Educational Communications & Technology (AECT). The instructional technology professionals were identified through the Instructional Technology Professional Practice Area of the 90 1990 membership list of the American Society for Training and Development (ASTD). The only two criteria imposed on the sample subjects, as a whole, were that the individuals were currently involved in their field of work, as identified for this Delphi, and that they had dealt with expert systems, optical data storage systems, and adult learners for at least one year. The researcher felt that these requirements were necessary to ensure that participants had adequate exposure to their fields. To ensure diversity in the responses, only one participant was selected from the same company or institution except for the companies involved in developing expert systems, where the maximum was two participants. (The rationale for this is explained below.) Samples from each group were selected using the following techniques. The ITED membership list contained 261 individuals. Each seventeenth member was selected to ensure one complete pass through the list. If an individual declined to participate in the Delphi or did not meet the one year criterion the second pass through the list began on the eighteenth member and then proceeded with each seventeenth member. This process was repeated until the requirement of 10 - 15 individuals had agreed to participate in the Delphi. The Instructional Technology Professional Practice Area of the ASTD membership directory contained 3,104 individuals. The procedure for selecting participants from this list was the same as for the ITED 91 membership list, only an interval of 207 was used. The AAAI Expert Systems membership list contained 11,000 individuals. Each 740th member was selected to ensure one complete pass through the list. As the list of companies involved in expert systems development contained only 12 firms it was decided to ask these 12 firms for referrals to other companies developing expert systems in an attempt to find participants from different companies. A roll of a die determined the interval for selecting from the original 12 Expert System companies to be three. The second pass through the list would start on the fourth company with the same interval, and so on. Four of the Expert Systems companies were also involved in optical data storage technology. These four companies were not included in the selection list for optical data storage to avoid biasing the opinions collected. As such, 44 firms were on the optical data storage list. Again, one individual was sought from each company. A roll of two dice determined the interval to be nine. Description of the Sample Three hundred forty nine phone calls were placed to secure the 74 participants. 16 were Expert Systems developers; 12 optical data storage developers; 16 professionals involved with the designing, planning and managing of training programs; 19 instructional technology 92 professionals; and 11 researchers in the field of artificial intelligence/expert systems. Table 2 shows the average number of years experience each of the participant groups possessed with expert systems, optical data storage technologies, and adult learning. Table 2. Participants Average Years of Experience by Field Expert Optical Data Adult Group Systems Storage Learning Expert System 7 4 10 Developers Expert System 1 6 3 4 Researchers Instructional Technologists 6 8 11 Optical Data Storage System 7 9 16 Developers Trainers 4 5 16 Collectively 6 5.8 11.4 Collectively, the group averaged 6 years experience with expert systems, 5.8 years experience with optical data storage system technologies, and 11.4 years experience in working with adult learners. 93 Development of the Delphi Exercise The Delphi Exercise for data collection can be characterized as a method for structuring communication within a group (Linstone & Turoff, 1979) for the purpose of allowing the group to develop a clearly defined and convergent pattern of major points while maintaining the minority opinion in a well-outlined form (Orlich, 1978). The participants in a Delphi are usually anonymous to each other and do not meet face-to-face. The Delphi administrator serves as their mutual contact. This structured communication may begin with the participants, independently, expanding on or exploring in greater detail the subject under discussion. This is followed by the members of the group expressing their degrees of agreement or disagreement and their supporting rationale for the views developed while exploring the subject. The communication is concluded when all previously gathered information has been analyzed by the group and a consensus of opinion is indicated. This study was an attempt at forecasting some possible effects on instructional technology for adult learners over the next five years as the result of combining expert systems and optical data storage technologies in computer— assisted instruction. The Delphi technique was-chosen as the tool to gather this information because, as Weaver (1972) points out, the purpose of the Delphi is to engage experts in conjecturing about the likelihood of an event 94 occurring in the future. Other considerations were also taken into account when choosing the Delphi for use with this group. Three of the five participant groups (trainers, instructional technologists, optical data storage developers) in this study might have been adversely affected by the necessity of a face—to-face group meeting. Using the Delphi, avoided the possibility that recognized leaders in the field of Artificial Intelligence would dominate other group members during the idea generating process. The anonymity afforded by the Delphi process prevents certain individuals from dominating the process in this manner. The participation of the expert systems developers and the optical data storage developers may also have been affected by a face-to-face meeting. As these individuals were involved in commercial product development, knowledge that a competitor is part of the group could have made them more reticent to share ideas. Again, the anonymity of the Delphi process would alleviate this problem. Finally, it was not economically nor temporally practical for this researcher to bring all of these individuals together for a face-to-face group process. The data collection effort for this research involved three questionnaires that comprised the Delphi Exercise. In the first questionnaire (Appendix C), the researcher stated the initial question for the participants to explore and expand upon. In the second questionnaire (Appendices H 95 & I), the researcher asked participants to offer a judgement and supportive reasoning on a collated list of responses from question one. These judgements were to be based on the possibility that the technologies described in the response will be combined in the next five years and will provide an advantageous impact on adult learners. Judgements were offered on a six-point Likert scale offering the following choices: 6 = Advantage for adult learners, Will be achieved technologically 5 = Advantage for adult learners, Possible to achieve technologically 4 = Advantage for adult learners, Impossible to achieve technologically 3 = No advantage for adult learners, Will be achieved technologically 2 = No advantage for adult learners, Possible to achieve technologically 1 = No advantage for adult learners, Impossible to achieve technologically Five responses to the first questionnaire suggested that combining expert system and optical data storage technologies would have negligible or no impact on adult learners. This area required a separate five—point Likert scale in order to establish group agreement or disagreement with these suggestions. This second Likert scale offered the following choices: 5 = Strongly Agree 4 = Agree 3 = No Opinion 2 = Disagree 1 = Strongly Disagree In the final questionnaire (Appendices M a N) a ranked list was compiled for the group and returned to the 96 participants. They were, once again, asked to offer a judgement on the same six-point and five-point Likert scales as in the second questionnaire for each item and to provide arguments supporting their judgements. Participants' judgements this time should be based on: 1) the possibility that the technologies described in the response will be combined in the next five years, 2) the combination of technologies will provide an advantageous impact on adult learners, 3) the individual's original judgement and reasons, 4) the group judgement value, and 5) supporting comments offered by the group for each item. An account of the development process for each questionnaire is detailed on the following pages. Round One Question: The purpose of this round was to identify how expert systems and optical data storage systems might be combined in the next five years and what impact these combinations might have on adult learners. This question also involved the development of a telephone script (Appendix A) to guide the researcher in the initial contact with a potential participant, and a cover letter (Appendix B) to accompany the first question (Appendix C). These materials were pilot tested by 5 individuals, 1 from each of the five respondent groups. Each reviewer was approached as the actual participant would be, with the initial telephone script. The exception being that they were aware of the pilot test nature of 97 their participation. The reviewers were sent the Question 1 Cover Letter, Question 1 with instructions, and a Pilot Test Cover Letter (Appendix D) soliciting the following feedback: a) Does the initial telephone conversation prepare the individual for this Delphi exercise? b) Does the cover letter provide enough detail as to the structure of the study? (Who is participating, how many individuals are involved, the timelines.) c) Do the instructions provide adequate background for the Delphi question? d) Is the purpose of the study clearly understood? e) Does the layout of the Delphi response form seem logical? f) How much time was needed to complete this Delphi response form? Results Of The Pilot Test: All of the reviewers commented that the topic of the study was of interest to them. This gave the researcher an indication of the study's value to individuals involved in developing Expert System and optical data storage technologies, conducting research into Expert Systems, and in developing learning systems. In examining the Round One pilot question materials, all of the reviewers indicated that the telephone conversation, cover letter, and instructions fulfilled 98 their objectives. The purpose of the study was clear to all and the structure of the Delphi response form posed no problems. Four of the reviewers completed the response form within the estimated 20 - 30 minutes. One reviewer required 45 minutes to complete the form, but indicated that some text research was included in that time in order to develop his ideas. Two additional comments resulted in modifications to the pilot materials: a) Three of the reviewers interpreted ‘formal education' in the Delphi instructions to exclude training situations. As such, ‘training' was added to the first sentence. b) Two reviewers mentioned that this study would require participants knowledgeable in the areas of expert system and optical data storage technologies, and adult learning. Item 3c was added to the telephone script to help screen for these individuals. With these edits, the development of the Round One Question was finalized and sent to participants. Round Two Question: The purpose of this round was two fold. First, to clarify the suggestions obtained in Round One through supportive statements and criticisms. Second, to establish a preliminary ranking of which technologies 99 would most likely be combined in the next five years and have the greatest impact on adult learners. The participants responded to the Round One question with 153 suggestions regarding the ways that expert systems could be combined with optical data storage technologies and the resulting impact on adult learners. The researcher constructed a composite list of these 153 suggestions (Appendix E). Because 153 suggestions would be an excessive number for most participants to offer judgements on in the second round, a process of systematic [elimination was used to reduce this number. Suggestions were combined and grouped according to the following criteria: a) Group suggestions based on their impact on adult learners. Suggestions were grouped together based on the impact these technologies might have on adult learners. Many of the suggestions contained compound ideas as to the impact the combining of these technologies might have on adult learners. These compound ideas were extracted from the original suggestion and placed in a group of related ideas. As such, a single participant's suggestion could contribute ideas to more than one group. When this sorting was complete, 26 groups had been formed. b) Group non-related suggestions. One of the 26 groups was composed of 17 suggestions that did not C) d) 100 seem to relate to any of the other suggestions based on the impact that combining these technologies might have on adult learners. After reviewing each of these 17 suggestions, the researcher decided to retain them for the second round as each seemed insightful and had the potential to stimulate ideas in the participants. Combine duplicate suggestions within groups. Within each of the remaining 25 groups, duplicate suggestions were combined into a single statement. Suggestions within an area typically varied with regard to how the expert system and optical data storage technologies were combined. In the case of the three groups: IDENTIFIES STUDENT'S LEARNING STYLE, PORTABILI'I‘Y, and CREATES PERFORMANCE SUPPORT SYSTEM, the resulting impact on adult learners also varied. Make statements grammatically correct. Remaining suggestions were then edited to produce a grammatically correct statement. The participants who signed their Round One questionnaire received a copy indicating which statements in Round Two their suggestions had contributed to. If they disagreed with the way the researcher had handled their suggestion(s), they were to indicate their disagreement on the Round Two questionnaire. 101 Table 5 in Chapter 4 shows the relationship of participant's suggestions to the final statements. Through this process the number of suggestions was reduced to 64 statements grouped into 26 areas which formed the first part of the Round Two questionnaire, the Statements Form (Appendix H). A Response Form (Appendix I) comprised the second part of the Round Two questionnaire. Only the Response Form was returned to the researcher. On the Instructions page of the Statements Form the researcher recommended that the participants note their judgements next to each statement. This would allow the participants to compare their Round Two judgements with the group values returned in Round Three. In responding to the Round Two questionnaire the participants were asked to first consider the 26 areas into which the statements had been grouped before making judgements on the individual suggestions. Participants were also informed that the suggestions within an area may vary only in how the technologies are combined. Participants were then to choose the alternative on the Likert scales closest to their judgement and offer supporting comments. Two cover letters were generated for the Round Two questionnaire. One, thanking those participants who signed their Question 1 form for returning it (Appendix F). The second letter to those participants who did not sign their 102 returned questionnaire and for those participants who did not return their Round One questionnaire (Appendix G). Round Three Question: The purpose of this final round was to permit the participants to review the ranking and comments generated from Round Two and to offer revised judgements and comments on this ranking. The development of Round Three of the Delphi began by reviewing all of the Round Two comments for indications that the researcher had misinterpreted a Round One suggestion in creating the Statements Form. No indications of this nature were present. However, it was felt that in the non-grouped statements number 44 should have been with the SIMULATIONS group and number 51 should have been with the INDIVIDUALIZES LEARNING group. Upon review, these suggestions were considered valid. Development proceeded with the construction of a spreadsheet for entering each participant's group (Training, Instructional Technology, Expert System Development, Expert System Research, Optical Data Storage) and their judgements from the Likert scales. This database was then used to calculate the averages for each of the 64 suggestions. A word processing file was also set up to record the supporting comments for each suggestion. Most supporting comments were entered verbatim. The editing that occurred was for grammatical clarity, to remove shorthand 103 abbreviations, and to abridge long comments for participants' convenience. .The Round Three questionnaire was also developed as two parts; a Comments Form (Appendix M) and a Response Form (Appendix N). Only the Response Form was returned to the researcher. Based on the analysis of the Round Two responses for all participants, Round Three suggestions were rank ordered on the Response Form ranging from 5.79, the technologies being combined within the next five years and having an advantage for adult learners, to 4.10, the . technologies can not be combined in the next five years but there would be an advantage for adult learners. The area dealing with the four suggestions that combining the technologies would have negligible or no impact on adult learners was rank ordered separately from 3.76, mild agreement, to 2.21, mild disagreement. The items on the Response Form for Round Three were not renumbered after they were placed in rank order. The researcher felt this would allow participants an easier reference to their Round Two judgements, if participants had noted their judgements on the Round Two Statements Form . The Round Three Comments Form began with 6 comments addressing the research in general. The comments for the specific suggestions were retained in numerical sequence. Comments for each suggestion were arranged so that similar 104 comments were contiguous and ranged from supporting the suggestions to disagreement. In responding to the Round Three questionnaire the participants used the same Likert scales that were used in Round Two. Participants were asked to first review the general comments that were made regarding the research. Then, for each suggestion, before offering a revised judgement and supporting comments, consider: First, the group judgement and the supporting comments for the suggestion. Second, the two original questions from Round Two; 1) What is the possibility that the technologies described in the suggestion will be combined in the next five years? 2) Will this combination of technologies provide an advantageous impact on adult learners? And, third, their original judgement and reasons from Round Two. Again, two cover letters were generated for the Round Three questionnaire. One for participants who signed the Round Two Response Form (Appendix K) and one for the remaining participants (Appendix L). Analyzing The Data Research questions 1 and 2 were answered from Round Three statement numbers 1 through 60 that had group means of 5.5 or greater. These statements were considered as technologically feasible to achieve and, if this technology was applied to adult learning situations, would be capable of affecting adult learners as described. 105 Research question 3 was answered by analyzing the 153 insights contributed by participants in responding to the Round One questionnaire which were used to develop the 64 statements for the Round Two questionnaire. A Statement was considered to be unique and to have originated solely from either the technology researchers and develOpment group (DEV), composed of expert system developers and researchers and optical data storage developers, or the human resource developers group (HRD), composed of training and instructional technology professional, if it was developed from two or more insights contributed from one group and none from the other. To test the null-hypothesis for the research question 4, the nonparametric Mann-Whitney test was employed. Because the assumptions of normality of distribution about the mean and equality of variance were not supported by the data, a standard parametric test, such as the t-test, could not be used. Of the nonparametric tests available, the Mann-Whitney was chosen as its purpose is to determine whether two uncorrelated means differ significantly from each other (Borg and Gall, 1979). Research question 5 was answered from the statement numbers 61 through 64 in Round Three. Any of these statements with group mean of 3.5 or greater was considered as a factor with a negative impact on the effective application of these combined technologies on adult learners. 106 Data Collection Schedule This Delphi technique involved three rounds of questionnaires over a period of eight months. The schedule of activities followed is outlined below. DATE ACTIVITY 6/03-10/91 Initial telephone contact was made with each of the (50 - 75) participants to obtain agreement to participate in the Delphi. The Round 1 question and cover letter were mailed out to the participant immediately following acceptance. 6/27/91 Mail postcards to Round 1 participants who had not yet responded to the questionnaire. 7/09/91 Telephone calls to Round 1 participants who had not yet responded to the questionnaire 10/11/91 Mail Round 2 question to participants. 10/18/91 Mail postcards to Round 2 participants who had not yet responded to the questionnaire. 10/29/91 Telephone calls to Round 2 participants who had not yet responded to the questionnaire 12/12/91 Mail Round 3 question to participants. 1/06/92 Mail postcards to Round 3 participants who had not yet responded to the questionnaire. 2/04/92 Telephone calls to Round 3 participants who had not yet responded to the questionnaire 2/28/92 Mail thank you postcards to Round 3 participants and indicate when they can expect to receive a final summary of the results. 107 Summary In Chapter III, the Delphi research questions were identified. The participants in the study were defined as being drawn from five groups: computer scientists researching artificial intelligence/expert systems, optical data storage developers, expert systems developers, industrial training professionals, and instructional technology professionals. Participants were chosen from the 1990 membership lists of the American Association for Artificial Intelligence (AAAI), Industrial Training and Education Division (ITED) of the Association for Educational Communications a Technology (AECT), Instructional Technology Professional Practice Area of the American Society for Training and Development (ASTD), and from companies identified using the Training Marketplace Directory. A Delphi exercise was employed in the data collection for this study. The rationale for using the Delphi exercise was discussed, as was the development of each question and the method of analysis. This development included a review of Question 1 by five individuals, representative of the participant groups before it was sent to the participants. The results of that review were largely positive, indicating the need for this study. The suggestions made by the reviewers for modifications to the questionnaire were provided. 108 How each of the research questions would be answered was then discussed. The answers to questions 1, 2, and 5 would be based on the rank ordering of the statements by the participants after Round Three. Question 3 would be answered by comparing the contributions of the participants' Round One insights to the development of the statements for the Round Two questionnaire. And, question 4 would be answered by an analysis of the Round Three rankings using the Mann-Whitney nonparametric statistical test for significant difference between uncorrelated means. Finally, a detailed schedule outlining the data collection procedure was included in Chapter Three. CHAPTER IV FINDINGS Introduction The purpose of this research was to determine expert judgements about possible technological combinations of expert systems and optical data storage systems that might occur within the next five years (1991 - 1996) and to determine the impact such combinations could have on adult learners if applied in the education and the training fields. This chapter presents an analysis and discussion of the data collected during the three rounds of the Delphi. Three research questions were posed as the focus of Round One, three research questions in Round Two, and four research questions in Round Three. Responses and analysis to each of these ten questions will be discussed in their order of collection later in this chapter. To begin the chapter, the response characteristics of the participants will be considered, examining both the response rate and quality of the responses. Response Characteristipg of the Participants Response Rate For the Three Questionnaires: The first round questionnaire was sent to seventy four participants. Thirty six participants responded with insights on how expert system and optical data storage technologies might be combined and on what might be their impact on adult learning. Six participants indicated that after reviewing 110 the question they did not feel qualified to respond and asked to drop out of the Delphi. Sixty eight questionnaires were sent out in the second round with thirty three participants offering judgements and one requesting to drop out of the Delphi. The final questionnaire was sent to sixty seven participants with forty responding. Table 3 summarizes these rates. Table 3. Response Rate of Mailed Questionnaires. Round Number Number Percent Number Sent Returned Returned Dropping Out l 74 36 49 6 2 68 33 49 l 3 67 40 60 O The questionnaires were sent to five subgroups that made up the Delphi participants: 19 instructional technology professionals (IT); 16 professionals involved with the design, planning and managing of training programs (T); 16 expert systems developers (ESD); 11 researchers in the field of artificial intelligence/expert systems (ESR); and 12 optical data storage developers (ODS). In the first round, the six participants who dropped were distributed among the subgroups as follows: 1 IT, 2 T, 1 BSD, and 2 ESR. The single participant who dropped during the second round was a member of the ESR subgroup. Table 4 summarizes the response rates for each of these subgroups. 111 Table 4. Response Rates for the Delphi Subgroups Total # of Round IT % T % ESD % ESR % ODS % Respondents 1 12 63 9 56 7 44 5 45 3 25 36 2 9 50 8 57 6 40 5 56 5 42 33 3 ll 61 11 79 7 47 5 63 6 50 40 % = Percent returned for the subgroup based on the number of participants in the subgroup at the beginning of each round. Delbecq, Van deVen and Gustafson (1986) suggest that 10 15 well-chosen participants can generate a full range of ideas in a Delphi with few new ideas being added once the group size exceeds thirty. In reviewing the literature on the Delphi technique no indication of a critical attrition rate that would invalidate the Delphi was found. Because of the anonymity in responding to the Delphi rounds it was necessary to send round two and three questionnaires to all participants who had not officially requested to be dropped from the technique. As such, participation in subsequent rounds did not depend on participation in previous rounds. Keeping in mind that not all returned questionnaires could be tracked by the researcher, it was found that 19 participants responded to all three rounds, 6 responded to only Rounds One and Two, 4 responded to only Rounds One and Three, and 7 responded to only Round One. Table 5 summarizes these figures. 112 Table 5. Estimated Individual Participation in the Delphi Rounds. * Anonymous responses not included. Participated in Number of Individuals All 3 Rounds 19 Only Rounds 1 a 2 6 Only Rounds 2 & 3 0 Only Rounds 1 & 3 4 Only Round 1 7 Only Round 2 0 Only Round 3 0 Quality of Participants' Responses: During the initial telephone conversations many individuals indicated enthusiasm for the topic. Several participants gave the researcher references to works that they felt could have bearing on the topic. Once the Delphi was under way the consistency of the return rates between Rounds One and Two, the increase in the return rate between Rounds Two and Three, and the known number of individuals that responded to all three rounds indicated that there was a core of participants committed to this research. Participants apparently had no difficulty in understanding the instructions as all returned questionnaires were completed correctly. Judging from the 153 suggestions offered in Round One and the liberal expression of comments to support judgements in Rounds Two and Three many of the participants 113 spent a substantial amount of time on the questionnaires. In addition, several articles that participants felt would be of interest to the researcher were included with returned Round One and Two questionnaires. Results of Round One The purpose of the first questionnaire (Appendix C) was to solicit responses to the following research questions: 1. Which combinations of the expert system and optical data storage technologies are most likely to be achieved in the next five years? 2. What contribution(s) to adult learning could the combinations of these technologies be expected to produce? 3. Which insights were unique and originated solely from each of the five subgroups: instructional technology professionals (IT); professionals involved with the design, planning and managing of training programs (T); expert systems developers (ESD); researchers in the field of artificial intelligence/expert systems (ESR) and optical data storage developers (ODS). The Round One Questionnaire provided participants with the opportunity to express their insights into how the technologies of expert systems and optical data storage systems might be combined in the next five years and the impacts that such technological combinations might have on adult learners. One hundred fifty three insights were suggested by the study participants (Appendix E). Three examples of these insights are provided in Table 6. 114 Table 6. Examples of Insights Provided from Round One How the Technologies Might Be Combined The Impact On Adult Learners 12. Optical media will permit Pictures can be used to clarify a storage of more motion and learner’s understanding of concepts audio with pictures. or decisions solicited during an "expert system" session. The altering of motion video could be as easy as the present rearrangement of still video. For example, in landscape architecture the location of shrubbery and ornamental: and the resulting effect of their shadows as the Sun passes across the sky during the day. 22. Expert System and any of The general impact will be the optical data storage negligible. Many aspects of technologies learning, as an example the affective domain, cannot be reduced to a level capable of being programed into a computer. As such, it is believed that Expert Systems will not be capable of capturing or analyzing the nuances and individual characteristics of learners, their styles of learning or their reaction to mediated instruction. 41. Expert System and CD-ROM Continual monitoring of the database student's learning process. Because 153 suggestions would be an excessive number for most participants to offer judgements on in the second round, a process of systematic elimination was used to reduce this number. Suggestions were combined and grouped according to the following 3 criteria: a) Group suggestions based on their impact on adult learners. The researcher began by grouping the suggestions together based on the impact combining b) C) 115 these technologies might have on adult learners. Many of the suggestions contained compound ideas as to the impact the combining of these technologies might have on adult learners. These compound ideas were extracted from the original suggestion and placed in a group of related ideas. As such, a single participant's suggestion could contribute ideas to more than one group. When this sorting was complete, 26 groups had been formed. Group non-related suggestions. One of the 26 groups was composed of 17 suggestions did not seem to relate to any of the other suggestions based on the impact that combining these technologies might have on adult learners. After reviewing each of these 17 suggestions, the researcher decided to retain them for the second round as each seemed insightful and had the potential to stimulate ideas in the participants. The researcher felt that this simulation might reflect elsewhere in the Round Two Questionnaire. Combine duplicate suggestions within groups. Within each of the 25 groups having related suggestions, duplicate suggestions were combined into a single statement. Suggestions within a group typically varied with regard to how the expert system and optical data storage technologies were combined. 116 In the case of the three groups; IDENTIFIES STUDENT'S LEARNING STYLE, PORTABILITY, and CREATES PERFORMANCE SUPPORT SYSTEM the resulting impact on adult learners also varied. Through this process the number of suggestions was reduced to 64 statements distributed across the following 26 groups: 1. Identifies Student's Learning Style 2. Performs Needs Analysis 3. Adjusts Program to Student's Pace 4. Maintains Motivation/Interest 5. Facilitates Interactive Learning 6. Facilitates Creativity 7. Individualizes Learning 8. Facilitates Self-Directed Learning 9. Facilitates Discovery Learning 10. Provides Realistic Simulations 11. Illustrates Dynamics of Complex Problems 12. Improves Basic Skills Instruction 13. Increases Retention 14. Assists in Retrieving Information from Databases 15. Acts as an Intelligent Evaluator During Learning 16. Expert System Has the Ability to Learn 17. Portability, Learning Can Occur Anywhere 18. Creates Performance Supports Systems 19. Lowers Cost of Training 20. Simplifies CAI Through a Graphic User Interface 21. Increases Use of Visuals 22. Increases Use of A/V 23. Facilitates Cost Effective Repurposing of Materials 24. Provides for Easier/More Efficient Updating of Materials 25. Non-Grouped Responses 26. Negligible or No Impact on Adult Learners Table 7 shows the relationship of participant's suggestions to the final statements used in the Round Two questionnaire. 117 Table 7. Suggestion Contributions to Final Statements Suggestion # ContributedHSuggestion # Contributed From to Rnd. 2 From to Rnd. 2 Appendix E Statement # Appendix E Statement # 1 34 44 2,10 2 16 45 4,12 3 22 46 36 4 12 47 60 5 12,38 48 47 6 19 49 8,40,42 7 44 50 42 8 14 51 40 9 25 52 40 10 13 i 53 2,5 11 37,38 1 54 30 12 41 55 17 13 13 56 22 14 55 57 29 15 27 58 22 16 37 59 36 17 22 60 21,41 18 18 61 13 19 30 62 36 20 27 63 27 21 56 64 57 22 61 65 22 23 16 66 31 24 16,36 67 22 25 15,16,20 68 16 26 10 69 40,42 27 11,23 70 40,42 28 14,62 71 12,41 29 6 72 17 30 7,20,40 73 22 31 45 74 43 32 l 75 11 33 1 76 53 34 3 77 54 35 19 78 18 36 51 79 17,39 37 13 80 13 38 23 81 4,12 39 1 82 25 40 46 83 21,36 41 23 84 42 42 52 85 25 43 8,41 86 2,4,5,12 118 Table 7 (cont'd). Suggestion # Contributed Suggestion # Contributed From to Rnd. 2 From to Rnd. 2 Appendix E Statement # Appendix E Statement # 87 5,17 120 21,30 88 2,4 121 21,30 89 2,41 122 40 90 24 123 33 91 2 124 2,24,41 92 4,12 125 49 93 7,12,15,24 126 63 94 4,12,41 127 22 95 38 128 30 96 24 129 4,12,31 97 58 130 64 98 12,24,41 131 17 99 22 132 26 100 48 133 38 101 7,41 134 50 102 2,24,41 135 22 103 4,6,23,39 136 21.30 104 59 137 17 105 32 138 7,12,24 106 2,7,12,24 139 2,12 107 14,41 140 30 108 8,9,41 141 8 109 17 142 43 110 31 143 2,4,12 111 35 144 31 112 22 145 17 113 14,22 146 2,9 114 12,41 147 4,12 115 14 148 28 116 17 149 24 117 24 150 31 118 12,15,24 151 22 119 21,30 152 22 153 64 Table 8 reports the distribution of suggestions that contributed to the final statements used in the Round Two Questionnaire across the participant subgroups: instructional technology professionals (IT); professionals involved with the design, planning and managing of training 119 programs (T); expert systems developers (ESD); researchers in the field of artificial intelligence/expert systems (ESR); and optical data storage developers (ODS). Three columns have been added to this table, HRD, DEV, and TOT. This was done based on Delbecq, Van deVen and Gustafson's (1986) guideline that 10 - 15 participants can generate a full range of ideas in a Delphi. Table 4 shows that there were enough total participants in the Delphi to satisfy this guideline. However, low participant response rate did impact the study of the subgroups. An examination of Table 4 reveals that ESD, ESR, and ODS were each well below 10 in all three rounds. IT and T response rates were marginal. In order to maintain a subgroup component of the study with valid numbers of participants the IT and T subgroups were combined into a single Human Resource Developers (HRD) subgroup. ESD, ESR, and ODS were similarly combined to form the technology researchers and developers (DEV) subgroup. The third column (TOT) gives the total number of suggestions that contributed to the final statement. 120 Table 8. Distribution of the Participants' Suggestions to the Round Two Statements by Participant Subgroup Round Two Number of Times Suggested By: Statement # I ESD I ESR I 003 I HRD I DEV I TOT H *3 |-] Il-‘Iw H H I w I IbI H~HHNUILD~INUNUII-‘NHI-‘UlmI-‘t-‘wI-‘wI-‘nbmeNwwmwmmmNNNIP-MNNOHNQ) INwlibl Ii-‘INI \OCDQO‘U'lthH ll-‘I-‘wt-‘IwINI l ...a O ...a ...a II-‘II-‘lt-‘I Ht‘k‘ Ulhtd leI HIa ~1m I HIaI m I II ...a N IU'IIII-‘lelI-‘It-‘I-‘I-‘I ...a m Il-‘wt-‘I I ...a \D I NNNNNNN O‘UIIbUNI-‘O I I H N \l lNII-‘IUNINNNInhl-‘NwNI-‘HIl-‘Hl-‘I li-‘INI-‘IUJNI N \D c» o H I I I Hral uwwu btuh3H II I H H I l u U" N (D Il—‘II—‘InbO‘lI-‘II—‘lmlmlbl :50»)wa 0“)QO I NH I l-‘NH I NI IN I I N p N INIwI uh so (did It—‘INt—‘ll—‘I I waNHl oh .5 IUIHIAI I HIdde<3¢sNO