.,.......§.._. . I .15... 33.4... m. wk .3... an...» in“! . .. :Lvhl. LE. . ”9......“ 5.5.9. a... fit 32:: . . ‘7‘“ .ul. . v. 1.. .zhavxtv . YI‘ a .4 p .2... n11 a. 1...... :D11 Izrnw ,II! :6» :. - 2.. 4 1:14 $3... mummy? ...u .. I 3.? .x 1.3.: ; 1. 33.117.11.30 5 ~ 3.2:... .2... s. u... S: . 3. Y. :3. 5.. . :3..n.......:v » 3 , 2:. . p. 5.7. .r T, 3.2.3.5. .41. i (.191. if; 3&1. 912:}. x. .55 .1!!! .. .5! Blah.) o: :45 . Lx‘.‘ .. 3.3.3.»; ’ 7.2:"! . .u. . . t u, .. 5.3.1.. 2.3).! 1‘ . r. 3: .(1 ... 1.13...- . .22... 1 1:31! 4 «.r.... ..\...r. ER. II... .1 i.» _ . .imafin». _. , Enififim 1.33... flaw 2.: {1.18, . z . .. u . ~\ . . 25-391.). . i THES'S Z Z (L0 LIBRARY M'chigan State University T This is to certify that the dissertation entitled AN INVESTIGATION OF LEARNER CONTROL AND METACOGNITION USING A WEB-BASED TRAINING PROGRAM presented by Rebecca J. Toney has been accepted towards fulfillment of the requirements for Ph.D. Social Science degree in 12% PM Major professor Date ("3 ’/ 7 ‘ 2 “V 0' MS U is an Affirmative Action/Equal Opportunity Institution 0-12771 O PLACE IN RETURN Box to remove this checkout from your record. To AVOID FINES return on or before date due. MAY BE RECALLED with earlier due date if requested. DATE DUE DATE DUE DATE DUE 11-02703, mnyzgmm .7 01be 044 2004 6/01 c:lC|RC/DateDue.p65-p.15 AN INVESTIGATION OF LEARNER CONTROL AND METACOGNITION USING A WEB-BASED TRAINING PROGRAM By Rebecca J. Toney A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Psychology 2000 ABSTRACT AN INVESTIGATION OF LEARNER CONTROL AND METACOGNITION USING A WEB—BASED TRAIN ING PROGRAM By Rebecca J. Toney The focus on trainees as active participants in the learning process increases the value of studying learner control designs and the facilitation of metacognition during training. Both learner control and metacognition are ideally suited for examination in a web-based training (WBT) program. The present study uses WBT to explore patterns of behavior in which trainees engage to seek out information, practice, and feedback. A four-cell factorial design allows the examination of two manipulations: metacognitive monitoring and metacognitive control. A positive impact is found for seeking behaviors on training outcomes, including learning, self—efficacy, and performance. The prompting of metacognitive monitoring and control did not produce the expected results, but the results shed light on questions for future research. ACKNOWLEDGMENTS Thanks to my chair, Kevin Ford, for introducing me to the fascinating and uncharted area of web-based training research, and for letting me risk returning to the seemingly unyielding area of metacognition research. He always pointed me back in the direction of “the constructs” when I wandered astray. Most importantly, with incredibly speedy tum-arounds, he helped me put together a proposal in an almost unheard-of time span of about three months so that I could join my husband in Cleveland. Thanks to my committee members, Steve Kozlowski, Rick DeShon, and Bruce Burns. They asked the difficult questions that made me think in a way that produced a better quality product. I have learned a great deal form them. Thanks to my colleagues who supported me—literally, by letting me live with them during the data collection process—and celebrated the important milestones with me. I will miss them all dearly. Thanks to my mother, father, and sister, who have waited patiently to see me again after this five-year sojourn. They have supported me with encouragement in all my life endeavors that have led me to this point. Thanks to my husband, John, who has helped motivate me to complete this degree with both carrots and sticks. If he had not taken the TV and the stereo with him to Cleveland during the summer of 1999, I may have never finished this dissertation! I am blessed to have a partner as wonderful as him to have by my side through this journey. Thanks to God for the energy, talent, and supportive persons in my life, without which I could not accomplish anything. iii TABLE OF CONTENTS LIST OF TABLES ............................................................................................................. vi LIST OF FIGURES ........................................................................................................... vii INTRODUCTION ............................................................................................................... l Learner Control ....................................................................................................... 1 Metacognition .......................................................................................................... 3 Web-Based Training ................................................................................................ 4 Learner Control and Seeking Behaviors ............................................................................. 8 A Typology of Seeking Behaviors .......................................................................... 9 Training Outcomes of Seeking Behaviors ............................................................. 15 Individual Differences Affecting Seeking Behaviors ........................................... 18 Operationalizations of Seeking Behaviors ............................................................ 21 Limitations in Research on Seeking Behaviors ..................................................... 22 Metacognition .................................................................................................................... 23 Descriptions of Metacognition .............................................................................. 25 Outcomes of Metacognition .................................................................................. 28 Operationalizations of Metacognition ................................................................... 30 Experiments Training/Manipulating Metacognition ............................................. 32 Limitation to Research on Metacognition ............................................................. 35 Hypotheses ........................................................................................................................ 36 Overview ............................................................................................................... 38 Effects on Breadth and Depth of Seeking Behaviors ............................................ 39 iv Effects on Training Outcomes ............................................................................... 45 METHOD .......................................................................................................................... 49 Design .................................................................................................................... 49 Participants ............................................................................................................ 49 Procedure ............................................................................................................... 49 Manipulations ........................................................................................................ 52 Measures ................................................................................................................ 54 RESULTS .......................................................................................................................... 60 Effects on Breadth and Depth of Seeking Behaviors ............................................ 60 Effects on Training Outcomes ............................................................................... 66 Summary of Results .............................................................................................. 70 DISCUSSION ................................................................................................................... 73 Individual Differences and Seeking Behaviors ..................................................... 74 Contribution of the Seeking Typology .................................................................. 79 Future Directions for Learner Control and the Seeking Typology ....................... 83 Metacognitive Manipulations ................................................................................ 88 Future Directions for Metacognition in Learner Control ...................................... 92 Conclusion ............................................................................................................. 93 REFERENCES .................................................................................................................. 96 APPENDIX A ................................................................................................................. 106 APPENDD( B .................................................................................................................. 108 APPENDD( C ................................................................................................................. 109 APPENDIX D ................................................................................................................. l 13 LIST OF TABLES Table 1. Means, Standard Deviations, and Correlations for Measured Variables ........... 61 Table 2. Regression of Individual Difference Variables on Depth (Complexity) of Information Seeking ........................................................................................... 65 Table 3. Regression of Seeking Behaviors on Learning .................................................. 67 Table 4. Regression of Seeking Behaviors on Efficiency ................................................. 68 Table 5. Regression of Seeking Behaviors on Final Self-Efficacy Ratings ..................... 68 Table 6. Regression of Seeking Behaviors on Training Performance ............................. 70 vi LIST OF FIGURES Figure 1. Expected pattern of relationships among individual differences, manipulations, seeking behaviors, and training outcomes. ................................ 39 Figure 2. Expected interaction between metacognitive skills and metacognitive manipulation ....................................................................................................... 42 Figure 3. Observed pattern of relationships among individual differences, seeking behaviors, and training outcomes ....................................................................... 71 Figure 4. Observed significant (and near significant) effects of individual differences on seeking behaviors, and of seeking behaviors on training outcomes ................... 72 vii INTRODUCTION The trend in training research over the past several years has been to more closely integrate fundamental principles of cognitive psychology into the study of learning processes (Ford & Kraiger, 1995; Howell & Cooke, 1989; Lord & Maher, 1991). This paradigm shift toward cognitive theory has led to an increased understanding of how information and skills are acquired and how they are transferred beyond the training environment (Tannenbaum & Yukl, 1992). Prior to this time most training research was ultimately based in practical issues. One of the most significant changes has been a focus on the trainee as an active participant in the learning process. Trainees are more likely to retain and transfer knowledge learned during training when they have actively produced the knowledge or skill (Campbell, 1988; Noe & Ford, 1992; Smith, Ford, & Kozlowski, 1997; Tannenbaum & Yukl, 1992). Noe and Ford note that providing training opportunities that allow trainees to control the learning process and to generate feedback on their progress can facilitate the active learning of adults. Smith and colleagues suggest that metacognitive skills are associated with active learning, and fostering metacognition during training can enhance the integration and adaptive transfer of knowledge acquired in training. Thus, learner control and metacognition are two theoretical areas that may provide useful means of prompting active learning on the part of trainees. Learner Control Learner control may be defined as an environment wherein the learner has input into and is responsible for the selection of instructional stimuli (Tennyson & Rothen, 1979). This can include some or all aspects that are traditionally uniform for all learners, such as the content of the learning material or the sequence and pacing of its presentation. The learner control literature has largely focused around comparisons to designs that use program control (where instructional stimuli are uniform for all) and adaptive control (where instructional stimuli are selected automatically based on trainee skill and needs). Tennyson and Rothen (1979) is a classic example of this learner control literature. Outcome measures are used to determine the relative success of the design, and post-hoe explanations are provided as to why learner control produced those outcomes (e. g., learners need more guidance during learner controlled training). However, the present research does not focus on this “comparative” issue. As Chung and Reigeluth (1992) note, the “. . .challenge is not whether or not learner control should be used, but rather how to maximize the learner’s ability to use the learner control available and to decide what kinds of learner control to make available” (p. 19). Little research has explored the process of how learners gather information when they are given control of their learning process. What kind of information do they seek out? Do they opt to receive guidance when it is available? Are there reliable differences among individuals in the patterns of knowledge and skill acquisition they choose? Answers to these and other questions would provide a better theoretical understanding of the process that learners experience and the choices they make in learner control environments. This understanding can then generate better predictions regarding when, why, and how to implement learner control designs. Learner control training environments provide some challenges because they lack the structure that learners are familiar with from their experience with traditional, program control training. Learners may need some assistance with the process of integrating material and interrelating concepts (Park & Hannafin, 1993; Smith, 1996). One means of promoting this deeper-level processing is facilitating the inclusion of metacognition in learner control environments (Smith et al., 1997). Metacognition Metacognition has been defined in a myriad of ways throughout the literature. In this research it is viewed as the conscious process of regulating cognition. Thus, it differs from self—regulation in that is it conscious and accessible (Flavell, 1979; Nelson, 1996) and it does not usually encompass affective components. Also, metacognition is often focused on a higher and more specific content, level, or process (e. g., the learning process) than is self-regulation (Carver & Scheier, 1982). We have long recognized the potential importance of metacognition to learning and task performance processes (Flavell, 1979). However, efforts to impart metacognitive skills in training environments have been equivocal (Bean, Singer, Sorter, & Frazee, 1986; Brown et al., 1997; Greiner & Karoly, 1976; Smith, 1996; Volet, 1991; Weissbein, 1996). Results from prompting metacognition have fared better than explicitly training metacognition (Jacobsen & Spiro, 1995; Pressley, Snyder, Levin, Murray, & Ghatala, 1987), although some have involved very simplistic designs (Lorene, Sturmey, & Brittain, 1992) and analyses (Avner, Moore, & Smith, 1980). One problem has been the variation in operationalization of metacognition in research, despite general agreement in metacognitive theory on two or three dimensions including planning, monitoring, and controlling/regulating (i.e., Garcia & Pintrich, 1994; Nelson, 1996). A better understanding of how these dimensions uniquely contribute to the learning process will further researchers’ ability to train or prompt metacognition successfully. Metacognition is important to active participation in learning (Smith et al., 1997), and particularly in learner control environments (Smith, 1996). Metacognition promotes the deep processing and integration of knowledge necessary in unstructured learning environments. Thus it becomes important to find a reliable means for facilitating metacognition in learner controlled training. Web-Based Training A new method for the dissemination of employee training is rapidly emerging: Web-Based Training (WBT). WBT is a computerized form of training that typically provides information to trainees in a linked network presentation, known as “hyperlinking.” By selecting hyperlinks with a computer—mouse interface, trainees can navigate among different sections of a computerized training course. WBT programs can be administered over internally accessed organizational Intranets and externally accessed Internets. Learner control and metacognition. WBT is moving from a didactive role to a facilitative role of trainers (Reeves & Reeves, 1997), as WBT programs increasingly do not just consist of “page tumers” where trainees progress directly through a fixed set of materials. The full capabilities of hyperlinking allows training designers to incorporate different amounts and types of learner control (e. g., sequence, timing, and amount of information received). The theoretical and empirical research of WBT programs will undoubtedly lead to the resurgence of the debate over the relative advantages and disadvantages of program control and learner control training. However, as previously stated, the present research does not focus on this issue. WBT programs designed and implemented in organizations will most probably entail a learner control framework, and the research literature is well served by investigations of this learner control design. The effectiveness of learner controlled WBT training programs is enhanced by trainees’ use of metacognition. The hyperlinked structure of WBT programs requires trainees to make conscious choices in the development of their knowledge. Gall and Hannafin (1994) advise though, “The mere selection of an option in hypertext does not ensure that the response and associated processing are deep, meaningful, and reflective” (p. 226). Jonassen and Grabinger (1993) echo this idea, maintaining that it “takes more than just a complex set of links to support the active construction of knowledge” (p. 15). The facilitation of metacognition in WBT can help trainees make sense of the training environment and make choices that best support learning. WBT programs provide a unique means of studying the combination of metacognition and learner control in prompting the active participation of learners. There are some established limitations of learner control that need to be overcome when designing WBT programs. First, learners new to a task or information frequently do not know how to sequence material when given control to choose topics. One way to ameliorate this problem is to offer trainees guidance as to which topics to select (Gall & Hannafin, 1994). Second, learners often do not naturally engage in metacognitive skills during training, integrating material that has been learned. Providing prompts throughout training that fosters metacognition may help trainees achieve more knowledge integration and better attend to their own learning process. Opportunity for research. There are a number of forces at work in industry that are enhancing the attractiveness of WBT and other technologically—based training programs (Goldstein & Gilliam, 1990; Schreiber, 1998). First, there is a need for constant re-training of workers, due to the rapid pace of change in products and regulations. Organizations are looking for a convenient means for providing “just-in- time” knowledge. Second, global economies have created geographically diverse workforces, which need to be trained in an efficient and standardized manner. Third, the increases in both competition and technology are driving organizations toward multimedia training programs that allow greater flexibility than traditional programs. These goals are being met via the use of satellite broadcasts, interactive video, and computer-based training on CD—ROM, as well as WBT on Intranets and Internets. The notion of hyperlinking information in a reference file—the key feature of WBT—goes as far back as 1945 (Gall & Hannafin, 1994). However, hyperlinked media is only now coming to widespread use with the advances of computerized Internets. Many advantages of WBT programs have been espoused in recent literature. Programs are easily updated and maintained because they enable the localization of resources and support services; they facilitate the presentation of standardized data, format, and training within a program (Fukai, Kitchen, & Aurelio, 1998). Training is more likely to be provided at the convenience of trainees, rather than trainers or training centers (Howard, 1998) and travel costs are reduced (Walker, 1998). Trainers are freed up to design new programs and assist individuals experiencing difficulty (Milheim, 1996). Accessibility is increased to trainees with visual and auditory impairments, trainees who speak English as a second language, and trainees who are more reticent (Warn, Compton, Levine, & Whitteker, 1998). Thus, due to advances in technology and changes in organizations and industry, WBT is becoming an attractive alternative to the traditional means of training employees. Despite the increasing interest in using WBT programs in organizations, much of the practice literature concerning WBT is not research-based. A few theoretical discussions include the formative evaluation taxonomies devised by Tessmer ( 1995), the framework for hypertext study developed by Gall and Hannafin (1994), and the research-based principles provided by Park and Hannafin (1993). Only a handful of empirical studies have been conducted using WBT programs, and they have not been published in mainstream psychological literature. Gay, Trumbull, and Mazur (1991) tracked the performance of students using a WBT program and the patterns of use of “search,” “browse,” and “guide” modes. J acobsen and Spiro (1995) compared training and transfer performance of students using a WBT program to students in a control group using a more rigid computer-based training program. Recent and ongoing empirical studies are beginning to explore individual difference variables with respect to performance in WBT programs (Brown, 1999; Simmering, 1999). The attractiveness of WBT as an efficient method for providing standardized, and yet flexible training for a diverse and widespread workforce positions it as a useful conduit for research. The popularity of the Internet is not likely to be a passing trend, and organizations will be making use of it in training regardless of whether or not there is solid research supporting its use. The following sections undertake an in-depth discussion of the two topics introduced here. First, the behavior of individuals in learner control environments is explored. Second, the literature on metacognition theory and research studies incorporating metacognition in training are reviewed and discussed. Finally, hypotheses concerning these topics are generated and incorporated into a WBT research experiment. Learner Control and Seeking Behaviors The main feature of learner control training is that it allows learners to direct their own learning process in some or all aspects. They are not provided with a completely structured path to learning the material, but instead are allowed to seek out their own path. Especially when learners are given control over the content and the sequence of training, this “path seeking” in which they engage plays an important role in the outcomes of the learning process. Thus, an examination of the seeking behaviors of learners can greatly enhance our understanding of the functioning and benefits of learner control environments. A growing body of research indicates that individuals can be active seekers of feedback regarding their performance on the job and in training environments (i.e., Ashford & Cummings, 1983), provided that they are motivated and have the ability to do so (DeShon & Milner, 1998). When designed to include hyperlinks to training materials, quizzes, exercises, and feedback, WBT is well suited to facilitating such feedback seeking behavior. In fact, WBT does not limit the trainee to only choosing the amount of feedback they desire; trainees can also seek additional information on a topic, seek more practice opportunities, and seek a variety of forms of feedback. For example, because of the nonsequential nature of hyperlinked systems, trainees “can become confused and unfocused if left entirely unguided” (p. 190, Gay, Trumbull, & Mazur, 1991; also Gall & Hannafin, 1994). Thus, WBT can include links to feedback that recommends to trainees future topics of information to be explored. As mentioned, there is a body of research that examines the feedback seeking behavior of individuals. There are also bits and pieces of research that examine other types of seeking behavior such as the seeking of information (i.e., Miller & Jablin, 1991), and the seeking of guidance (i.e., Tennyson, Christiansen, & Park, 1984). However, there does not appear to be any theoretical research that draws these different types of seeking behaviors into a coherent whole. The following section constructs a typology of seeking behaviors that are relevant to WBT, specifically including feedback seeking, information seeking, and practice seeking. A Typology of Seeking Behaviors Issues of depth and breadth are central to the construction of a typology of seeking behaviors. Trainees can seek a broad range of feedback, information, and practice, as well as depth within each of these sources. Gall & Hannafin (1994) have discussed breadth and depth of knowledge bases. Breadth is described as the “relative measure of the diversity of . . . subject matter” and depth is “the amount and complexity of information available on a given topic.” With respect to breadth of seeking behavior, trainees can seek different combinations of information, practice, and feedback during their learning. Greater breadth of seeking behavior is characterized by seeking all three, whereas only seeking feedback (or only information, or only practice) would constitute a lesser breadth of seeking behavior. Sasscer and Moore (1984) found that no one pattern of seeking rules (information), examples, and practice exercises was associated with training success. However, two-thirds of trainees evidenced patterns that omitted at least one of the three categories, indicating variability within the breadth of trainees’ seeking behaviors. Simultaneously, trainees can access different levels of feedback, information, and practice; this constitutes the depth of seeking behavior. Gall and Hannafin note that an advantage of providing depth within an information source accommodates learners with different levels of prior knowledge. Presumably providing depth of feedback and practice similarly accommodates learners with individual differences relevant to seeking behaviors. Depth of seeking behavior can take on a variety of forms. For example, it can refer to simply the amount of feedback seeking in which a trainee engages. Thus, greater depth of seeking would be indicated by more total feedback seeking instances. However, there may be other forms of depth, such as content or complexity. Virtually any aspect of seeking behavior that is constituted of varying levels can be conceptualized as depth of seeking behavior. In the following sections, the three proposed types of seeking behavior— feedback, information, and practice—are described and relevant literature is reviewed. Then potential conceptualizations of depth within that source of seeking behavior is proposed. Feedback seeking. Feedback seeking has been relatively well researched in comparison with other seeking behaviors. In the present research, feedback seeking is conceptualized as behaviors engaged in by a trainee to determine their past or future progress through material to be learned in the training environment. However, some past research has used this conceptualization with different terminology, notably “information 10 seeking” (Butler, 1993). Research is presented according to the conceptualization that it supports as defined by this paper, regardless of the terminology used in the original work. This allows a better distinction between feedback seeking and information seeking (discussed in the next section). VandeWalle & Cummings (1997) note that feedback can be sought by inquiry or by monitoring. Miller & Jablin (1991) expand the distinction to include overt questions, indirect questions, limit—testing, disguised requests, observation of specific events, and general surveillance. Morrison & Bies, (1991) also distinguish feedback seeking behaviors according to the way requests are presented: a basic question that can stand alone or be supplemented by an explanation for the request, a positive spin, or a negative spin (“Did I do well/poorly?”). Feedback can also be differentiated by the content that is sought. Kozlowski, Weissbein, Brown, Toney, and Mullins (1997) have categorized feedback by content organized into three groups: descriptive feedback, evaluative feedback, and guidance feedback. Descriptive feedback provides a trainee with objective information about task performance. Evaluative feedback provides subjective information about task performance, which can have the trainee’s own past performance or other trainees’ performance as the referent. Guidance feedback directs the trainee toward the tasks that should be performed next, and can be specific to the trainee’s performance or generalized to all trainees. This content categorization scheme is also present in Butler’s (1993) research. Objective feedback (similar to descriptive feedback) assesses how the trainee’s performance is meeting the task demands. Normative feedback (similar to evaluative feedback) assesses the trainee’s performance relative to a reference group. Task 11 information (similar to guidance feedback) clarifies the task demands or strategies and presents the best solutions to the topic at hand. Finally, Miller and Jablin (1991) likewise include content distinctions between appraisal feedback (descriptive) and relational feedback (evaluative). The inclusion of guidance along with the more traditional objective and normative aspects of feedback warrants further discussion. Tennyson et al. (1984) note that guidance feedback can suggest the amount, sequence, or time Optimal for future instruction. Gall & Hannafin (1994) recommend providing trainees with guidance by advising them of alternative approaches to take with regard to progressing through training. As options increase with WBT, trainee disorientation can also increase. To remedy this problem, trainees can be provided feedback in the form of guidance in selecting related topics that have not yet been examined. However, guidance is not automatically provided following task performance; trainees must seek out the guidance feedback. Thus, trainees who feel they need guidance have the option of choosing to receive it, but those who don’t wish to receive guidance do not. Individual differences that determine which trainees will or should opt for guidance are discussed in a later section. The content of feedback seeking can be viewed as depth of feedback seeking. The different types of feedback—descriptive, evaluative, and guidance—allow trainees to explore in greater depth the meaning of their level of training performance. Trainees can seek feedback from zero, one, two, or three of these sources of content. However, the content itself that is chosen is likely to be most predictive of training outcomes. One form of feedback seeking depth that has not yet been discussed is the frequency of 12 feedback seeking. The connection here is straightforward; the more frequently a trainee reviews any or all sources of feedback reflects greater depth of seeking behavior. Information seeking. DeShon & Milner (1998) discuss how gathering information and applying it to work is becoming increasingly important in the “information age.” The rapid change currently affecting all aspects of work life requires adaptability on the part of workers, and the ability to seek out necessary information assists workers in becoming adaptable. Major and Kozlowski (1997) also discuss the necessity for organizational newcomers to proactively seek task information when sufficient information to perform job duties is not given. Compared to feedback seeking behaviors, there is much less conceptual development or research involving the seeking of information. In the present research, information seeking is conceptualized as behaviors engaged in by a trainee to become familiar with material (facts, relationships, etc.) related to the content of the instruction. For example, Gay and colleagues (1991) devised a hyperlinked training program that allowed trainees to choose links to different topics of information about insects. Seeking out the topic of insects and pesticides can be viewed as an information seeking behavior. This would be distinguished from choosing a link to a quiz on insects and then seeking feedback on quiz performance. Information seeking has been categorized by the content of information sought (Miller & Jablin, 1991) and by the frequency with which it is sought (Ashford & Cummings, 1985). Content and frequency have already been discussed as potential facets of depth of seeking behaviors. Another way in which information seeking can be categorized, that has not received attention in research, is by complexity or difficulty. l3 Trainees may seek out information in varying levels of complexity in relation to one training t0pic. This is perhaps the most intuitive conceptualization for depth of seeking behavior. The more complex information the trainees seek out, the greater the depth at which they are seeking the information. Practice seekigg. Ford, Quinones, Sego, and Sorra (1992) determined that the opportunity to perform trained tasks on the job is important for successful transfer of the trained skills. Goska and Ackerman (1996) found that the opportunity to practice during training also improves initial transfer by twice as much as those without practice opportunities. Brown (1999) found that higher levels of activity during training resulted in greater learning. Clearly, the practice of learned skills and application of knowledge is and important aspect of successful training programs. However, little research exists that examines choices trainees make with regard to taking advantage of opportunities to practice. A small part of a study by Fisher (1995) allowed trainees the choice of whether or not to work a sample problem, and found evidence for better performance for those who completed the problem. In the present research, practice seeking is conceptualized as behaviors engaged by trainees to apply learned information and skills in practice exercise opportunities during training. Like information seeking, practice seeking can be differentiated by content, frequency, and complexity, and these can all be conceptualized as facets of seeking behavior depth. Additionally, practice seeking can differ in the variability that it provides for trainees. Schmidt and Bjork (1992) discovered that greater variability in training practice results in greater transfer. Thus, trainees can be provided opportunities for practice that are quite similar to the task learned, or opportunities that vary more, and are 14 less similar to the learned task. As with complexity, the greater the variability that is sought, the greater the depth of practice seeking on the part of the trainee. Training Outcomes of Seeking Behaviors The effects of feedback seeking, and other seeking behaviors, on training outcomes may be similar to the effects of other interventions such as participative goal setting and decision making (Milner, 1999). The following sections discuss how the different seeking behaviors are expected to affect traditional outcomes of training such as learning and performance. In this research, learning refers to the acquisition of knowledge and skills, and performance refers to the practical demonstration of learned knowledge and skills. Chung & Reigeluth (1992) suggest that other training outcomes that may be affected by seeking behaviors include the effectiveness, efficiency, and appeal of training. However, these authors make no specific predictions and did not empirically study the effects of seeking behaviors. Despite this lack of empirical study, efi‘iciency is also discussed in the following sections as efficiency is often cited as a reason for employing WBT. Self-efiicacy is another outcome of training that will be examined (Ford, Smith, Weissbein, Gully, & Salas, 1998). Outcomes of feedback seeking Researchers have suggested that feedback seeking enhances the adaptability of workers in an organization (DeShon & Milner, 1998). Employees who are able to monitor how they are responding to changes in their work environment are better able to successfully adapt. Presumably, this relationship would hold in training environments as well. Trainees who are able to monitor their 15 learning progress should be able to adjust and perform better, both in training and beyond training. However, most research relating feedback to training outcomes in learner control environments is not based on feedback seeking but on feedback provision. For example, Clariana, Ross, and Morrison (1991) found that providing feedback produces better learning than not providing feedback. Tennyson et al. (1984) found that providing guidance feedback was best for learning, effectiveness, and efficiency outcomes. Kearsley and Hillelsohn (1982) recommend giving trainees a record of their progress and introducing a competitive element by providing evaluative feedback in order to increase performance. Gilman (1969) found “guided feedback” to produce better performance than descriptive feedback or no feedback. Finally, Kearsley & Hillelsohn ( 1982) determined that giving feedback at the end of a test rather than during a test resulted in the best retention of learned information. Clearly, more support needs to be evidenced for feedback seeking during training, rather than just feedback provision. Also, much of the literature regarding feedback seeking is concentrated on seeking feedback within the context of job performance rather than training and learning environments (Ashford & Cummings, 1983, 1985; Ashford & Northcraft, 1992; VandeWalle & Cummings, 1997). Although traditional training programs have not given trainees a choice as to whether feedback is provided or not, the capabilities of WBT to make feedback optionally accessible in multiple forms suggests that this is a necessary step to take in training research. This study expects that feedback seeking primarily affects the training outcome of performance. Trainees who periodically seek feedback about their learning progress 16 should be able to take that information and practically demonstrate its application within or outside the training program. Aspects of feedback seeking may also impact the efficiency of training. Efficiency, in training situations, is a ratio of learning to the amount of time it takes to achieve that learning. Trainees who seek guidance may be more efficient in their learning of the material. However, the frequency of feedback seeking may impact training outcomes in counteracting ways. High frequencies of feedback seeking may increase learning and performance, but may simultaneously decrease efficiency. Outcomes of information and practice seeking. Researchers have also suggested that information seeking leads to enhanced adaptability of trainees (DeShon & Milner, 1998). Workers who seek information are better able to predict changing demands and assess new situations, which leads to better adaptation. Information seeking is also expected—intuitively—to influence the learning outcome of training, as the depth of information that trainees seek out most likely translates directly into learning. The depth of practice experienced by trainees is similarly likely to translate into performance outcomes (Goska & Ackerman, 1996). The frequency of both information and practice seeking is likely to affect the efficiency of training in the same manner as does the frequency of feedback seeking. Although a certain level of seeking facilitates better learning, too much information and practice seeking may decrease efficiency. Finally, the depth of both information seeking and practice seeking may function to increase the self-efficacy of trainees at the end of training. The more complex and frequent their seeking of information and practice, the more confident they may be in the knowledge and skills they have acquired. 17 In one research study that allowed trainees to seek practice opportunities, those who chose to work a sample problem performed better than those who did not, but no effect on learning was found for practice seeking (Fisher, 1998). This result fits with the expectation that practice seeking impacts the training outcome of performance. Individual Differences Affecting Seeking Behaviors Numerous authors have noted that individual differences in trainees are important to the successful outcomes of WBT initiatives (Jonassen & Grabinger, 1993; Tsai, 1988) and learner control environments (Chung & Reigeluth, 1992). Some of the individual differences proposed or evidenced include ability (Butler, 1993; Chung & Reigeluth, 1992; DeShon & Milner, 1998; Steinberg, 1989), age (Chung & Reigeluth, 1992; Warr & Bunce, 1995), conscientiousness (Porter, 1997), external propensity (Fedor, Rensvold, & Adams, 1992), goal orientation (Butler, 1993; Reeves & Reeves, 1997; VandeWalle & Cummings, 1997), metacognitive skills (Chung & Reigeluth, 1992; Gall & Hannafin, 1994; Gay et al., 1991; Nelson, 1996), motivation to learn (Chung & Reigeluth, 1992; DeShon & Milner, 1998; Kinzie & Berdel, 1990; Lanza & Roselli, 1991; Porter, 1997; Simmering, 1999), need for achievement (Kjlch & Feldman, 1992), precourse attitudes (Hiltz, 1993), prior knowledge (Ashford & Northcraft, 1992; Chung & Reigeluth, 1992; Gall & Hannafin, 1994; Gay, 1986; Lee & Lee, 1991; Miller & Jablin, 1991; Steinberg, 1989), self efficacy (Major & Kozlowski, 1997; Miller & Jablin, 1991) and tolerance for ambiguity (Ashford & Cummings, 1983; Miller & Jablin, 1991). The following section details how some of these individual differences are expected to influence the typology of feedback, information, and practice seeking behaviors. 18 Antecedents of feedback seeking. The first individual difference that is likely to affect the feedback seeking of a trainee is the prior experience trainees have. It is important to distinguish between two different dimensions of prior experience, even though both may produce similar outcomes. The first dimension, which has received some attention in research, is the prior experience trainees have with the training topics and content. Some researchers suggest that trainees with less experience will seek more feedback (Miller & Jablin, 1991). However, Gay (1986) suggests that those who have more experience will seek more guidance feedback in order to make more efficient use of their time. The second dimension is the prior experience that trainees have with the media or method used for training. This has received little research, as previous training research has predominantly focused on traditional, instructor—led training courses. However, with WBT, the prior experience trainees have had with a variety of web-based events likely impacts how they operate in a WBT environment. For example, trainees who lack experience with web-based programs may be more likely to explore every single link available in the WBT program to make sure they do not miss anything. Although the trainees may learn a great deal, they are not taking advantage of the flexibility WBT affords and are unlikely to achieve efficiency. Thus, this dimension of prior experience should be distinguished and examined, even though trainees who lack content experience may behave similarly to trainees who lack internet experience. Self-efficacy (or self-confidence; self-esteem) is another individual difference that impacts seeking behavior. Miller & Jablin (1991) note that trainees with low self—esteem engage in less feedback seeking behavior. Presumably this is due to the self-perception l9 that one cannot successfully accomplish the learning task and a desire to avoid the negative feedback. However, low self-efficacy may likewise lead to greater seeking of guidance, for the same reason. This hypothesis has not been examined to date, although some researchers have suggested it (Major & Kozlowski, 1997), and others found evidence that individuals with low performance expectations sought more feedback (Northcraft & Ashford, 1990). Goal orientation is also expected to influence seeking behavior (Reeves & Reeves, 1997; VandeWalle & Cummings, 1997). For example, Butler (1993) found that those trainees who possess a mastery orientation are more likely to seek guidance, while those trainees with a performance orientation are more likely to seek evaluative feedback. VandeWalle and Cummings found that individuals with a mastery orientation are more likely to seek feedback than individuals with a performance orientation (mediated by cost and value perceptions). Finally, metacognitive skills may affect the feedback seeking of trainees. Particularly, metacognitive monitoring, as described by Nelson (1996), necessitates the observation of performance or learning feedback for later use in the metacognitive control processes. Antecedents of information seeking and practice seeking. Individual differences affect information and practice seeking behaviors similarly. Chung and Reigeluth (1992) suggest that trainees with greater prior knowledge seek information in less depth than those with less experience. Gall and Hannafin (1994) predict that prior knowledge leads to more efficient information seeking. While Lee and Lee (1991) included both prior knowledge and practice seeking in their design, they did not assess their relationship; they only reported that greater knowledge lead to more effective practice. 20 Major & Kozlowski (1997) discuss the intuitive expectation that higher self- efficacy leads to greater proactivity in information seeking. However, they maintain that those low in self-efficacy are more likely to perceive greater informational deficits (“real or imagined”) and seek more information. Although they did not test a direct relationship between efficacy and information seeking, they did find evidence for moderation. When tasks were interdependent and information was accessible, individuals with lower self- efficacy engaged in more information seeking. Gay and colleagues (1991) hypothesized that metacognitive skill would affect the information seeking behavior of trainees in a positive manner. They administered an “information processing” scale that assessed metalearning strategies such as elaboration, interrelating topics, and monitoring. No direct measurement of information seeking was taken though, and trainees with greater metacognitive skills appeared to achieve lower performance scores. Gall and Hannafin (1994) propose that the metacognitive skill level of trainees influences the breadth and depth of information sought. Metacognitive control skills, as described by Nelson (1996), likely leads to more information and practice seeking because these behaviors provide the best opportunities to change or adapt learning and performance. Operationalizations of Seeking Behaviors Most research involving feedback seeking has been examined within the job performance context (e.g., VandeWalle & Cummings, 1997). Butler (1993) has operationalized objective (descriptive) feedback as performance score, normative (evaluative) feedback as percentile score, and task information (guidance) as the three best solutions to the task. Butler found low intercorrelations among these different 21 feedback seeking behaviors (-.O9 to -.25). Gay et al. (1991) made a “guide mode” available to trainees throughout the training program. This guidance feedback listed for the trainee alternative areas for exploration based upon their tracked pattern of previous choices. Information and practice seeking have been operationalized in research even less often than feedback seeking; the few discussions of these topics have been theoretical. Gay et al. (1991) allowed trainees to choose which information topics they wanted to explore in the “browse mode.” Fisher (1995) allowed trainees to determine whether or not they wanted to complete a practice problem before continuing on to other topics or the performance test. These two studies were the only ones found to operationalize these elements of seeking behaviors. Limitations in Research on Seeking Behaviors The study of trainees’ seeking behaviors indicates several limitations in current research. First, the study of feedback seeking usually occurs within a performance context. Although interpretations and hypotheses can be made from these studies, research on feedback seeking in training and learning environments is needed. Training contexts are more likely to evoke a sense of a self-determined, intrinsic motivation for feedback seeking, whereas performance contexts may evoke more of a sense that feedback seeking is controlled by the valued extrinsic rewards of improving performance. Deci (1992) indicates that self-determined behavior that is intrinsically motivated will lead to more creative and flexible responses. Thus, the study of feedback seeking behavior is likely be different between learning and performance contexts. 22 Second, the outcomes evidenced in feedback studies typically are resulting from feedback that is automatically provided to participants. Research is needed in training contexts where trainees have the option of seeking feedback, and particularly the option to seek feedback in multiple forms. Third, there is little or no theoretical development of the constructs of information seeking and practice seeking. The present study suggests different conceptualizations for how information or practice may be sought by trainees at varying depths. The levels of content, frequency, complexity can indicate these depths of seeking behavior, and variability sought by trainees. Related to the previous limitation, the fourth limitation is the lack of direct research or operationalization of information and practice seeking. The present study intends to address these limitations through the empirical testing of hypotheses derived from this review of seeking behaviors. Metacognition Authors and researchers in the areas of WBT and computer-based instruction have called for the inclusion of metacognition in training design (Reeves & Reeves, 1997). These authors note that WBT has unique capabilities and a great deal of potential for developing metacognitive support. Glaser and Bassok (1989) remind us that self- regulatory skills such as metacognition are important outcomes of learning, and should thus be developed in training. However, aside from any benefits WBT may provide in promoting metacognition, WBT also increases the metacognitive demands on learners (Gall & Hannafin, 1994). Particularly, metacognitive skills are more important in unstructured learning I 23 environments (Smith, 1996; Smith et al., 1997), and WBT often represents such an unstructured environment. Chung and Reigeluth (1992) support this idea by identifying that the provision of strategy control (or metacognition) is useful when learners can control the number, types, and sequence of displays and examples in training. Control at this level is characteristic of the internet, and available for incorporation into WBT programs. Learners facing this lack of structure develop a more systematic, effective approach toward learning when they use metacognitive skills (Smith, 1996). Thus, researchers in WBT and computer-based instruction suggest that the inclusion of metacognition in training design will prompt learners to think more about their own learning process (Cates, 1992; Park & Hannafin, 1993). Metacognition will also promote deeper processing of information (Aberson, Berger, Emerson, & Romero, 1997; Jonassen & Grabinger, 1993; Milheim, 1996), particularly by assisting learners in integrating material and identifying interrelationships among learned concepts (Lanza, 1991; MacLachlan, 1986; Milheim, 1996; Park & Hannafin, 1993; Smith, 1996). Additionally, there have been continued calls for including metacognition in training design outside the specific area of WBT. For example, Redding (1990) recommends providing learners with metacognitive guidance in the form of strategies, heuristics, and mental models. Osman and Hannafin (1992) recommend embedding metacomprehension strategies within lessons or afford learners the opportunity to practice metacognitive strategies. Gall & Hannafin (1994) recommend “generic prompts to relate knowledge to previous acquired knowledge, conduct self-checks of understanding, or generate individual questions” (p. 228). Similarly, Howell and Cooke 24 (1989) suggest the inclusion of explicit diagnostic probes and implicit tests in the training material. Chi (1996) has found that “prompting” and “scaffolding” are two techniques used by tutors to promote students’ learning. Finally, Steinberg (1989) recommends inserting well-designed questions into lessons to assist learners in identifying important information in learner control environments. Training and educational researchers have recommended the use of metacognition in training contexts, and particularly in WBT programs. This is because WBT can support the development of metacognitive skills, yet also demands metacognitive skills for effective learning. The inclusion of metacognition in the design of WBT programs is expected to promote deeper processing of information, or a “mindfulness” during the learning process (Ivancic & Hesketh, 1995). Thus, a more detailed examination of the construct of metacognition is warranted. Descriptions of Metacognition Some authors have described metacognition in an uncomplicated way: the use of reflective thinking to develop self, task, and strategy awareness (Ridley, Schutz, Glanz, & Weinstein, 1992). However, a number of researchers have developed very specific descriptions and taxonomies of metacognition. First, two prototypical taxonomies are described; then others are presented and integrated with the prior examples. Garcia & Pintrich (1994) describe metacognition to include three mechanisms: monitoring, planning, and regulating. They note that these components are related, and even empirically correlated, but can be discussed and understood as separate elements. Monitoring involves tracking attention and comprehension during learning, as well as self-testing using questions about the learned material. Planning involves the analysis of 25 the learning situation to determine what cognitive strategies to activate, the priming of relevant prior knowledge, and the facilitation of organizing new knowledge. Regulating involves the correction of ineffective behavior and remediating deficits in comprehension. These three mechanisms work closely together. The monitoring mechanism allows the learner to evaluate the effectiveness of the planning mechanism and determine the extent to which the regulation mechanism should involve either maintenance or adaptation of the planned behavior and strategies. Nelson ( 1996) describes two aspects of metacognition: monitoring and control. There are two "levels" involved: the "object—level" at which the behavior or cognition is occurring, and the "meta-level" at which the behavior or cognition is regulated. In metacognitive monitoring, the object-level apprises the meta-level of what is occurring at the object-level. In metacognitive control, the meta-level modifies the object-level; this can be compared to the planning and regulating mechanisms of the Garcia and Pintrich model. The remainder of the descriptions of metacognition echoes these two or three basic categories of monitoring, planning, and regulating/controlling (henceforth called “controlling”). Some descriptions use these same terms, while others use different terminology. In the latter case, parenthetical information identifies the basic category with which it is interpreted to overlap. Flavell (1979) identifies three roles of metacognition: prompting goal setting and goal revision (planning), integrating and interrelating new and existing knowledge, and activating cognitive and metacognitive strategies (monitoring, controlling). Karoly ( 1993) identifies four aspects of metacognition: monitoring, planning, evaluating, and revising goal-directed behavior 26 (controlling). Ford and Kraiger (1995) describe metacognition as involving the monitoring, planning, and revising of goal-relevant behavior (controlling), and the understanding of relationships between task demands and individual capabilities. Kluwe (1987) identifies aspects of metacognitive control and metacognitive regulation, each having several components. Metacognitive control includes classification (i.e., "What am I doing?"; monitoring), checking (i.e., "How am I doing?"; monitoring), evaluation, and prediction (planning). Metacognitive regulation (controlling) includes either the modification or maintenance of: the content or target, processing capacity, processing intensity, and processing speed. Schraw and Dennison (1994) identify both knowledge of cognition and regulation of cognition, and provide evidence of a strong correlation between the two aspects of metacognition. Osman & Hannafin (1992) identify five components of metacognition: metamemory, metacomprehension (monitoring, controlling), self-regulation, schema training, and transfer. Bandura (1991) identifies three processes of self-regulation: self- observation (monitoring), self-judgment, and self-reaction (planning, controlling). Carver and Scheier (1982), and Miller, Galanter, and Pribram (1960) both identify the "Test-Operate-Test-Exit" self-regulation feedback loop, where testing exemplifies monitoring and operating/exiting exemplifies controlling. Although the various models of metacognition differ in some ways, they all recognize at least two basic subdivisions of metacognition. First there is some aspect of monitoring inherent in all taxonomies and descriptions of metacognition. Monitoring one’s own cognition and learning is an important part of the metacognitive process. Second, there is some aspect of exercising control or regulating behavior evident in all 27 models of metacognition. Metacognitive monitoring and metacognitive control seem to work together in most models, as monitoring provides the input upon which the decision to regulate is based (e. g., Nelson, 1996). These two components of metacognition form the foundation for the manipulation of metacognition in the present research. Outcomes of Metacognition In his conceptual piece, Flavell (1979) proposes that metacognition aides both learning and performance in complex tasks. Since then, numerous researchers have established empirical links with these training outcomes, as well as with transfer. The following sections describe the training outcomes of interest in this research: task performance, task learning, and seeking behaviors. Performance. High metacognitive skill has been associated with successful performance, while low metacognitive skill has been associated with failure in a number of studies (Alderman, Klein, Seeley, & Sanders, 1993; Ford et al., 1998; Swanson, 1990; Pintrich & De Groot, 1990; Pokay & Blumenfeld, 1990). For example, Alderman et al. (1993) found that less successful college students (as determined by class grade) reported less self-monitoring behaviors. Ford and colleagues (1998) found that trainees who engaged in greater metacognitive activity performed better and demonstrated greater levels of transfer than those engaging in less metacognitive activity. Vadhan and Stander (1994) found that better metacognitive skills (predicting one’s own exam grade) correlated well with performance (actual exam grade). Swanson (1990) found that metacognitive skills had a greater effect on performance than did general cognitive ability. Finally, Ridley et a1 (1992) found that the interaction between metacognitive skills and goal setting produced the best decision-making performance. 28 In each of these studies, metacognitive skills or activity was measured, as opposed to manipulated. In this way, metacognitive skills are viewed from more of a “trait” perspective. A later section presents studies that expressly manipulated the metacognitive skills of learners through instruction or other means, representing a “state” perspective. Learning. In addition to enhancing performance, metacognition has been found to enhance learning outcomes. In fact, metacognitive activity has even been defined as evidence of learning (Kraiger, Ford, & Salas, 1993). Ford and colleagues (1998) found that trainees who engaged in greater metacognitive activity demonstrated greater knowledge on a test of training material than did those engaging in less metacognitive activity. In research by Gay et al. ( 1991 ), students needed self-regulatory skills to navigate hypertext systems successfully and learn effectively. In related research, Nelson, Dunlosky, Graf, and Narens (1994) determined that the use of metacognition (as measured by Judgments of Learning) to re-allocate study effort led to better learning than other methods of allocating study effort. Etalapelto (1993) found that experts were superior to novices in both task and strategic metacognitive knowledge. Finally, Rinehart and Platt ( 1984) discuss Dewey's research on learning and summarize research demonstrating that metacognition improves learning. As with metacognition in studies of performance outcomes, metacognition in these studies was measured rather than manipulated. Seeking behavior. A third outcome of metacognition is the depth of seeking behaviors that it may promote. Although these seeking behaviors can also be seen as a necessary part of the ongoing metacognitive process, they are important outcomes of 29 training in their own right. As discussed previously, Nelson (1996) has identified two aspects of metacognition: monitoring and control. There is reason to believe that metacognitive monitoring will lead to greater feedback seeking behavior. VandeWalle and Cummings (1997) describe monitoring as one of two ways in which feedback seeking is achieved. Metacognitive control, on the other hand, will likely lead to greater information and practice seeking behaviors. Acquiring more knowledge or skills through information seeking and applying the knowledge or skills through practice are the primary means trainees have of controlling their training-related behavior. These three outcomes—performance, learning, and seeking behaviors—have largely been studied in relation to metacognitive skills as an individual difference variable. However, the relationships of metacognition with seeking behaviors are expected to be similar when metacognition is manipulated. The next sections describe research that has expressly manipulated metacognition in training programs. Operationalizations of Metacognition Metacognition has been operationalized in a number of different ways in empirical research studying this cognitive phenomenon. The following section describes some of the Operationalizations of monitoring, planning, and controlling aspects of metacognition in recent literature. One operationalization of the construct, however, is more general. Ridley et al (1992) measure metacognition with the Private Self- consciousness subscale of Academic Self-Regulation Scale, developed by Fenigstein, Scheier, & Buss (1975). Monitoring. Carver & Scheier (1990) attempted to assess the "metamonitoring" of learners by having them visualize possible courses of action as positive behavior 30 scenarios. Learners were then asked to monitor their rates of progress. Nelson et al (1994) has measured metacognitive monitoring by having learners make Judgments of Learning regarding vocabulary words learned. Nelson has also measured “metamemory” by having learners rate their Feelings of Knowing for learned material (Nelson, 1996; Nelson & Narens, 1990). Other researchers have also operationalized metacognitive monitoring to include learners engaging in “compensatory functions”: determining difficult to learn material, self-testing, and looking for holes in their learning (DiVesta & Morena, 1993; Simpson, Hayes, Stahl, & Weaver, 1988). Although Ford and Kraiger (1995) do not describe an empirical study, they recommend providing reflection time to prompt metacognitive monitoring. This reflection time should include prompts for the learner to question, clarify, summarize, and make predictions about learned material (Glaser & Bassok, 1989). Glaser and Bassok discuss three Operationalizations of metacognitive monitoring that were developed in different educational programs. First, Brown & Palincsar (1984) produced a method called reciprocal teaching, wherein the teacher first models the appropriate metacognitive monitoring skills and then allows learners to practice the skills. This process involves the “scaffolding” of teacher guidance (see also, Chi, 1996), which can be withdrawn as learners become more adept at metacognitive monitoring. Second, Scardamalia, Bereiter, and Steinbach (1984) developed a similar method called procedural facilitation, where students learning to write receive prompts to identify goals, strive for cohesion, and evaluate progress. Clearly, this method also incorporates metacognitive planning. Third, Schoenfeld (1983) teaches learners to ask self-regulatory 31 questions and evaluate alternative courses of action through a heuristical mathematics problem-solving method. This method also has implications for metacognitive control. Planning and controlling. Earley, Wojnaroski, and Prest (1987) have operationalized the planning aspect of metacognition as the setting of specific goals. Weissbein (1996) asked trainees to determine what task items would be most difficult to learn in order to plan what to focus on during study time. Smith (1996) instructed trainees to list the behaviors they would practice in order to reach their learning goal. Finally, Greiner and Karoly (1976) trained students in planning strategies that enabled them to decide the content, amount, and other aspects of their study time. Metacognitive control has received direct research as well. Zacks (1969) assessed metacognitive control as the reallocation of study time. In the Nelson et al (1994) study, metacognitive control was also measured by the learners’ use of Judgments of Learning to reallocate study time. Similarly, Justice and Weaver-McDougall (1989) operationalized metacognitive control as the learners’ changed judgments of relative effectiveness of different task strategies they have tried. Pressley, Levin, and Ghatala (1984) assessed metacognitive control via the learners’ modification of goals and actions. Experiments Training/Manipulating Metacogpition Training metacognition. A number of empirical research studies have attempted to instruct trainees how to engage in metacognitive behaviors, including planning, monitoring, evaluating, and adapting their learning process and training performance. First the successful experiments are presented. Bean et al., (1986) trained high school students to summarize, generate questions, and evaluate a reading passage. When these metacognitive skills were combined with a specific "graphic organizer” strategy, the 32 result was better test performance and generalization to a more difficult reading than students who were trained only to use outlining or the organizer strategy. In Seabaugh and Schumaker’s ( 1994) within-subjects study, when students were instructed on monitoring and self-reinforcement they had a better completion rate in self-paced, individualized lessons. Volet (1991) taught undergraduates planning strategies, monitoring, and evaluation. The trained group produced better grades and better transfer of learning to new problems compared to the control group. Finally, Greiner & Karoly (1976) found that students trained in metacognitive monitoring, planning, and self-reward spent more time studying and performed better (on several indicators) than groups trained in monitoring alone, or monitoring and self-reward. Not all attempts to train metacognitive skills have been effective, however. Weissbein (1996) taught metacognitive skills (planning learning, evaluating learning, and evaluating strategies), emphasized their importance, and provided opportunities to practice the learned metacognitive skills. Trainees in the control group engaged in all practice activities, but without specific instruction or mention of metacognition. (The control group may therefore have experienced a form of metacognitive prompting, rather than training, which is discussed in the following section.) Weissbein found minimal support for only the quantity of metacognition on training outcomes. Brown et al (1997) similarly found no support for metacognitive training in improving learning or performance. The authors suggest that the lack of results may be due to a poor measure of metacognitive activity, or a poor manipulation of metacognitive skills with insufficient guidance. Finally, Smith (1996) found only an interaction of metacognition with guidance. Trainees were taught to plan, monitor, and evaluate task performance and why 33 this process was important. They were also asked questions, given time to reflect, and required to set goals and create plans for meeting their goals. Thus, Smith’s study included a combination of metacognitive training and metacognitive prompts. Prompting metacognition. A number of studies have attempted to induce metacognitive activity by prompting the appropriate behavior, rather than trying to explicitly train metacognitive skills. Avner et al., (1980) instructed students in a chemistry lesson. Throughout the lesson there were prompts for “active processing” which required responses on the part of the students in order to continue the session. Significantly less errors were recorded in later chemistry experiments for students who were prompted during the lesson. J acobsen and Spiro (1995) placed emphasis during training on deducing interrelations among and developing integration for course topics, which resulted in worse performance but better transfer. Lorene, Sturmey, and Brittain (1992) asked stroke patients to monitor for comprehension as they read through an information packet. Monitors recalled more on the knowledge test than did controls. Pressley et al. ( 1987) measured “perceived readiness for exam performance,”— the predicted percentage of items recalled—before reading a passage, after reading, and after the test. Predictions were better after reading only when questions were asked during reading. Questions interspersed throughout the passage were determined to provide better results than questions massed at the end of the passage. Veenman, Elshout, and Busato (1994) found metacognition improved training outcomes; however, the metacognitive manipulation was confounded with a guidance manipulation. From these examples it can be seen that prompting metacognitive activity seems to lead to better success than attempting to explicitly train metacognitive skills. 34 Limitations to Research on Metacognition Research on metacognition is undoubtedly more developed than that of seeking behaviors; nevertheless, limitations exist in this literature as well. One of the most notable limitations has been the inconsistent results associated with programs that have attempted to directly train metacognitive skills. Research has suggested that metacognitive skills can be fostered during training by directly teaching the use of such skills and by indirectly prompting trainees to engage in metacognition. The prompting method may produce more consistent results. Another inconsistency is the operationalization of metacognitive dimensions within empirical studies. Theoretical discussions of metacognition seem to agree on two basic components of metacognition—monitoring and control. However, these components are not often operationalized independently in empirical research. Such a separation would allow better examination of how these components contribute uniquely to training outcomes. Although there are clear benefits to facilitating metacognition in learner control environments, little research has incorporated metacognition in studies of learner control. Learner control literature typically focuses on the comparison of learner control and program control designs, to the exclusion of other relevant constructs such as metacognition. Finally there is no direct connection in theory or research between metacognition, seeking behaviors, and training outcomes. Nelson (1996) describes the framework of metacognitive monitoring and metacognitive control in such a way that suggests that 35 these components of metacognition might differentially produce feedback, information, and practice seeking on the part of trainees. Hypotheses WBT programs allow greater flexibility in training design and delivery than do traditional, instructor-led training programs. One benefit of this is that training can be designed to fit with the individual differences of trainees. Trainees who have different needs can select different paths to training completion. However, this greater flexibility can also be detrimental, as it can lead to greater confusion. Trainees may need more direction than is ordinarily provided in order to successfully negotiate the learning environment. Theory and research can contribute to WBT by determining how to best leverage the advantages and remedy the deficiencies associated with this method. The literature reviewed highlighted limitations in current research relevant to WBT. These limitations form the foundation upon which the present study is built. First, the study represents a training environment, to allow an exploration of the effects of seeking behaviors on the part of trainees. Past research has tended to examine feedback seeking in a performance environment (e. g., Ashford & Northcraft, 1992). Because the focus in the present study is on training, the individual difference variables that are examined are those that have been found to be particularly relevant to the learning process. Specifically these individual differences include prior experience with training content or methods (Gay, 1986; Miller & Jablin, 1991), mastery and performance orientations (Ames & Archer, 1988; Butler, 1993; Ford et al., 1998), self-efficacy (Miller & Jablin, 1991; Schunk, 1989; Mathieu, Martineau, & Tannenbaum, 1993), and metacognitive skills (Ford et al., 1998). Thus, one contribution of this research is the 36 exploration of the effects of individual differences in trainees on their engagement in seeking behaviors in a WBT environment. Seeking behaviors are particularly relevant in learner controlled WBT environments, as trainees typically have numerous options for the path they chose to progress through training. The construct of feedback seeking has been well researched; two other types of seeking behaviors, information seeking and practice seeking, have been less widely examined. Past research has tended to focus on one type of seeking behavior at a time. Another contribution of the present research, therefore, is to theoretically and operationally unite these three types of seeking behaviors in one study; in this way, a typology of seeking behaviors is constructed and tested. However, the opportunity to engage in different seeking behaviors in learner controlled WBT programs may leave trainees disoriented or confused. They may not achieve the same stability of knowledge structure that might result from a more structured approach to learning (Gall & Hannafin, 1994). Thus, a study that includes opportunities to engage in seeking behaviors is also well served to include metacognitive prompts in its design. Prompts to engage in metacognition will assist learners in integrating knowledge they are gaining in varied sequences and from different sources, including information, practice, and feedback. Finally, metacognition has been represented in theory as having different dimensions, including monitoring and control (i.e., Nelson, 1996). However, past research has typically combined the theoretical dimensions of metacognition into a single operationalization. The present study separates metacognitive monitoring and metacognitive control operationally in order to examine differential effects these 37 dimensions may have on training outcomes. More importantly, the different dimensions of metacognition may differentially impact the seeking behaviors of trainees. Thus, the unique combination of seeking behaviors and metacognitive manipulations presented in this study contributes to the literature in two ways. First, it allows the investigation of whether the presence of metacognitive prompts can benefit trainees in a learner controlled WBT program with myriad seeking opportunities. Second, it allows the investigation of the differential power of the monitoring and control aspects of metacognition to impact different seeking behaviors. Overview The general pattern of relationships expected for the variables examined is presented in Figure l. The individual difference variables are exogenous, and they are expected to influence both the breadth and depth of seeking behaviors. The impact of a number of individual difference variables on training success has long been known. However, this study proposes that, in learner control environments such as WBT, they have their effects through the process of seeking information, practice, and feedback. The metacognitive manipulations are also exogenous variables and are expected to interact with the relationship of individual differences (particularly metacognitive skills) and depth of seeking behaviors. Metacognitive manipulations are also expected to influence the training outcomes directly. This direct effect is specified, as metacognition likely impacts learning and performance through process variables that are not examined in this study such as the development of knowledge structure. Finally, breadth and depth of seeking behaviors impact the training outcomes, completing the process. 38 Breadth of Seeking Behaviors Individual Differences Training Outcomes Depth of Seeking Behaviors Metacognitive Manipulations Figure 1. Expected pattern of relationships among individual differences, manipulations, seeking behaviors, and training outcomes. The specific hypotheses for each of the separate variables and factors are detailed in the remainder of the section. They are not presented in a figure, due to the method of data analysis. Instead, hypotheses are presented in a multiple regression framework, with each hypothesis representing a regression of several predictors on one dependent variable. Effects on Breadth and Depth of Seeking Behaviors Breadth of seeking behzgiors. The first general expectation is that individual difference variables impact the breadth of seeking behaviors in which trainees engage. The specific individual differences that are expected to have this impact are mastery orientation and prior experience. Trainees with a high mastery orientation tend to demonstrate a willingness to explore and try new approaches in order to better accomplish their goal of learning (Ames & Archer, 1988). One way in which they might 39 explore is to try the different links to information, practice, and feedback. Presumably by exploring these links they would find that the variety helps them to learn better, and they would continue to engage in a greater breadth of seeking behaviors than trainees with a low mastery orientation. Similarly, trainees with less experience—either with the content of training or the WBT method—would more likely explore all of the sources available to them, in order to increase their knowledge and experience. H1: Breadth of seeking behaviors will be predicted by mastery orientation (+), and prior experience (-). Depth of feedback seeking. The content of feedback seeking is represented by descriptive feedback, evaluative feedback, and guidance feedback. Individual difference variables are expected to influence each type of feedback content. For example, trainees with a high mastery orientation and trainees with less prior experience are expected to seek more descriptive feedback because they want to improve their knowledge. Monitoring the accuracy of newly acquired knowledge can help correct deficiencies in that knowledge. H2a: Descriptive feedback seeking will be predicted by mastery orientation (+), and prior experience (-). However, trainees with a high performance orientation prefer to prove their competence in any given area, and desire to demonstrate their superiority to others (Ames & Archer, 1988). Thus trainees who are highly performance oriented are expected to seek more evaluative feedback than other trainees, as this provides an indication of how they are faring compared to others in training. H2b: Evaluative feedback seeking will be predicted by pelformance orientation (+). 40 Trainees with a high mastery orientation are also expected to seek more guidance feedback, as it assists them in their quest to learn. Trainees low in self-efficacy may seek more guidance, due to their lack of confidence in their own abilities. They prefer to seek assistance in determining the best course for learning, as do trainees with less prior experience. H2c: Guidance feedback seeking will be predicted by mastery orientation (+ ), self- efiicacy (-), and prior experience (-) Trainees with high mastery orientations, trainees with low prior experience, and trainees with low self-efficacy are expected to engage in more total feedback seeking than other trainees, for the reasons that have been previously discussed. Additionally, metacognitive skills may influence the frequency of feedback seeking in which trainees engage. Metacognitive skills include the tendency to monitor learning behavior, and seeking feedback is central to the monitoring process. H3: Frequency of feedback seeking will be predicted by mastery orientation (+), prior experience (-), self-efficacy (-), and metacognitive skills (+) Trainees who receive prompts to engage in metacognitive monitoring are also expected to engage in greater depths of feedback seeking. Thus, an interaction is expected between the trait and state perspectives: the metacognitive skills (trait) of a trainee and the metacognitive monitoring manipulation (state). The basic form of this interaction is depicted in Figure 2. The interaction is such that trainees who have low metacognitive skills benefit most from the metacognitive monitoring prompts, as it boosts them to the same level of feedback seeking as those trainees with high metacognitive skills. However, trainees with higher metacognitive skills benefit less from the 41 No MC Manipulation § . . MC Manipulation a .C.‘ d) m 01) I: E Q) Q) m Low High MC skill MC skill Figure 2. Expected interaction between metacognitive skills and metacognitive manipulation. metacognitive monitoring manipulation, and those with extremely high skill may even be somewhat hindered by the prompts. H4: Metacognitive monitoring will interact with metacognitive skills such that, for trainees not receiving metacognitive monitoring prompts there is a strong positive relationship between metacognitive skills and depth of feedback seeking, whereas for trainees receiving metacognitive monitoring prompts there is a weak or no relationship between metacognitive skills and depth of feedback seeking. Depth of information seeking. Trainees with high mastery orientations are expected to engage in information seeking at greater depths than those with low mastery orientation, because they would believe that more complex information enhances their learning. Trainees with more prior experience are expected to seek out more complex information because they already have a foundation of knowledge at a lower depth. Trainees with greater self-efficacy are expected to seek information at higher levels of complexity because they perceive themselves as having the ability to handle the challenge of greater complexity. 42 H5: Complexity of information seeking will be predicted by mastery orientation (+), prior experience (+ ), and self-efficacy (+ ) Trainees with high mastery orientations, trainees with low prior experience, and trainees with low self-efficacy are expected to engage in more total information seeking than other trainees in order to increase their learning, knowledge, and confidence. However, trainees high in performance orientation are not interested in increasing learning, but instead are motivated to complete the training program as soon as possible. High performance oriented trainees thus engage in lower frequency of information seeking. Additionally, metacognitive skills may influence the amount of information seeking in which trainees engage. Trainees with higher metacognitive skills are expected to seek out information necessary to building their knowledge based on their assessment of deficiencies in their learning progress. H6: Frequency of information seeking will be predicted by mastery orientation (+), performance orientation (-), prior experience (-), self-efficacy (-), and metacognitive skills (+) Trainees who receive prompts to engage in metacognitive control are likewise expected to seek information at greater depths, as the acquisition of new information is central to controlling the learning process. Thus, an interaction is expected between the metacognitive skills (trait) of a trainee and the metacognitive control manipulation (state). The basic form of this interaction is depicted in Figure 2. The interaction is such that trainees who have low metacognitive skills benefit most from the metacognitive control prompts, as it boosts them to the same level of information seeking as those trainees with high metacognitive skills. However, trainees with higher metacognitive skills benefit less from the metacognitive control manipulation, and those with extremely high skill may even be somewhat hindered by the prompts. 43 H7: Metacognitive control will interact with metacognitive skills such that, for trainees not receiving metacognitive control prompts there is a strong positive relationship between metacognitive skills and depth of information seeking, whereas for trainees receiving metacognitive control prompts there is a weak or no relationship between metacognitive skills and depth of information seeking. Depth of practice seekiig. Trainees with high mastery orientations are expected to engage in practice seeking at greater depths than those with low mastery orientation, because they believe that more complex practice enhances their learning. Trainees with more prior experience are expected to seek out more complex practice because they already have practiced their skills at a lesser depth. Trainees with greater self-efficacy are expected to seek practice at higher levels of complexity because they perceive themselves as having the ability to handle these challenges. These individual differences are expected to affect variability of practice seeking in a similar manner; however, the scope of the present study does not allow for the inclusion of practice variability. H8: Complexity of practice seeking will be predicted by mastery orientation (+), prior experience {-1- ), and self-efficacy (+ ). Trainees with high mastery orientations, trainees with low prior experience, and trainees with low self-efficacy are expected to engage in more total practice seeking than other trainees in order to improve their newly acquired skills. Again, trainees high in performance orientation are not interested in skill improvement, as they prefer to practice skills they have already perfected. High performance oriented trainees are thus expected to engage in lower frequency of practice seeking. Metacognitive skills may influence the amount of practice seeking in which trainees engage. Trainees with higher metacognitive skills are expected to seek out practice opportunities necessary to building their skills based on their assessment of skill deficiencies. 44 H9: Frequency of practice seeking will be predicted by mastery orientation (+),performance orientation (-), prior experience (-), self-efficacy (-), and metacognitive skills (+ ) Trainees who receive prompts to engage in metacognitive control are expected to seek greater depths of practice than other trainees, as the practice of learned skills is central to controlling the learning and performance processes. An interaction is expected between the metacognitive skills (trait) of a trainee and the metacognitive control manipulation (state). The basic form of this interaction is depicted in Figure 2. The interaction is such that trainees who have low metacognitive skills benefit most from the metacognitive control prompts, as it boosts them to the same level of practice seeking as those trainees with high metacognitive skills. However, trainees with higher metacognitive skills benefit less from the metacognitive control manipulation, and those with extremely high skill may even be somewhat hindered by the prompts. H10: Metacognitive control will interact with metacognitive skills such that, for trainees not receiving metacognitive control prompts there is a strong positive relationship between metacognitive skills and depth of practice seeking, whereas for trainees receiving metacognitive control prompts there is a weak or no relationship between metacognitive skills and depth of practice seeking. Effects on Training Outcomes Learning. The greater breadth of seeking behaviors in which trainees engage likely influences their learning during training. Trainees who learn information, practice their newly acquired skills, and then monitor feedback on their progress are more actively engaged in learning and are thus more likely to improve their learning (Noe & Ford, 1992). Additionally, trainees are expected to enhance their learning when they engage in greater depths of information seeking, the type of seeking behavior most directly related to learning. 45 H11: Learning will be predicted by breadth of seeking (+), complexity of information seeking (+), and frequency of information seeking (+) Trainees who are prompted to both monitor and control their learning process are expected to result in the best learning of all trainees. This is because these trainees are monitoring the results of their learning behavior, and then using this information to determine how to adjust their learning process. Following any adjustments, they once again monitor their progress, and the cycle continues. Those trainees who are just monitoring or just controlling their learning process are less likely to take advantage of the full cycle of metacognition to improve their learning. H12: Metacognitive monitoring and metacognitive control will interact such that trainees receiving prompts for both metacognitive monitoring and metacognitive control will learn better than all other groups. Efficiency. The efficiency construct is a ratio of learning to the amount of time spent learning. No matter how quickly trainees progress through the training, they cannot achieve efficiency if they have not adequately learned the training material. However, trainees who have learned a great deal by engaging in great depths of feedback, information, and practice seeking may also take a large amount of time to complete the training. The more time taken to complete training, the less efficient the learning process. One of the advantages espoused for WBT is that it allows flexibility of seeking behaviors and thus allows efficiency of learning (Schreiber, 1998). Trainees are expected to judiciously choose among the links to feedback, seeking those links that will best enhance their learning. Therefore, the frequency with which trainees engage in these seeking behaviors is expected to have negative relationships to the efficiency outcome. The more frequently a trainee seeks information, practice, or feedback, the less efficient their learning process. 46 H13: Efiiciency will be predicted by frequency of feedback seeking (-), frequency of information seeking H, and frequency of practice seeking (-). Final self-efficacg Active participation in learning leads to a strong sense of capability with regard to learned material and skills. Thus, increased breadth of seeking behaviors of trainees is expected to enhance self-efficacy at the completion of training. Additionally, trainees who engage in greater depths of information seeking and practice seeking are expected to have higher self-efficacy following training. Initial self-efficacy is also likely to predict final self—efficacy, so it should be used as a covariate in analyses of other relationships to final self-efficacy. H14: Final self-efiicacy will be predicted by self-efiicacy (+), breadth of seeking (+), complexity of information seeking (+), frequency of information seeking (+), complexity of practice seeking (+ ), and frequency of practice seeking (+) Performance. The greater breadth of seeking behaviors in which trainees engage is expected to influence their performance during training. In addition to improving learning, trainees who are more actively engaged during training are also more likely to improve their performance (Noe & Ford, 1992). Trainees seeking descriptive feedback and guidance feedback are likely improve their training performance, whereas those seeking evaluative feedback may hinder their training performance. This is because descriptive and guidance feedback provide specific information that can be used to adapt learning processes; evaluative feedback doesn’t inform the learning process except to signal how an individual is doing normatively. Overall, trainees who engage in greater depths of feedback seeking are expected to enhance their performance, as it enables them to assess their learning progress. Additionally, trainees are expected to enhance their performance when they engage in greater depths of practice seeking, the type of seeking behavior most directly related to performance. 47 H15: Performance will be predicted by breadth of seeking (+), content of feedback seeking (descriptive +, evaluative -, guidance +), frequency of feedback seeking (+), complexity of practice seeking (+), and frequency of practice seeking (+) Trainees who are prompted to both monitor and control their learning process are also expected to result in the best performance of all trainees. As previously discussed, this is because these trainees are taking advantage of the full cycle of metacognitive monitoring and metacognitive controlling. H16: Metacognitive monitoring and metacognitive control will interact such that trainees receiving prompts for both metacognitive monitoring and metacognitive control will perform better than all other groups. 48 METHOD Design The present study incorporated a crossed 2 x 2 factorial design. The first factor was metacognitive monitoring (MCM) versus no metacognitive monitoring. The second factor was metacognitive control (MCC) versus no metacognitive control. Thus there were four groups: one received both MCM and MCC; one received only MCM; one received only MCC; and one control group that received no manipulations. Participants Participants included 160 volunteers from psychology courses at a large Midwestern university, who received class credit for their participation. Two participants did not complete the experiment, and three participants’ data were excluded from analyses because of missing data, resulting in a total of 155. Procedure Participants completed the experiment on a computer equipped with Microsoft Internet Explorer 4.0 and Microsoft Word 97 (with web-authoring tools installed). Upon beginning the experiment they read instructions that included a brief description of the content and nature of the training program. Trainees were told that they would learn how to design their own internet web page, and that instruction would be provided at basic, - advanced, and expert levels in each of five lessons. They were told that they would have the opportunity to practice what they learned, that they would be required to take a basic- level quiz at the end of each lesson, and that they could receive feedback on their quiz results. They were also told that at the conclusion of the training they would submit the 49 internet address for a web page that they created, which could include any elements that they have learned during the training. Next, participants read the informed consent information and indicated their agreement to participate by entering their personal identification number (PD) and continuing with the training program. Anyone who did not wish to participate could discontinue the training program at that point, or at any point during the training. The consent form explained the nature and procedures of the experiment, the risks and benefits of the experiment, and their right to withdraw participation at any time without penalty (Appendix A). Contact information was provided for the researcher, Psychology Department, and University Committee on Research Involving Human Subjects. Participants then completed several short measures of their individual characteristics. Specifically, they answered questionnaires to establish their mastery and performance goal orientations, self-efficacy, metacognitive skills, and prior experience with the internet and creating web pages. Participants also were asked to provide an ACT or SAT score, if available, to be used as a surrogate measure of cognitive ability. Following these measures, trainees were asked to open the Microsoft Word program and save a document into HTML format. (HTML is the programming language used to create web pages.) This HTML document would be available for their use in practice exercises, and to construct their web page to submit at the conclusion of the training. The experimenter was available to assist any trainees who need help with this step, and throughout the experiment.1 1 . . . . Help was provrded when questions arose because the trarnrn g program was unclear, or there were technical problems. 50 Five training lessons were presented to participants. Training lessons could be accessed in any sequence; the only limitation was that the quiz for each lesson must be taken. Each lesson presented information at basic, advanced, and expert levels of difficulty on one t0pic of web page design. Participants could choose one or more levels of information to read and learn (“Information seeking”). Trainees also had the opportunity to choose practice exercises at these three levels of difficulty (“Practice seeking”), and the lesson quiz which presented basic-level multiple choice questions. Following the mandatory quiz, trainees could opt to view any of three kinds of feedback (“Feedback seeking”): descriptive feedback that showed the correct answers; evaluative feedback that showed the number correct for the average trainee; and guidance feedback that suggested which lesson trainees should learn next. Some trainees experienced one or both of the metacognitive prompt manipulations. MCM and MCC prompts appeared following the selection of a new topic. MCM prompts asked trainees to reflect on the lesson they had just learned and how it related to other lessons, as well as what they knew and didn’t know about the lesson. MCC prompts asked the trainee to reflect on strategies that they used to learn, how they could improve them, and to plan their learning for the next lesson. When all five training lessons had been completed the participants were reminded that they would submit the address for the web page that they created, so that the experimenter could assess their learning of web page design skills. They were then asked to submit the internet address for their web page. At this point, the experimenter checked to ensure that the web pages were properly loaded onto the Internet via the University server. Participants completed a final, paper-and-pencil questionnaire to assess their 51 post-training self-efficacy. Then they read a debriefing statement detailing the nature of the experiment and had an opportunity to ask the experimenter questions about the study (Appendix A). They were also given an address to a website that contained all of the information and practice exercises for the five lessons, so that they could continue to work on their web pages. The experimenter indicated that she would be available to answer questions about web page design for two weeks after completion of the experiment. These efforts were made to prompt continued work on the web pages after the training, with the intent of measuring performance transfer. Manipulations The manipulations described in this section each represented one of two factors that were completely crossed in the design. Participants were assigned to one of these four conditions on a random basis; the program made the random assignment when the participant began the training. As a result, 38 received only metacognitive monitoring prompts (MCM), 39 received only metacognitive control prompts (MCC), 39 received both prompts (MCM & MCC), and 39 participants did not receive any prompts. Metacognitive monitoring. A few experimental studies have operationalized and manipulated monitoring components of metacognition. Greiner and Karoly (1976) asked students to monitor their progress in a course by recording their study activity in a log. Weissbein (1996) taught learners how to employ a process of self-testing. They were instructed to look for gaps in their knowledge by determining how well they know something and how well they needed to know it. Finally, when Pressley et al. (1987) interspersed questions in a reading passage, students were asked to answer the questions 52 in their heads, rather than looking back at the information, in order to see whether they knew the information or not. These elements were incorporated into this study’s manipulation of metacognitive monitoring. Prompting trainees at the end of each lesson, rather than training them and then relying on them to remember to self-test, was expected to be a relatively strong manipulation. Requiring trainees to write down answers to the prompts, rather than to just think about them, was also expected to strengthen the metacognitive manipulation. Finally, asking the trainees to think without looking back at the lesson would help them achieve the knowledge integration that was a proposed benefit of metacognition. As detailed in Appendix B, trainees were asked to think about how they were learning the information in the training program. (The same metacognitive prompts were provided between each of the training lessons.) Trainees were asked to note what information they knew and what information they needed to know. They were also asked to relate the information they had just learned to information in the previous lesson and the upcoming lesson. Responses were to be entered for each prompt before the trainee could move on to the next lesson. These responses were recorded to be used as a manipulation check, if necessary. Metacognitive control. Some of the same experimental studies have also operationalized and manipulated control or regulation components of metacognition. Greiner and Karoly (1976) asked students to structure in advance when, where, and how much studying they were going to do. This represents planning, which can be considered an aspect of metacognitive control. Weissbein (1996) manipulated metacognitive regulation by asking learners to think about what approach or strategy they were using to 53 learn, whether it was working, and how they could improve it. Specific questions included, "What strategy did you use? How did you know if it was working? Name three ways you can improve.” Again, these elements were incorporated into this study's manipulation of metacognitive control. Prompting trainees at the end of each lesson, requiring them to think about, and then write down their responses was expected to be a relatively strong manipulation. Combining both the evaluation of learning strategies and a planning of future strategies would help trainees achieve the integration of knowledge that was a proposed benefit of metacognition. As detailed in Appendix B, trainees were asked to think about how they were learning the information in the training program. (The same metacognitive prompts were provided between each of the training lessons.) Trainees were asked to note what approach or strategy they were using to learn the information in the last lesson and how they could improve it. Then they were asked to list three steps in their plan for approaching the next lesson. Responses were to be entered for each prompt before the trainee could move on to the next lesson. These responses were recorded to be used as a manipulation check, if necessary. Measures Individual differences. The first individual difference, prior experience, actually represented a multidimensional construct. Prior experience with the internet and a variety of web-based activities such as browsing, searching, and making purchases over the web established the participant's experience and comfort level with participating in a web-based training program. This prior experience with web programs was assessed 54 with 8 items. Response options to the items were on a five-point Likert scale ranging from “all the time” (5) to “never” (1). One item was dropped due to unreliability, resulting in a scale reliability of (1:80. Another dimension, prior experience with web page design, was assessed with one item asking participants to estimate the number of hours of direct experience they had with designing and creating web pages. These scales are presented in Appendix C. Participants' self-eflicacy for the training course was measured with a 9-item scale adapted from Pintrich and De Groot (1990). The original scale demonstrated high internal consistency (a = .89), and the minor revisions did not alter the reliability (0t = .90). An example of the items in this scale is, “I’m certain I can understand the ideas taught in this course.” Response options to the items were on a five-point Likert scale ranging from “strongly agree” (5) to “strongly disagree” (1). This scale is presented in Appendix C. Mastery and performance goal orientation was assessed with a l6-item scale developed by Button, Mathieu, and Zajac (1996). The mastery factor (8 items) demonstrated and alpha level of .85, and included items such as, “The opportunity to learn new things is important to me.” The performance factor (8 items) demonstrated an alpha of .84, and included such items as, “I feel smart when Ican do something better than most people.” Response options to the items were on a five-point Likert scale ranging from “strongly agree” (5) to “strongly disagree” (1). This scale is presented in Appendix C. The level of metacognitive skills of participants was measured with a 7-item scale adapted from Pintrich and De Groot (1990). The internal consistency reliability of the original 9-item scale was adequate (or = .74). However, it was necessary to revise the 55 wording of the scale and remove 3 items relating to effort management strategies. One item from Pokay and Blumenfeld (1990) was revised and added, but was dropped from the final scale used in analyses due to unreliability. An example of the items in this scale is, “I ask myself questions to make sure I know the material I have been studying.” Response options to the items were on a five-point Likert scale ranging from “strongly agree” (5) to “strongly disagree” (1). This scale is presented in Appendix C. Finally, the measure of cognitive ability was derived from participants’ ACT or SAT scores, which most students were able to provide. ACT and SAT scores were then transformed to z-scores using normative data from the 1999 administrations of these standardized tests. The z—scores were used in analyses involving cognitive ability. Breadth of seeking behaviors. The breadth of seeking in which trainees engage was established by examining the links chosen by the trainees and recorded in the database. Trainees could have a breadth score that ranges from zero (0) to three (3) for each training lesson. For each of the types of seeking behavior in which they engaged— feedback seeking, information seeking, and practice seeking—they received one point toward their breadth score. These scores were then averaged across the five lessons, resulting in a zero (0) to three (3) range for the entire training program. Depth of seeking behaviors. The depth of feedback seeking was established for both frequency and content dimensions. The frequency of feedback seeking was represented by the number of times that the trainee accessed any link to feedback, summed across all lessons, resulting in a zero (0) to seventeen ( 17) range. The content of feedback seeking was reflected in the particular feedback links chosen by the trainee. Trainees who sought feedback on the correct answers to the quiz questions were scored 56 one point for seeking descriptive feedback. Trainees who sought feedback on the average quiz score of trainees were scored one point for evaluative feedback. Trainees who sought feedback that suggested two lessons to learn next were scored one point for guidance feedback. When summed across lessons, this produced a zero (0) to five (5) range for each type of feedback seeking—descriptive, evaluative, and guidance feedback seeking. The depth of information seeking was established for both frequency and complexity dimensions. The frequency of information seeking was represented by the number of times the trainee accessed any link to information, including multiple instances of seeking the same information link. This resulted in a zero (0) to twenty-six (26) range. The complexity of information seeking was scored on a zero (0) to three (3) scale, with one point scored for each level of complexity sought. This value was then averaged across all lessons. Thus, if a trainee sought information only at the basic level, they were scored one point; if a trainee sought information at the basic and advanced level, or only at the advanced level, they were scored two points. In this way, complexity reflected the highest level at which a trainee sough information, whereas frequency reflected the total number of levels and number of times at which the trainee sought information. The depth of practice seeking was established for frequency and complexity dimensions. The frequency of practice seeking was represented by the number of times that the trainee accessed any link to practice, including multiple instances of seeking the same practice link. This resulted in a zero (0) to twenty-five (25) range. The complexity of practice seeking was scored on a zero (0) to three (3) scale, with one point scored for 57 each level of complexity sought. The assessment of complexity of practice seeking mirrored the assessment of complexity of information seeking, reflecting the highest level at which a trainee sought practice. This value was averaged across all lessons. Training outcomes. The learning of trainees was assessed with a eight-item multiple-choice quiz in each lesson. Opportunities to seek information and practice were presented before the quiz was taken. Opportunities to seek feedback were presented after the quiz was taken, as the feedback was based on quiz performance. One point was scored for each correct answer on the lesson quiz. The learning scores were summed across all lessons, with a scale reliability of 0t=.62. Thus, learning was represented by a scale of zero (0) to forty (40), with forty representing the highest level of learning. The efliciency of the trainees in completing the training program reflected both the level of their learning and the time it took to complete the training. Thus, the trainee’s learning score (0-8) for each lesson was divided by the total time (in seconds) it took the trainee to complete that lesson. The efficiency values were then averaged across all lessons. The final self-efficacy of trainees was assessed with two paper-and-pencil measures taken after the participants submitted the address for their web page. One measure was a 6—item scale adapted from Toney and Kozlowski (1999), with an internal consistency reliability of or = .86. An example of items on this scale is, “I am certain I can manage the requirements of web page design, even when problems occur.” Response options to the items were on a five—point Likert scale ranging from “strongly agree” (5) to “strongly disagree” (1). This scale is presented in Appendix C. A second measure was a more traditional, Bandura scale that asked participants to estimate their confidence in 58 their ability to complete 18 different tasks, such as creating a table or inserting a graphic. Each rating was made on a 0 to 100 percent scale, and were then averaged to create the second self-efficacy measure ((12.86). One task rating was dropped because the computer program used in the training did not operate correctly, thus not allowing trainees the opportunity to attempt this task. These scales and ratings are presented in Appendix C. The performance of trainees was established by a "checklist" rating of the final web page for which they provided an address at the completion of training. These 20 performance ratings closely mirrored the tasks of the final self-efficacy ratings. Trainees were scored one point if they had completed the task, as evident in their web page. The measure created from these ratings was the average of all tasks completed (0:261). One task rating was dropped because the computer program used in the training did not operate correctly, thus not allowing trainees the opportunity to attempt this task. Trainees did not receive feedback regarding their performance score unless they contacted the experimenter and specifically requested it (none did). These ratings are presented in Appendix C. 59 RESULTS Means, standard deviations, and correlations for the measures are presented in Table 1. Two prior experience measures (one measuring experience with web programs and one measuring experience with web page design) were positively and significantly correlated, r=.20, p=.01. Prior experience with web programs demonstrated stronger correlations with other variables in the study, therefore it was used in the analyses calling for an experience variable. Likewise, the two final self-efficacy measures were positively and significantly correlated, r=.44, p<.01. The self-efficacy measure constructed of confidence ratings demonstrated much stronger correlations with other variables, therefore it was used in the analyses of training outcomes. Overall, the individual difference variables had little impact on the breadth and depth of seeking behaviors. However, the seeking behaviors had strong effects on all of the outcomes of training. Finally, the metacognitive manipulations did not demonstrate the expected interactions with each other and with metacognitive skills, nor did they produce any significant main effects. Detailed results of the analyses are presented in the following sections. Effects on Breadth and Depth of Seeking Behaviors Breadth of seeking behaviors. Mastery orientation and prior experience were expected to impact the breadth of seeking behaviors in which trainees engage. However, these individual difference variables did not demonstrate the expected relationship, R2=.00. Neither mastery orientation nor prior experience explained the variance in 60 Goscccoo 63m: 2:. one. .38.. 3.0mm. o3. *Nfim. who. mmor ever 30.- 5. mm. b:E< o>§cmou 8310mm. moo. iwwm. "INS. imam. mvo. moor .wo. mmor mm. 93 3853—5 2: iwmmr 3.0mm. NS. 80. omo. over 08.. $0. mm. Sum :36 u>EcwooSoE A33 mmo. 30.. as. mvo. So. :o.- 36.. mm. Rum .Eoto uocmgotom G3 30mm. *02. $2. $0.. 53.. coo. we. Se .325 bucmflz 83 imam. mmor 20.- Sou go. no. mmm 6050835“me -- «no: So. 08. v8... 3.3 mm: 05:53 .5me at? .2636:— : t -- : -- sous—smEME oZ -- -- -- -- 002 can 202 -- -- : 35:8 02 -- -- wccozcoE OE macaw—2:52 c5 95:: DUE m-m 522 Eats; .32 53m @8me oz e202 8: 202 am :82 .mo_na:a> 3:532 cot. 223280 was .m:o_§>oQ Emecfim .382 ._ 635. 61 000050.80 030d 62 000.- 0. .. 000.- 000. 000. 000. 0.0. 000. 000. 00..- 00.. 00.0 V3.3.030 8:00.00 000.- 000.- 00.. 000. 000. 000. 000. 000. 000.- 000.- 00.. 00.0 000008.. 03.00.30. . .0. 000. 000. .00. 000. 0.0. :0.- 000.- 000.- 000. 00.. 0. .0 V300.0000 900.6000. 000.- 000. .00. 000. 000. :0. 0.0.- .00. .00. 000.- 00.0 00.0 000008.. 005002". 000. 000.- 000.- 00..- ..0. 000. 000. 000.- 000. 000.- 00.0 00.0 808.0 005002". 000. 000. 0...- 000.- *.0.. 000. 000. 000. 000. 00..- 00.0 00.0. 005088020 000. 000.- 000.- 00..- 000. .00. 000.- 000.- 000. .00.- 00. 00.. 8.60:. 00.058 .0.. 000. 00..- 000.- $0.0. 000. 000. .00. 000. 00..- .0. 00.0 005020500 000.- 000.- 000. 000. 000.- 000. 000. 0.0. 000.- 0.0.- 00. 00.0 00.0.000 508.0. 0223.5.» 9.23m 00.. 000. 00..- 000. .....0.0. .00.. 00.. 000.- 000. 00..- 00.0 00.0 85:80.00 00..- 000. 00..- .0.. 000. .00. 000. 000.- 000.- 000.- 0..0 00.0 00.58.. .00. 000. 000. 000.- 0.0.- 000.- 000.- 000.- 000. 000. 00. 00. 02.0.0000 ..000. ..0.- 000.- 000. 1000. .2000. 000.- 0.0. 00.. 000.- 00.0. 00.00 0000.00.08.05”. ..000. 000. 000.- £000. .2000. 3.0.0. 000. 000. 000.- 0.0.- .0. 00.0 280000.00205 00.0—03:0 ”SEEP c5 3:00: DUE m0 0.0.). Eaton. .00.). .0000. 80.2.00. 02 00.52 8.). $6.). 0.0 000.2 .0280 . 0.000 ..0. v 0...... .00. v 0... 00.5006 05 no 00005020.. E 0000.000 000 00.....00..00 .0.-.30. 02.80.0000 .0500... .082 1.00. 1.000. 3.000. 1.0.0.. 0.00.. 3.00. 0*0om. .2000. 0.0.... 3.0.6 om..- 08. 03.. 0o. 0.0.0000 00:00.20 -- 3.000.100. 00.. 00o. om.- 00o. 1.00.. 8.. 0.00.. L0..- 08.. 00o.- 0o.. 0.000000>.00=.0>m -- 5.000. to. 0mo.- 0oo. 08.1.0.0. 00o. 0.00.. 00o. omor mmo. omor 0.0.0000 0>.00...000Q -- 0.00.. 00o. 00.. 08.1.00. 00.. ..c..0om. 00o.- ..o.- o0o.- 00o. 0.0.0000 00:05.02". -0.-.000. ..000. 3.0.0. 3.0.0. ._.._.000. 000.- ..00..- 1000. 00..- .00.- 8.82.. 0000002”. -1000. 3.000. moor I000. moor *on..- 0.1.0. 00o.- 8.. 600.30.00.02... -- .2000. 1.30. 2.000. 08.. 0.00..- 1.000. o0..- Co. 00.80:. 0.0.0.800 : 2o. 3.000. mo..- 0.00..- .1000. Row 5.. 80.00.0800 .- .oo. 3000. 300-00.- oo..- .0....- 00o. 50005 0030...? 050.000 20.. 0mo. 00o.- 3.000. oho. *0om. 00008000000. .00.. 00..- 00..- 0.0. 3.000. 00.503 -- .0.. .00.. 1.0..- 080.0000. 00.0.00. 0... 00.00.000.00 .00.”. .00.. 0.00.. 0.000 0000.00 .00.". 00:50.05 0550:. .0>0 0.00000 0.00000 0000000. 8... 00:00.0. 0...: 0.00 0.000 .0000. .0000. .0000. .000”. .000”. 0.500 4.800 50005 E80000. 500-. 0.00m. m0 m0 00 000 0.0.50. . 0.0.0,.- 63 breadth of seeking. Thus, Hypothesis 1 was not supported. This result could possibly be due to the low range (1 to 3) and standard deviation (.48) of the breadth of seeking variable. Depth of feedback seeking. None of the expected relationships were found for content of feedback seeking, resulting in a lack of support for all three parts of Hypothesis 2. Trainees with a high mastery orientation and trainees with less prior experience did not attempt to improve their knowledge by seeking more descriptive feedback than others (R2=.01). Trainees with a high performance orientation did not attempt to ascertain their superiority by seeking more evaluative feedback than others (R2=.02). Trainees with a high mastery orientation, low self-efficacy, or less prior experience did not attempt to increase their learning or bolster their confidence by seeking more guidance feedback (R2=.01). Finally, high mastery orientations, low prior experience, low self—efficacy, and high metacognitive skills did not spur trainees on to engage in more total feedback seeking than others (R2=.01), failing to support Hypothesis 3. Although the effects of the individual differences were too small to attain significance, all except prior experience were in the expected direction. Trainees who received prompts to engage in metacognitive monitoring were expected to engage in greater frequency of feedback seeking, producing an interaction between metacognitive skills and metacognitive monitoring. Neither a main effect for metacognitive monitoring (R2=.00) nor an interaction (R2201) were uncovered. Hypothesis 4 was not supported. Depth of information seeking One individual difference was found to explain the variance in complexity of information seeking (R2=.06, p<.05), in partial support of Hypothesis 5. Trainees with more prior experience (13:.20, t=2.39, p<.05) were found to seek out more complex information, a result expected due to their foundation of basic knowledge (Table 2). High mastery orientations and greater self-efficacy did not, as they were expected to, similarly explain variance in complexity of information seeking. Despite the promising result of the previous hypothesis, Hypothesis 6 did not receive support. Trainees with high mastery orientations, low performance orientation, low prior experience, low self-efficacy, or high metacognitive skills did not engage in more total information seeking than others, as expected (R2=.05). Trainees who received prompts to engage in metacognitive control were expected to engage in greater frequency of feedback seeking, producing an interaction between metacognitive skills and metacognitive control. Neither a main effect for metacognitive control (R2=.01) nor an interaction (R2203) were uncovered. Hypothesis 7 was not supported. Table 2. Regression of Individual Difference Variables on Depth (Complexity) of Information Seeking. Step: Variables R2 df B 1: Individual differences 058* 3, 150 Experience .204* Mastery orientation -. 140 Self-efficacy .079 *p < .05 65 Depth of practice seeking. None of the individual difference variables explained the variance in complexity of practice seeking, thus failing to support Hypothesis 8. Trainees with high mastery orientations, more prior experience, and/or greater self- efficacy were expected, but not found, to engage in practice seeking at greater levels of complexity (R2=.O4). In fact, this regression was close to significance, but singularly due to the effect of low mastery orientation. Opposite of expectations, trainees with low mastery orientation tended to seek more complex practice exercises. Once again, high mastery orientation, low performance orientation, less prior experience, high self—efficacy and/or low metacognitive skills did not explain the frequency of practice seeking (R2=.04). Therefore, Hypothesis 9 also received no support. Trainees who received prompts to engage in metacognitive control were expected to engage in greater frequency of feedback seeking, producing an interaction between metacognitive skills and metacognitive control. Neither a main effect for metacognitive control (R2=.Ol) nor an interaction (R2=.03) were uncovered. Hypothesis 10 was not supported. Effects on Training Outcomes Learning. As expected, the seeking behaviors in which trainees engaged influenced their learning during training (Rzz. 13, p<.01). Trainees who sought a breadth of information, practice, and feedback were more actively engaged in learning and thus learned more ([3:34, t=4.47, p<.01; Table 3). However, the depth of information seeking (complexity and frequency) did not contribute to explaining the variance in learning. Thus, Hypothesis 11 received partial support. 66 Table 3. Regression of Seeking Behaviors on Learning. Step: Variables R2 (If [3 l: Seeking variables .126** 3, 151 Breadth of seeking .341 ** Complexity of information seeking -. 127 Frequency of information seeking .019 *p < .05; **p < .01 Trainees who are prompted to both monitor and control their learning process were expected to result in the best learning of all trainees. However, no main effects or interaction were found for the effect of metacognition on learning, failing to support Hypothesis 12. Efficiency. The frequency with which trainees engaged in seeking behaviors was expected to have a negative relationship to the efficiency outcome—the more time spent seeking, the less efficient the learning process. Although this negative relationship was found for frequency of feedback, information, and practice seeking, it was not quite strong enough to attain significance (R2=.04; Table 4). The pattern of relationships were in the expected direction, but not quite significant, failing to support Hypothesis 13. Final Self-Efficacg After controlling for initial self-efficacy, which was expected to and did predict final self-efficacy (R2=.11, [3:34, t=4.38, p<.01), one seeking behavior was found to also predict final self-efficacy (R2=.32, p<.01). In a second step in the regression, complexity of information seeking (132.51, t=2.45, p<.05) had a positive effect on final self-efficacy, providing partial support for Hypothesis 14 (Table 5). Other 67 Table 4. Regression of Seeking Behaviors on Efficiency. Step: Variables R2 df B l: Seeking variables .044t 3, 151 Frequency of feedback seeking -. 100 Frequency of information seeking -.1 16 Frequency of practice seeking -.06O lp = .08 Table 5. Regression of Seeking Behaviors on Final Self-Efficacy Ratings. Step: Variables R2 df ARZ Adf 13“ l: Covariate .112** 1, 152 -- -- Self-efficacy .335** 2: Seeking variables .316** 6, 147 .204** 5, 5 Breadth of seeking -.095 Complexity of information seeking .508* Frequency of information seeking -.096 Complexity of practice seeking .176 Frequency of practice seeking -. 146 3‘The BS refer to standardized regression coefficients associated with each step of the hierarchical regression. *p < .05; **p < .01 68 seeking behaviors that did not help explain the variance in final self-efficacy as expected were breadth of seeking, frequency of information and practice seeking, and complexity of practice seeking. Performance. A number of seeking behaviors were expected to explain the variance in trainee performance. A strong relationship was indeed found between seeking behaviors and performance (R2=.36, p<.01; Table 6). The majority of this relationship was carried by breadth of seeking (B=-.42, t=-3.93, p<.01) and complexity of . .-_....._...' practice seeking (13:.52, t=2.38, p<.05). The negative effect of seeking breadth is surprising, considering the strong positive relationship of seeking breadth to learningz. Practice was the seeking behavior most closely related to performance, so the positive { effect of seeking complexity on performance was to be expected. Feedback seeking content and frequency, as frequency of practice seeking did not help to explain the variance in performance, so Hypothesis 15 was only partially supported. Trainees who are prompted to both monitor and control their learning process were expected to result in the best performance of all trainees. However, no main effects or interaction were found for the effect of metacognition on performance. A main effect for metacognitive monitoring was close to significance, but in the opposite direction of what was expected. Trainees who were not prompted to monitor their learning process performed slightly better than those who were prompted to monitor. Hypothesis 16 was not supported. 2 Measures of learning and performance were found to be uncorrelated in this study (see Table l). 69 Table 6. Regression of Seeking Behaviors on Training Performance. Step: Variables R2 df [3 l: Seeking variables .360** 7, 147 Breadth of seeking -.422** Frequency of descriptive feedback seeking .088 Frequency of evaluative feedback seeking .043 Frequency of guidance feedback seeking -.01 1 F Frequency of feedback seeking .224 i Complexity of practice seeking .517* Frequency of practice seeking .164 L *p < .05; **p < .01 Summary of Results Figure 3 is a revised version of Figure 1, showing the changes in the expected relationships among the independent variables, the seeking variables, and the outcome variables. Overall, the individual difference variables examined in this study had virtually no effect on the information, practice, and feedback seeking behaviors. Only prior experience was found to have a positive relationship with complexity of information seeking. The manipulated prompts for metacognitive monitoring and metacognitive control did not interact with metacognitive skills to affect the seeking behaviors (nor did they directly affect seeking). The metacognitive manipulations also did not directly or interactively predict the training outcomes. The shortcomings of these measured and manipulated variables are discussed. 70 Breadth of Seeking Behaviors Prior Experience Training Outcomes Depth of Seeking Behaviors Figure 3. Observed pattern of relationships among individual differences, seeking behaviors, and training outcomes. The bright spot in the results was the strong effect that some of the seeking L7 behaviors had on the training outcomes. Breadth of seeking positively predicted learning, and frequency of seeking negatively predicted efficiency. Complexity of information seeking bolstered self-efficacy, and complexity of practice seeking improved performance. Figure 4 presents a more detailed map of the significant relationships found in this study. These results lend support to the typology of feedback seeking presented here, and are discussed in the remainder of the paper. 71 Breadth of . , Learning Seeking 0 Frequency of Seeking Complexity of S elf- ‘ Information ‘ W Efficacy Complexity of Practice Seeking Efficiency O Prior Experience Performance O Figure 4. Observed significant (and near significant) effects of individual differences on seeking behaviors, and of seeking behaviors on training outcomes. 72 DISCUSSION One of the goals of training research over the past decade has been to increase the active participation of learners in the learning process. Researchers have proposed and confirmed that learners who are more cognitively engaged, or more “mindful,” are more likely to increase their learning. Such learners are also more likely to better transfer their learning to task performance. Two ways of promoting cognitive, active learning include the use of learner control designs in training programs (Noe & Ford, 1992) and fostering the metacognitive process of learners (Smith, Ford, & Kozlowski, 1997). The present study examined the learner control paradigm from a non-comparative :0 standpoint; it was not compared to a program control or adaptive control design, to determine whether learner control was a better or worse method for learning. Instead, the behaviors in which learners engaged when given control of their learning environment were examined. The behaviors studied here included the feedback seeking, information seeking, and practice seeking of trainees. These three types of seeking behaviors were examined to determine whether they were influenced by individual differences in trainees, and whether the seeking behaviors subsequently led to differences in the training outcomes of learning, efficiency, self-efficacy, and performance. This study also prompted learners to engage in metacognition during the learning process. Some trainees were prompted to monitor their learning process by writing down what they knew and didn't know about the previous training lesson, and about how that lesson related to other training lessons. Some trainees were prompted to control their learning process by writing down what strategy they were using to learn, whether or not it 73 was working, and what steps they planned to take in the next training lesson to improve their learning. The metacognitive prompts were expected to improve the learning process for trainees who did not possess strong metacognitive skills. Additionally, trainees who received both monitoring and controlling prompts were expected to demonstrate the best training outcomes. The following sections examine the results of this study, first for learner control and seeking behaviors, and second for prompting of metacognitive monitoring and control. Limitations are addressed, particularly for those areas where expected effects were not found. Some post-hoc analyses are presented; these were conducted to explore reasons for the lack of some of the expected results. Contributions are discussed, and future directions for these areas of training research are proposed. Individual Differences and Seeking Behaviors The results of this study evidenced very little effect of the individual difference variables on the seeking behaviors. To investigate this lack of effect, first the individual difference variables themselves were examined. Relationships among these variables were as one would expect. Mastery orientation, self-efficacy, prior experience, and metacognitive skills were all positively related to one another. Performance orientation was uncorrelated with mastery orientation, and was negatively correlated with metacognitive skills. Thus, the variables were not “behaving” in a manner out of the ordinary. For a couple of the individual difference variables, and for one seeking behavior, their psychometric properties may have contributed somewhat to the overall lack of relationships (see Table 1). First, the metacognitive skill measure demonstrated low 74 reliability (0t=.70). Although this level of reliability may be considered acceptable according to traditional standards, it is considerably lower than the reliability of the other measured individual differences. Second, there was low variance in the mastery orientation measure, and it also had a rather high mean when compared to the other individual difference variables in this study. Third, the breadth of seeking measure demonstrated a lower variance than the other measures of seeking. Theoretically, it had the same 0-to-3 range as the various measures of depth of seeking. Practically it only had a 1-to-3 range, because all trainees engaged in at least one of the three types of seeking behavior (usually information seeking was always engaged). The limited range of breadth of seeking naturally reduced L the variance in this measure. (This lack of variance may also help explain why the significant effects this variable had on two different training outcomes were in opposite directions.) Fourth, there was high variance in the measurement of prior experience with the training content—creating a web page—but it was due to a strong positive skew. The median and modal number of hours were both zero, and after removal of outliers the variance of this measure would then be very low. When coded as a dichotomous variable (1 = web design experience, 0 = no experience) and entered into the regression equation, it produced no differences in the results. Thus, it was not presented in the analyses. If this measure of content experience had better psychometric properties and was suitable for use in analyses, prior experience may have had more effect on the seeking variables. In training programs where there may be more variance in the content experience of trainees, prior content experience may play a greater role. 75 Another reason for the lack of effects could be due to a variable that accounts for most of the variance, and needs to be used as a covariate in analyses. Cognitive ability is a factor that is frequently strong in learning environments, and can overshadow effects of other variables. Therefore, the measure of cognitive ability was entered first into the regression equations as a covariate. However, seeking behaviors were not significantly influenced by cognitive ability, so using it as a covariate produced no meaningful differences in results. Finally, alternate forms of the seeking variables were used in some analyses in further attempt to find a relationship with individual difference variables. The hypotheses that proposed an interaction between metacognitive skills and the metacognitive manipulations (4, 7, and 10) called for an effect on the depth of seeking behavior. In the analyses, depth of seeking was represented by the complexity of seeking. However, the frequency of seeking could also represent depth, and frequency of seeking evidenced slightly more variance than complexity of seeking. Thus, these analyses were re-run using frequency as the dependent variable. Unfortunately, a similar lack of results was produced. The likely reasons for the lack of effect of independent variables on seeking behaviors can be framed in terms of two issues: (1) problems with the measured individual differences in this sample, and (2) the presence of a strong situation that decreased the effects of the individual differences. College student samples may demonstrate less variance than samples from the general working populations on measures of cognitive ability and mastery orientation. Students who gain admittance to college are likely to have consistently higher levels of general cognitive ability, and those 76 who choose to attend may have a stronger orientation toward learning. This particular study was also of high interest to most who chose to participate. Students were curious and eager to learn how to create a web page, and likely perceived this skill to be very useful. Participation in the study was voluntary, so students with extensive experience in web page design or low interest in web page design would not have been likely to participate in this study. For the above reasons, the variance was effectively reduced in cognitive ability, mastery orientation, prior experience, and any variables related to motivation to learn. Another individual difference artifact that is always of concern in human-subjects research is the “good subject effect.” The design of (and instructions for) the training left some subjects confused, as they thought they were required to go through all parts of all lessons. Thus, those participants who desired to be a “good subject” for the research would have engaged in seeking behaviors at all levels of complexity, regardless of their natural inclination to do so outside of a research study. Therefore, this individual difference may have clouded the relationship of other individual differences to seeking behaviors. Due to these problems with some of the measured variables, a principal components, rotated factor analysis was run to ensure that these measures represented distinct constructs. When seven factors were forced, the measures loaded cleanly onto separate factors. However, the Scree plot suggested that there were only five factors. When these five factors were examined, the experience with training content item loaded onto the same factor as the self-efficacy items and the mastery and metacognitive skills items all loaded onto the same factor. This factor is consistent with the high correlation 77 that is evident between the mastery and metacognitive skill scales. This combined “meta- mastery” scale maintains the reliability level of the mastery scale ((1:84), and actually improves upon the reliability of the metacognitive skill scale. The learner control design used in the study may have additionally reduced the effects of the individual difference variables on seeking behaviors. Learner control training may produce a strong situation, which limits the operation of personality variables and other individual differences (Weiss & Adler, 1984). Allowing—even requiring—learners to direct their own learning process may evoke a mastery frame: a situation that promotes an orientation toward learning over and above an individual’s natural orientation toward learning. In the present study, although trainees were informed that they had to submit their web page for review, it was otherwise framed as a strong learning situation. The diagnostic nature of the quizzes and feedback, the lack of a performance-based reward, and even the metacognitive manipulations all likely helped to establish this learning frame. When combined with the high motivation to learn demonstrated3 by most participants in this particular study, the learning situation may have been very strong indeed, and may have reduced the effects of any individual differences on seeking behaviors. The implication for practice is that individual differences may be not important to the success of learners in strong learning situations (e. g., ordinary job training), whereas individual differences may play a greater role in situations where training is used as a tool for advancement (e. g., pay-for-performance training). 3 As observed by the experimenter. 78 Contribution of the Seeking Typology Prior research has investigated some aspects of seeking behaviors, particularly feedback seeking. However, this study brought together three types of seeking behaviors—feedback, information, and practice seeking—and investigated them together as a coherent typology of behaviors in which learners engage when given control of their learning environment. The correlations among some of the measures of the seeking behaviors were high, so a principal components, rotated factor analysis was conducted to ensure that the proposed three types of seeking behaviors were, in fact, distinct constructs. The analysis replicated the proposed typology of feedback, information, and practice seeking. Breadth of seeking loaded highly on multiple factors, as would be expected since it is a multi-dimensional construct representing all three types of seeking. > The resulting impact on training outcomes of learning, self—efficacy, and performance suggests that this typology of seeking behaviors can make a contribution to research in the area of learner control designs for training. First, trainees who engaged in a greater breadth of seeking behaviors increased their learning compared to others. This effect is consistent with what was expected, as these trainees were taking advantage of the full range of information, practice, and feedback available to enhance their knowledge and skills. Unexpectedly, however, greater breadth of seeking behavior led to decreases in performance. Part of the problem may have been that the performance measure incorporated a motivational element as well as a knowledge/skill element. Trainees might not include a particular feature on their web page because they did not learn how to create the feature, or because they did not wish to have that particular feature on their web page for design 79 or aesthetic reasons (e. g. “scrolling text”). Another culprit could be a time-on-task tradeoff. The trainees who were engaging in a greater breadth of seeking behaviors were spending more time in the training program, and spending less time working on their web page—resulting in less time spent completing the performance measure. Theoretically there was no such limit on the time that could be spent on the performance measure, as trainees were instructed that they could work on their web page and receive instructor assistance for two weeks after the experiment. Practically, though, only seven trainees F " returned to work on their web page following the training session, making the time-on- task tradeoff a practical reality. Second, trainees who engaged in greater depth of seeking behaviors improved “1 their training outcomes. Those who sought practice at more complex levels increased their training performance. This result was expected, because the practice of the skills learned during the lessons allowed for a more smooth transition to the performance of those tasks at the completion of training. Those who sought information at more complex levels increased their self-efficacy at the end of training. This result was also expected, as the knowledge gained from the information learned during the lessons contributed to a stronger sense of capability to perform the learned tasks. Complexity of information seeking did not, however, influence the learning of trainees. The design of the learning measure itself is undoubtedly the major contributor to this lack of effect. The learning measure—the end-of-lesson quizzes—only measured knowledge at the basic level. The reason for this decision was to enhance the “optional” nature of the three levels of complexity within the lessons. Unfortunately, in hindsight, 80 this decision may have also reduced the effect that information seeking could have on the learning variable in this study. Although complexity of practice seeking did lead to improved trainee performance, feedback seeking did not contribute to improving performance as expected. This could be related to the theoretical breakdown of feedback examined here. Perhaps the typology of descriptive, evaluative, and guidance feedback was not useful in this design. Chi (1996) proposed another typology of feedback relevant to tutoring situations, that may be more useful in learner control designs. Corrective feedback provides right/wrong information to the trainee, and is equivalent to descriptive feedback. Didactive explanations provide long narratives that reveal and support the correct answer. Suggestive feedback indirectly alerts the trainee to the problem without revealing the exact nature of the problem or the correct answer. This is similar to the notion of guidance feedback, except that it more actually represents feedback, rather than the “feedforward” of this study’s conceptualization. Chi considers suggestive feedback to be a form of scaffolding, which works in concert with prompting to enhance learning and transfer. Another reason for the lack of effect for feedback seeking on performance could be that it interacts with individual difference variables to affect performance. As one can see in Figure 4, the pattern of results involves only main effects from the seeking behaviors to the training outcomes. Since the individual differences did not directly affect the seeking behaviors, producing a mediation effect on the outcomes, perhaps they interact with the seeking behavior in producing a moderated effect on the outcomes. In this instance, it was found that both evaluative feedback and guidance feedback 81 interacted with mastery orientation (as measured by the combined “meta-mastery” scale of mastery orientation and metacognitive skill) to affect performance. Seeking guidance feedback had a stronger positive effect on performance for trainees with higher meta- mastery orientation (13:1.39, t=2.00, p<.05). Conversely, seeking evaluative feedback had a marginally stronger effect on performance for trainees with lower meta-mastery orientation (B=-2.04, t=-1.90, p=.06). Another measure of feedback seeking, frequency of seeking descriptive feedback, interacted with meta-mastery orientation to affect final self-efficacy. Seeking descriptive feedback lead to higher final self-efficacy for trainees with high meta-mastery orientation relative to trainees with low meta-mastery orientation (0:2.21, t=2.06, p<.05). In similar analyses, cognitive ability was also found to interact with seeking behaviors, moderating the relationship to training outcomes. Trainees with higher cognitive ability were better able to maximize the positive effect of seeking complex information on their performance (13:1.66, t=2.05, p<.05). Trainees with lower cognitive ability were actually able to increase their learning efficiency by seeking more descriptive feedback, relative to trainees with higher cognitive ability (02-1-32, t=2.10, p<.05). Finally, although there was no interaction found, these additional analyses revealed positive main effects for descriptive feedback seeking (02.51, t=1.94, p<.05) and guidance feedback seeking ([3:56, t=2.18, p<.05) on learning. Thus, when more complex relationships were examined among individual differences, seeking behaviors, and training outcomes, more significant results were generated. However, it must be noted that these analyses are post hoc and may capitalize on chance; they should be used to guide future hypotheses of research in this area. 82 The individual difference variables examined in this study did not provide much of a contribution to understanding the engagement of seeking behaviors by trainees in a learner control environment. This lack of effect has been discussed, and the most apparent reasons for it are the low variance of the individual difference variables and the evocation of a strong learning situation. Although there was only one effect for individual difference variables on seeking behaviors, there are potential implications of that effect that should be discussed. Prior experience was found to be a predictor of the complexity of information seeking in which trainees engaged. Initially this effect might seem disturbing—that prior experience would be an important factor in the learning process, and that those who have more experience would benefit most from training. However, in this study, the measure of experience that was used in analyses was the experience that trainees had with the training method, not the training content. Thus, trainees who had more experience with web-based activities (entertainment, classes, shopping) were more likely to engage in more complex seeking in the web-based training environment. Additionally, the progression of this effect through the learning process did not result directly in increased learning or performance. The effect chain for trainees with more experience was that they sought more complex information, which then led to increased self-efficacy at the completion of training. Thus, trainees who are more familiar with web-based activities are likely to feel more confident following training in their ability to perform the tasks learned in a web-based training course. Future Directions for Learner Control and the Seeking Typology Sufficient evidence was gathered in this study to merit the further examination of the typology of feedback, information, and practice seeking behaviors presented here. 83 Clearly more work needs to be done to identify individual differences that affect or interact with seeking, as well as situational antecedents of seeking behaviors. Cognitive skills such as written comprehension and memorization may play a role (Fleishman & Mumford, 1989), as may personality variables such as conscientiousness (Barrick & Mount, 1995; Colquitt & Simmering, 1998; Gellatly, 1996), openness to experience (Barrick & Mount, 1995), and cognitive playfulness (Martocchio & Webster, 1995). Age may be an important demographic factor that affects seeking behaviors in WBT programs (Warr & Bunce, 1995), but was not examined here due to the likely low variance of age in a college student sample. The individual differences that were examined here were mostly situational (prior experience) and motivational (self-efficacy, goal orientation) in nature; other motivational and situational individual difference variables may have a stronger impact. For example, individuals may encounter different situations prior to training that may lead them to engage in different patterns of seeking behavior, such as receiving different levels of information or support from their organization with respect to training, or whether their participation is voluntary or mandatory (Baldwin & Magjuka, 1997). Individuals may also experience different motivations when participating in training, including different attitudes toward training in general (Warr & Bunce, 1995) and different levels of “motivation to learn” (Colquitt & Simmering, 1998; Noe, 1986; Quinones, 1995). Subcomponents of motivation to learn which may affect seeking behaviors include curiosity, perceptions of relevance/usefulness, perceptions of difficulty, valence/instrumentality/expectancy, and completion goals (Brown, 1999; Keller, 1983; Mathieu & Martineau, 1997; Simmering, 1999; Warr & Bunce, 1995). 84 However, as discussed previously, the lack of effects may have been due to a strong learning situation, rather than the selection of individual differences that are unrelated to seeking behaviors. Thus, another productive line of research could examine the differences between learner controlled training programs that are framed as strong learning situations, strong performance situations, or not framed (weak situations). Presumably the weak situation would allow for more effects of individual differences. Results may demonstrate that, depending on the goals of training, effects of individual differences can be either enhanced or minimized. Another area of exploration can involve the relationship seeking behaviors have with other variables and outcomes throughout the learning process. Such analyses would shed more light on how seeking behaviors operate as they influence learning, self- efficacy, and training performance. For example, an aspect of learner control that has not seen as much research is the sequence choice of learners who are allowed to select the sequence of content to learn during the training. In the present study, a descriptive analysis of the data4 showed that the participants, who were allowed to select the sequence in which the lessons were presented, did choose different lesson sequences. About half chose the same lesson sequence, while the other half chose a variety of other lesson sequences. However, all trainees started with either the lesson on “Designing your page” or “Backgrounds, text, and tables,” and virtually all finished with “Putting it on the web.” The trainees may have tried to use strategies to determine the best lesson sequence (e. g., cues in the lesson titles). Although in this study there did not seem to be differences in training outcomes based upon the lesson sequence chosen, future studies 4 A random sample of 10% of participants was examined in the descriptive analysis. 85 that allow trainees control over content sequence may wish to examine the strategies that trainees use to select a sequence and what effect they may have on training outcomes. Related to content sequence is the sequence of seeking behaviors. Some trainees sought information, practice, and feedback at one level of complexity before progressing to the next level of complexity. Some trainees sought all levels of information before seeking practice or feedback. Some trainees sought information first, some sought practice first, and some began with the quiz and feedback seeking. Different patterns of seeking behavior could be ordered along a continuum, anchored by liner versus non- linear cognitive search behaviors. There may be cognitive individual differences that lead to the selection of more linear or non-linear sequences. In addition to the examination of individual differences that lead to certain search behaviors, the effect of the different search sequences on training outcomes should also be explored. Other process analyses may also be valuable to supplementing the results found in the outcome analyses of this study. For example, certain changes in seeking behaviors may provide evidence of strategy changes on the part of the trainees. Feedback seeking early in training may affect subsequent seeking behaviors. Trainees who identify that their learning or performance is not as high as they would like it to be in early lessons may begin engaging in more information or practice seeking during later lessons; or, they may wish to avoid negative feedback and withdraw from feedback seeking during the remainder of training. Trainees who identify that their learning is on track may maintain their information and practice seeking, or they may decrease their feedback seeking. Other patterns of decreasing or increasing seeking behaviors may also become apparent, including changes due to fatigue effects or boredom. 86 Finally, there are some practical implications of the results found in this study. First, inclusion of seeking opportunities in a learner control training environment contributed negatively to the efficiency of training. Although the effect did not reach significance, the result is worth discussing. Since most training programs strive to meet both efficiency and effectiveness goals, it is important to examine ways in which this lack of efficiency could be reduced or eliminated. One method might be to discuss with trainees prior to training how to use the program efficiently. Another method might be to determine what individual differences contribute most to this inefficiency. For example, trainees with more content experience might be able to use the learner control training program more efficiently compared to less experienced trainees. Perhaps those more experienced trainees could be routed into a learner controlled training session, whereas the less experienced trainees could be routed into a program controlled training session. The practical implications for the lack of relationship between seeking behaviors and learning are more difficult to interpret. As discussed previously, in this study there was no measure of learning for the more complex levels of the lessons. This acted, in effect, as a restriction of range on the criterion side and decreased the relationship between the variables. However, presuming no restriction of range and still no relationship, one would have to conclude that something other than seeking behaviors is responsible for the variance in the learning outcome. There could be a direct effect of an individual difference variable on learning that is not mediated by seeking behaviors. In this study, that did not seem to be the case; however, there were a number of individual differences that were not measured in this study, such as motivation to learn and conscientiousness. 87 If seeking information more frequently and at more complex levels does not enhance learning, then it appears that merely providing trainees with opportunities to seek information is not enough. One practical implication that is supported by this research is that trainees need to be provided with opportunities to practice the learned skills and to receive feedback on their learning. The positive relationship found between breadth of seeking and learning bolsters this notion. Another implication is that some trainees are better integrating and finding more meaningful the information that they are seeking. However, as discussed in the next section, this notion did not receive support in the present study. Metacognitive prompts to help trainees engage in more active learning were not successful. Metacognitive Manipulations In this study, the prompting of metacognitive activity during training did not yield improvements in training outcomes as expected. Neither the manipulation of metacognitive monitoring nor of metacognitive control produced differences in learning, self-efficacy, or performance from that of trainees who did not receive prompts. These prompts were also expected to interact with individual differences in metacognitive skill, such that trainees with less skill in using metacognition would engage in training behaviors in a manner similar to trainees with greater skill. This interaction between trait and state metacognition was not observed. Due to the overall lack of results for the metacognitive manipulations, some additional analyses were conducted to see if there were effects on variables other than the basic training outcomes. First, it was proposed that the trainees who were prompted to engage in metacognition would have been more likely to return to a previous lesson and 88 re-study. One of the functions of metacognition is to focus the learner (i.e., monitoring) on areas of deficiency so that the deficiency can be remedied (i.e., controlling, Nelson, 1996; see also Bandura, 1991; Carver & Scheier, 1982). However, no differences due to metacognitive manipulations were observed for the amount of time spent re-studyin g a previous lesson; also each lesson was re-studied by less than ten percent of the participants. !‘ Another proposition was that trainees prompted to engage in metacognition would ' spend more time with the lessons, monitoring and controlling their learning process l throughout training. Instead, the opposite effect was observed: those who were not F“. "a ot-‘nu ‘. prompted to engage in metacognition spent more time on the lessons than did those who were prompted. At the same time, all groups spent about the same amount of time on the entire training program. (Trainees were not constrained to spend the same amount of time on the training program. An upper limit of three hours was imposed, but the vast majority of participants finished well within this amount of time.) Thus, a time-on-task tradeoff existed for those receiving metacognitive prompts; the time spent responding to the metacognitive prompts reduced the time that trainees spent learning the content during the lessons. Because the participants were free to choose how long to spend on the training program, this tradeoff indicates that there may have been some type of fatigue effect of the metacognitive manipulation that led trainees to spend less time on the lessons. This issue has potentially strong implications for the use of metacognitive prompts in learner controlled training environments. There are a number of elements that may have played a role in limiting the efficacy of the metacognitive manipulations, or otherwise explaining the lack of results in 89 this area. However, a couple of the “usual suspects” did not emerge as likely candidates for contributing to the lack of results. First, there was ample variance in the dependent variables—the training outcomes of learning, self-efficacy, and performance—to provide room to find differences. A number of effects of seeking behaviors on these outcome variables were, in fact, found. Second, the manipulations appeared to produce responses that are consistent with the notion that participants took them seriously and did engage in metacognitive activity. Thus, the responses can be seen as an indication that the manipulations were strong enough. Despite this indication of the manipulation’s success, the manipulations may not have been appropriate given the structure of the training. For example, the metacognitive control manipulation asked trainees to note the strategy they were using to learn the information contained in the lesson, to evaluate it, and to list the steps they would take in the next lesson to improve their learning. In such a short training course on a limited amount of information, there may not be a sufficient number of strategies available for trainees to pursue and change strategies based on metacognitive evaluation of one’s learning process. Likewise, the “integration” focus of the metacognitive monitoring manipulation may not have been particularly useful in a training program consisting of only five lessons that are fairly well related. The scope of the training program may have been one limitation to the success of the metacognitive manipulations. Perhaps the prompting of metacognitive strategies and integration would be more effective in a longer-term, more complex training program. Another significant limitation is the lack of a manipulation check measure. Although the responses to the metacognitive prompts are available, and indicate that the 90 prompting did lead to the desired metacognitive activity, there was no measure of metacognitive activity for the control group. Therefore an important question cannot be answered: Do trainees naturally engage in metacognition during the learning process, thus making the prompting of metacognition unnecessary? The analysis of initial pilot data5 for this study suggests that this supposition may be true. Trainees were asked following the training whether they engaged in either monitoring their learning process or evaluating/changing the strategies they were using to learn; they were then asked if they were prompted to engage in these behaviors. A number of trainees who were not prompted to engage in metacognition reported doing so. Although based on a small sample size (N=9), this lends credibility to the argument that metacognitive activity is naturally engaged during learning and therefore prompts may be ineffectual. The final limitation is related to the previous suggestion that metacognition is naturally engaged during learning. It may be that metacognition is naturally engaged during learner controlled learning environments. Learners who have control of the content, sequence, and/or pacing of the learning process engage in a variety of seeking behaviors, as demonstrated here. As learners engage in these seeking behaviors, they may self-generate feedback from the quizzes or practice opportunities. This self- generation of feedback may represent metacognitive monitoring, and may also function as a prompt to engage in metacognitive control of learning. In this way, seeking behaviors may not be distinct constructs from metacognitive activities such as monitoring and controlling. Instead they are all part of the larger self-regulatory process; separation of the elements of the process at this lower level does not produce meaningful results. 5 The initial pilot included 9 participants; a later pilot included 46 participants. 91 Future Directions for Metacognition in Learner Control There are many questions as yet unanswered by the literature that has empirically examined metacognition that justify its further examination, despite the failure to uncover effects in this study. First, researchers must determine when metacognition is automatic and when it needs to be prompted. For example, do individual difi‘erences in metacognitive skills play a primary role in the need for metacognitive prompting? In this study the measurement of such differences in skill was slightly unreliable, and the development of a reliable measure would substantially aid the answering of this question. Or does the situation dictate when and how metacognition should be prompted? Are prompts needed in all learning environments? Are they more or less necessary in learner controlled environments, and does the level of control that learners have make a difference? Does the complexity of the training program play a role; do training programs that require more strategy use during learning benefit more from the prompting of metacognition? Or is the prompting of metacognitive activity an interference in the learning process, producing fatigue effects or other disruptive influences? Answering any or all of these questions will significantly advance knowledge of the role of metacognition in training. Finally, the limitation discussed previously may lead to a fruitful area for future research: that seeking behaviors may function to prompt metacognition. If this is the case, it may necessitate a return to comparing learner control and program control training—with a focus on the role that seeking behaviors play independent of prompts to engage in metacognition. Should research determine that seeking behaviors are, in fact, indistinct from metacognitive activity, this may reveal the most efficacious method yet 92 for “prompting” metacognition, as this study has evidenced positive effects of seeking behaviors on training outcomes of learning, self-efficacy, and performance. Conclusion Theoretically there is a strong basis for incorporating both learner control and metacognitive prompting in the design of WBT programs. Some of the theoretical propositions were borne out in the present research study, including the positive impact of breadth and depth of seeking behaviors on outcomes of learning, self-efficacy, and training performance. However, the lack of results for the inclusion of metacognition in WBT was conspicuous. Some important changes to the design of this study may allow its theoretical merits to be supported more conclusively. The theoretical model presented here may be empirically supported by a similar experiment conducted with the following four key changes. 1. The metacognitive activity of learners should be measured as well as manipulated. Ford and colleagues (1998) developed a measure of metacognitive activity, and the inclusion of such a measure would allow for a check of the manipulation effectiveness. This measure would also shed light on whether metacognitive activity is naturally engaged during learner control training programs. 2. A contrasting experimental condition representing program control training would likewise address the question of whether learner control training induces metacognitive activity. It was discussed earlier that the learner control environment might have been strong situation that produced a mastery frame for all learners. The strong situation could have overpowered the effects of the metacognitive manipulations, as well as eliminated the effects of individual differences. Adding one or more program 93 control conditions would allow for the comparison of metacognitive activity in training programs of different situation strength with regard to learning orientations. 3. A measure of motivation, and particularly motivation to learn, would help identify whether there was low variability and high means with respect to the learning orientation of the sample. If addressed both prior to and during the training, the measure of motivation would also allow for a distinction between a sampling problem and the operation of a strong learning situation. Additionally, using different content in the training program would help isolate any idiosyncrasies in the results of this study due to high intrinsic interest of the learners. 4. Finally, a well-developed opportunity for and measure of transfer would identify whether the effects of the metacognition were simply not realized due to the immediacy of the performance measure. J acobsen and Spiro (1995) had found a similarly poor effect of metacognition on training performance. However, the benefits of the metacognitive activity were evident in the transfer performance of learners. The design of the present study did not provide an optimal means for trainees to demonstrate transfer of learned skills. These four recommended revisions to the design of this experiment are highlighted because they would allow for a re-test of the empirical support for the theoretical model proposed here. However, there are other directions relevant to this general area of research that have been discussed. Research needs to further explore the individual differences that may be antecedent to the typology of seeking behaviors. Also, patterns of seeking behaviors and changes in these patterns are areas in which research can contribute further to the development of a better understanding of the functioning of 94 learner control environments. Future research needs to continue to explore when and how to facilitate metacognition in the learning process, and to continue to develop more reliable means of measuring metacognitive skill in learners. Despite the challenges identified in this research, there are many fruitful avenues for research using WBT to investigate learner control designs. The incorporation of metacognition in learning processes has long been fraught with inconsistent results, suggesting the importance and extreme need for continued research in this area, rather than the abandonment of it. This paper has presented ideas for future directions of research that will be beneficial to areas of both metacognition and learner control. 95 REFERENCES Aberson, C. L., Berger, D. E., Emerson, E. P., & Romero, V. L. (1997). WISE: Web interface for statistics education. Behavior Research Methods, Instruments, & Computers, 29, 217-221. Alderman, M. K., Klein, R., Seeley, S. K., & Sanders, M. (1993). Metacognitive self-portraits: Preservice teachers as learners. Reading Research and Instruction, 3, 38-54. Ames, C., & Archer, J. (1988). Achievement goals in the classroom: Students’ learning strategies and motivation processes. Journal of Educational Psychology, 80, 260- l 267. I lI‘J-I—A l’ Ashford, S. J., & Cummings, L. L. (1983). Feedback as an individual resource: Personal strategies of creating information. Organizational Behavior and Human Performance, 32, 370-398. -nm Ashford, S. J ., & Cummings, L. L. (1985). Proactive feedback seeking: The JV instrumental use of the information environment. Journal of Occupational Psycholggy, 58, 67-79. Ashford, S. J., & Northcraft, G. B. (1992). Conveying more (or less) than we realize: The role of impression management in feedback seeking. Oganizational Behavior and Human Decision Proces& 53, 310-334. Avner, R. A., Moore, C., & Smith, S. (1980). Active external control: A basis for superiority of CBI. Journal of Computer-Based Instruction, 6, 115-1 18. Baldwin, T., & Magjuka, R. J. (1997). Organizational context and training effectiveness. In J. K. Ford and associates (Eds), Improving training effectiveness in work organizations (pp. 99-128). Hillsdale, NJ: Lawrence Erlbaum Associates. Bandura, A. (1991). Social cognitive theory of self-regulation. Organizational Behavior and Human Decision Processes, 50, 248-287. Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimensions and job performance: A meta analysis. Personnel Psychology, 44, 1-26. Bean, T. W., Singer, H., Sorter, J ., & Frazee, C. (1986). The effect of metacognitive instruction in outlining and graphic organizer construction on students' comprehension in a tenth-grade world history class. Journal of Reading Behaviorfl. 153-169. Brown, A. L., & Palincsar, A. S. (1984). Reciprocal teaching of comprehension- fostering and monitoring activities. Cognitive Instruction, 1, 175-177. ‘ 96 Brown, K. G. (1999). Individual differences in choice during learning: The influence of learner goals and attitudes in web-based training. Unpublished doctoral dissertation, Michigan State University, East Lansing. Brown, K. G., Mullins, M., Weissbein, D., Toney, R., & Kozlowski, S. W. J. (1997, April). Mastery goals and strategic reflection: Preliminary evidence for learning interference. In S. W. J. Kozlowski (Chair), Metacognition and training Symposium presented at the 12th Annual Conference of the Society for Industrial and Organizational Psychology, St. Louis, MO. Butler, R. (1993). Effects of task- and ego-achievement goals on information seeking during task engagement. Journal of Personality and Social Psychology, 6; 18- 31. Button, S. B., Mathieu, J. E., & Zajac, D. M. (1996). Goal orientation in organizational research: A conceptual and empirical foundation. Organizational Behavior and Human Decision Processes, 67, 26-48. Campbell, J. P. (1988). Training design for performance improvement. In J. P. Campbell, R. J. Campbell & Associates (Eds.), Productivity in organizations (pp. 177- 216). San Francisco: Jossey-Bass. Carver, C. S., & Scheier, M. F. (1982). Control theory: A useful conceptual framework for personality-social, clinical, and health psychology. Psychological BulletirL 92, 111-135. Carver, C. S., & Scheier, M. F. (1990). Origins and functions of positive and negative affect: A control-process view. Psghological Bulletin, 97, 19-35. Cates, W. M. (1992). Fifteen principles for designing more effective instructional hypermedia/multimedia products. Educational Technology, 32, 5-11. Chi, M. T. H. (1996). Constructing self-explanations and scaffolded explanations in tutoring. mmlied Cognitive Psychology, 10 (special issue), S33-S49. it Chung, J ., & Reigeluth, C. M. (1992). Instructional prescriptions for learner control. Educational Technology, 32, 14-20. Clariana, R. B., Ross, S. M., & Morrison, G. R. (1991). The effects of different feedback strategies using computer-administrated multiple-choice questions as instruction. Educational Technology, Research, and Development, 39, 5-17. Colquitt, J. A., & Simmering, M. J. (1998). Conscientiousness, goal orientation, and motivation to learn during the learning process: A longitudinal study. Journal of Applied Psychology, 83, 654-665. Deci, E. L. (1992). On the nature and functions of motivation theories. Psychological Science, 3, 167-171. 97 DeShon, R. P., & Milner, K. R. (1998). Individual differences in information seeking behavior and team performance (Research proposal to Aon Consulting Research Program). East Lansing, MI: Michigan State University, Department of Psychology. DiVesta, F. J ., & Morena, V. (1993). Cognitive control functions of study activities: A compensation model. Contemporary Educational Psychologg 18, 47-65. Earley, P. C., Wojnaroski, P., & Prest, W. (1987). Task planning and energy expended: Exploration of how goals influence performance. Journal of Applied Psychology, 72, 107-114. Etalapelto, A. (1993). Metacognition and the expertise of computer program comprehension. Scandinavian Journal of Education Research, 37, 243—254. Fedor, D. B., Rensvold, R. B., & Adams, S. M. (1992). An investigation of factors expected to affect feedback seeking: A longitudinal field study. Personnel Psychology, £15, 779-805. Fenigstein, A., Scheier, M. F., & Buss, A. H. (1975). Public and private self- consciousness: Assessment and theory. Journal of Counseling and Clinicalflchology, 3;, 522-527. Fisher, S. L. (1995). The role of cognitive effort and learning outcome demands in skill acquisition and learning. Unpublished master’s thesis, Michigan State University, East Lansing. Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 34, 906-911. Fleishman, E. A., & Mumford, M. D. (1989). Individual attributes and training performance. In I. L. Goldstein and associates (Eds), Training and development in organizations (pp. 183-255). San Francisco: Jossey-Bass. Ford, J. K., & Kraiger, K. (1995). The application of cognitive constructs and principles to the instructional systems model of training: Implications for needs assessment, design, and transfer. In C. L. Cooper & I. T. Robertson (Eds), International Review of Industrial and Orgnizational Psychology (pp. 1-48). Chichester: John Wiley & Sons. Ford, J. K., Quinones, M. A., Sego, D. J ., & Sorra, J. S. (1992). Factors affecting the opportunity to perform trained tasks on the job. Personnel Psychology, 45, 511-527. Ford, J. K., Smith, E. M., Weissbein, D. A., Gully, S. M., Salas, E. (1998). Relationships of goal orientation, metacognitive activity, and practice strategies with learning outcomes and transfer. Journal of Applied Psychology, 83, 218-233. 98 Fukai, D., Kitchen, K., & Aurelio, D. (1998). Building customer relations: Web- based training for the home improvement industry. In D. A. Schreiber & Z. L. Berge (Eds), Distance training: How innovative organizations are using technology to maximize learning and meet business objectives (pp. 71-91). San Francisco: Jossey-Bass. Gall, J. E., & Hannafin, J. M. (1994). A framework for the study of hypertext. Instructional Science, 22, 207-232. Garcia, T., & Pintrich, P. R. (1994). Regulating motivation and cognition in the classroom: The role of self-schemas and self-regulatory strategies. In D. H. Schunk & B. J. Zimmerman (Eds), Self-regulation of learning and performance: Issues and educational applications (pp. 127-153). New Jersey: Lawrence Erlbaum. Gay, G. ( 1986). Interaction of learner control and prior understanding in computer-assisted video instruction. Journal of Educational Psychology, 78, 225-227. Gay, 0., Trumbull, D., & Mazur, J. (1991). Designing and testing navigational strategies and guidance tools for a hypermedia program. Journal of Computing Research, 1, 189-202. Gellatly, I. R. (1996). Conscientiousness and task performance: Test of a cognitive process model. Journal of Applied PsychologL 81, 474-482. Gilman, D. A. (1969). Comparison of several feedback methods for correcting errors by computer-assisted instruction. Journal of Educational Psychology, 60, 503-508. Glaser, R., & Bassok, M. (1989). Learning theory and the study of instruction. Annual Review of Psychology, 40, 631-666. Goldstein, I. L., & Gilliam, P. (1990). Training system issues in the year 2000. American Psychologist, 45, 134-143. Goska, R. E., & Ackerman, P. L. (1996). An aptitude-treatment interaction approach to transfer within training. Journal of Educational Psychology, 88, 249-259. Greiner, J. M., & Karoly, P. (1976). Effects of self-control training on study activity and academic performance: An analysis of self-monitoring, self-reward, and systematic-planning components. Journal of Counseling Psychology, 234 495-502. Hiltz, S. R. (1993). Correlates of learning in a virtual classroom. International Journal of Man-Machine Studies, 39, 71-98. Howard, B. (1998). Increasing employee knowledge and understanding of operational systems: Integrating multiple technologies at N YNEX. In D. A. Schreiber & Z. L. Berge (Eds), Distance training: How innovative organizations are using technology to maximize learning and meet business objectives (pp. 92-114). San Francisco: Jossey- Bass. 99 Howell, W. C., & Cooke, N. J. (1989). Training the human information processor: A review of cognitive models. In I. L. Goldstein (Ed), Training and development in organizations (pp. 121-182). San Francisco: Jossey-Bass. Ivancic, K., & Hesketh, B. (1995). Making the best of errors during training. Training Research Journal, 1, 103-125. J acobsen, M. J ., & Spiro, R. J. (1995). Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: An empirical investigation. Journal of Educational Computing Research, 12, 301-333. Jonassen, D. H., & Grabinger, R. S. (1993). Applications of hypertext: Technologies for higher education. Journal of Computing in Higher Education, 4, 12-42. Justice, E. M., & Weaver-McDougall, R. G. (1989). Adults’ knowledge about memory: Awareness and use of memory strategies across tasks. Journal of Educational Psychology, 81, 214-219. Karoly, P. (1993). Mechanisms of self-regulation: A systems view. Annual Review of Psychologyg44, 23-52. Kearsley, G., & Hillelsohn, M. J. (1982). Human factors considerations for computer-based training. Journal of Computer-Based Training, 8, 74-84. Keller, J. M. (1983). Motivational design of instruction. In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status (pp. 383- 434). Hillsdale, NJ: Lawrence Erlbaum Associates. Kilch, N. R., & Feldman, D. C. (1992). The role of approval and achievement needs in feedback seeking behavior. Journal of Managerial Issues, 4. 554-570. Kinzie, M., & Berdel, R. (1990). Design and use of hypermedia systems. Educational Technology, Research, and Development, 38, 61-68. Kluwe, R. H. (1987). Executive decisions and regulation of problem solving behavior. In F. E. Weinert & R H. Kluwe (Eds), Metacognition, motivation, and understanding (pp. 31-64). Kozlowski, S. W. J., Weissbein, D. A., Brown, K. G., Toney, R. J., & Mullins, M. E. (1997). Guiding the development of deployable shipboard training systems: Enhancipg skill acquisition, adaptability, and effectiveness (N61339-96-K-0005). Orlando, FL: Naval Air Warfare Center, Training Systems Division. Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311-328. 100 Lanza, A. (1991). Some guidelines for the design of effective hypercourses. Educational Psychology, 3L 18-22. Lanza, A., & Roselli, T. (1991). Effects of the hypertextual approach versus the structured approach on students’ achievement. Journal of Computer-Based Instruction, 1_8_, 48-50. Lee, S. S., & Lee, Y. H. K. (1991). Effects of leamer—control versus program- control strategies on computer-aided learning of chemistry problems: For acquisition or review? Journal of Educational Psychology, 83, 491—498. Lord, R. G., & Maher, K. J. (1991). Cognitive theory in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds), Handbook of Industrial and organizational Psychology (pp. 1-62). Palo Alto, CA: Consulting Psychologists Press. Lorene, L., Sturmey, P., & Brittain, H. (1992). Evaluation of a meta-cognitive strategy to improve the information gained from a stroke information pack. Stress Medicine, 8, 111-112. MacLachlan, J. (1986). Psychologically based techniques for improving learning with computerized tutorials. Journal of Computer-Based Instruction, 13, 65-70. Major, D. A., & Kozlowski, S. W. J. (1997). Newcomer information seeking: Individual and contextual influences. International Journal of Selection and Assessment, 5, 16-28. Martocchio, J. J., & Webster, J. (1995). Effects of feedback and cognitive playfulness on performance in microcomputer software training. Personnel Psychology, $553678. Mathieu, J. E., & Martineau, J. W. (1997). Individual and situational influences in training motivation. In J. K. Ford and associates (Eds), Improving training effectiveness in work organizations (pp. 193-222). Hillsdale, NJ: Lawrence Erlbaum Associates. Mathieu, J. E, Martineau, J. W., & Tannenbaum, S. I. (1993). Individual and situational influences on the development of self-efficacy: Implications for training effectiveness. Personnel Psychology, 48, 125-147. Milheim, W. D. (1996). Interactivity and computer-based instruction. Journal of Educational Technology Systems, 24, 225-233. Miller, G. A., Galanter, E., & Pribram, K. H. (1960). Plans and the structure of behavior. New York: Adams, Bannister, Cox. Miller, V. D., & Jablin, F. M. (1991). Information seeking during organizational entry: Influences, tactics, and a model of the process. Academy of Management Review, fl, 92-120. 101 Milner, K. R. (1999). Supervisors’ impressions of employees who seek job performance feedback. Unpublished master’s thesis, Michigan State University, East Lansing. Morrison, E. W., & Bies, R. J. (1991). Impression management in the feedback seeking process: A literature review and research agenda. Academy of Management Review 16 522-541. Nelson, T. O. (1996). Consciousness and metacognition. American Pwhologist, 51, 102-116. Nelson, T. 0., & N arens, L. (1990). Metamemory: A theoretical framework and new findings. The Psychology of Learning and Motivation, 26, 125-141. Nelson, T. 0., Dunlosky, J ., Graf, A., & Narens, L. (1994). Utilization of metacognitive judgments in the allocation of study during multitrial learning. Psychological Science. 5, 207-213. Noe, R. A. (1986). Trainees’ attributes and attitudes: Neglected influences on training effectiveness. Academy of Maaagement Review, 11, 736-749. Noe, R. A., & Ford, J. K. (1992). Emerging issues and new directions for training research. Research in Personnel and Human Resources Management, 10, 345-384. Northcraft, G. B., & Ashford, S. J. (1990). The preservation of self in everyday life: The effects of performance expectations and feedback context on feedback inquiry. Organizational Behavior and Human Decision Processes, 4L 42-64. Osman, M. E., & Hannafin, M. J. (1992). Metacognition research and theory: Analysis and implications for instructional design. Educational Technology, Research, and DevelopmentL40, 83-99. Park, I., & Hannafin, M. J. (1993). Empirically-based guidelines for the design of interactive multimedia. Educational Technology Research and DevelopmentAl, 63-85. Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of a classroom academic performance. Journal of Educational Psychology, 82, 33-40. Pokay, P., & Blumenfeld, P. C. (1990). Predicting achievement early and late in the semester: The role of motivation and use of learning strategies. Journal of Educational Psychology, 82, 41-50. Porter, L. R. (1997). Creating the virtual classroom: Distance learning with the internet. New York: John Wiley & Sons. Pressley, M., Levin, J. R., & Ghatala, E. S. (1984). Memory strategy monitoring in adults and children. Journal of Verbal Learning and Verbal Behavior, 23, 270-288. 102 Pressley, M., Snyder, B. L., Levin, J. R., Murray, H. G., & Ghatala, E. S. (1987). Perceived readiness for examination performance (PREP) produced by initial reading of text and text containing adjunct questions. Reading Awarenesst, 219-236. Quinones, M. A. (1995). Pretraining context effects: Training assignment as feedback. Journal of Applied Psychology, 80, 226-238. Redding, R. E. (1990). Metacognitive instruction: Trainers teaching thinking skills. Performance Improvement Quarterly, 3, 27-41. Reeves, T. C., & Reeves, P. M. (1997). Effective dimensions of interactive learning on the World Wide Web. In B. H. Khan (Ed.), Web-based instruction (ch. 6). Englewood Cliffs, NJ: Educational Technology Publications. Ridley, D. S., Schutz, P. A., Glanz, R. S., & Weinstein, C. E. (1992). Self- regulated learning: The interactive influence of metacognitive awareness and goal- setting. Journal of Experimental Education, 60, 293-306. Rinehart, S. D., & Platt, J. M. (1984). Metacognitive awareness and monitoring in adult college readers. Forum for Reading, 15, 54-62. Sasscer, M. F., & Moore, D. M. (1984). A study of the relationship between leamer-control patterns and course completion in computer-assisted instruction. Programmed Learning and Educational Technology, 21, 28-33. Scardamalia, M., Bereiter, C., & Steinbach, R. (1984). Teachability of reflective processes in written composition. Cognitive Science, 8, 173-190. Schmidt, R. A., & Bjork, R. A. (1994). New conceptualizations of practice: Common principles in three paradigms suggest new concepts for training. Psychological Science 3 207-217. Schoenfeld, A. H. (1983). Problem solving in the mathematics curriculum: A report, recommendations, and an annotated bibliography. Math Association of America Notes No. l. Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19, 460-475. Schreiber, D. A. (1998). Organizational technology and its impact on distance training. In D. A. Schreiber & Z. L. Berge (Eds), Distance training: How innovative gganizations are using technology to maximize learning and meet business objectives (pp. 3-18). San Francisco: Jossey-Bass. Schunk, D. H. (1989). Self-efficacy and achievement behaviors. Educational Psychology Review, 1, 173-208. 103 Seabaugh, G. O., & Schumaker, J. B. (1994). The effects of self-regulation training on the academic productivity of secondary students with learning problems. Journal of Behavioral EducationA, 109-133. Simmering, M. J. (1999). The effects of learner control and individual differences on learning outcomes. Doctoral dissertation under preparation, Michigan State University, East Lansing. Simpson, M. L., Hayes, C. G., Stahl, N., & Weaver, D. (1988). An initial validation of a study strategy system. Journal of Reading Behavior, 20, 149-180. Smith, E. M. (1996). The effects of individual differences, discovery learning, and metacognition on learning and adaptive transfer. Unpublished doctoral dissertation, Michigan State University, East Lansing. Smith, E. M., Ford, J. K., & Kozlowski, S. W. J. (1997). Building adaptive expertise: Implications for training design strategies. In M. A. Quinones & A. Ehrenstein (Eds), Training for a rapidly changing worlgglace (pp. 89-118). Washington, D. C.: American Psychological Association. Steinberg, E. R. (1989). Cognition and learner control: A literature review, 1977- 1988. Journal of Computer-Based Instruction; 16, 117-121. Swanson, H. L. (1990). Influence of metacognitive knowledge and aptitude on problem solving. Journal of Educational Psychology, 82, 306-314. Tannenbaum, S. I., & Yukl, G. (1992). Training and development in work organizations. Annual Review of Psychology, 43, 399-441. Tennyson, R. D., & Rothen, W. (1979). Management of computer-based instruction: Design of an adaptive control strategy. Journal of Computer-Based Instruction, 5, 63-71. Tennyson, R. D., Christiansen, D. L., & Park, S. I. (1984). The Minnesota adaptive instructional system: An intelligent CBI system. Journal of Computer-Based Instruction, 1 1, 2-13. Tessmer, M. (1995). Formative multimedia evaluation. Training Research Journal 1 127-149. Toney, R. J ., & Kozlowski, S. W. J. (April, 1999). Effects of evaluative feedback and task difficulty on learning and training performance. In S. Gully (Chair), Learning to fail or failing to learn? The role of errors, failure, and feedback in learning environments. Symposium presented at the 14th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA. Tsai, C. J. (1988). Hypertext: Technology, applications, and research issues. Journal of Educational Technology Systems, 17, 3-14. 104 Vadhan, V. & Stander, P. (1994). Metacognitive ability and test performance among college students. Journal of Psgycholggg 128, 307-309. VandeWalle, D., & Cummings, L. L. (1997). A test of the influence of goal orientation on the feedback-seeking process. Journal of Applied Psychology, 82, 390- 400. Veenman, M. V. J ., Elshout, J. J ., & Busato, V. V. (1994). Metacognitive mediation in learning with computer-based simulations. Computers in Human Behavior, 10, 93-106. Volet, S. E. (1991). Modeling and coaching of relevant metacognitive strategies for enhancing university students’ learning. Learning and Instruction, L 319-336. Walker, S. (1998). The value of building skills with online technology: Online training costs and evaluation at the Texas Natural Resource Conservation Commission. In D. A. Schreiber & Z. L. Berge (Eds), Distance training: How innovative organizations are using technology to maximize learning and meet business obfictives (pp. 270-286). San Francisco: Jossey-Bass. Warn, M., Compton, C., Levine, S., & Whitteker, S. (1998). Graduate programs at a distance: A partnership between the California Department of Rehabilitation and San Diego State University. In D. A. Schreiber & Z. L. Berge (Eds.), Distance training: How innovative organizations are using technology to maximize learning and meet business objectives (pp. 155-186). San Francisco: Jossey-Bass. Warr, P., & Bunce, D. (1995). Trainee characteristics and the outcomes of open learning. Personnel Psychology, 48, 347-375. Weiss, H. M., & Adler, S. (1984). Personality and organizational behavior. Research in organizational behavior, 6, 1-50. Weissbein, D. A. (1996). The effects of goal type and metacognitive training on complex skill acquisition: Implications of the limited resources model. Unpublished master’s thesis, Michigan State University, East Lansing. Zacks, R. T. (1969). Invariance of total learning time under different conditions of practice. Journal of Experimental Psychology, 82, 441-447. 105 APPENDIX A Consent and Debriefing Information Informed Consent Explanation of research The learning behaviors of trainees in a web-based training program will be examined. Procedures and estimate of time You will complete a training program that will teach you how to create a web page. You will have opportunities to practice and receive feedback on your progress. At the end of the study you will be asked to create and submit your own web page. The study is expected to take 3 hours. Participation Participation in this study is voluntary. You may choose not to participate in some or all parts of the study. You may discontinue the experiment at any time without penalty. Confidentiality Your privacy Will be protected to the max1mum extent allowable by law. Data gathered from you during this study will be strictly confidential. Your responses will remain anonymous in any research reports. At your request, your results may be made available to you. Risks and costs There are no risks or costs associated with your participation. Principal investigator Head of the Department of Psychology University Committee on Research Involving Human Subjects Rebecca Toney, toneyreb@msu.edu 353-2880 Dr. Gordon Wood 355-9563 David E. Wright 355-2180 Agreement to Participate The procedures and possible risks of the experiment have been explained. Do you understand and fully consent to participate in the study described above? D Yes E] No If you marked “Yes,” please enter your PID number: A******** If you marked “No,” please exit the experiment at this time. There are a number of books at your local library from which you can you can learn the same information without participating in a research study. 106 Debriefing The study in which you just participated was designed to examine what links you chose (links to information, practice, and feedback) and how that affected your learning process. The investigator is also examining the effects of metacognition (monitoring and regulating your own behavior) on your learning process and training performance. If you have any questions about this study or would like to receive a copy of the results when they are complete, please notify the investigator now or by e-mail at toneyreb@msu.edu. Thank you for participating in this study. 107 APPENDIX B Manipulations Metacognitive Monitoring “You have just completed one lesson of the training program and are ready to begin the next. Take a couple of minutes now to think about how you are learning the information you need to know in order to create your web page. You don’t need to go back and look at the last lesson—there are no “right” answers in this section, so just jot down your thoughts. “First, think about what information from the last lesson you know, and enter some brief notes about it here: “Second, think about what information you need to know from the last lesson but you may not remember it or know it well enough, and enter some brief notes about it here: “Third, think about how the information you have just learned in the last lesson relates to information you learned in previous lessons: “Finally, think about how the information you learned in this last lesson might relate to information you will be learning in the next lesson you have selected: Metacognitive Control “You have just completed one lesson of the training program and are ready to begin the next. Take a couple of minutes now to think about how you are learning the information you need to know in order to create your web page. You don’t need to go back and look at the last lesson—there are no “right” answers in this section, so just jot down your thoughts. “First, think about what approach or strategy you were using to learn the information in the last lesson. What was your approach? Did it work?: “Second, think about how you might improve your approach to learning the information in the next lesson. What will you do the same? What will you do differently?: “Third, list three steps in your plan for approaching the next lesson: 108 II APPENDIX C Measures Prior Experience Scale Participants will mark one of the five boxes shown below, which will follow each statement. C] All the Time E] Frequently E] Sometimes [:1 Seldom El Never 1. I spend time making my own page for the web. 2. I surf the web for enjoyment. 3. I use the internet to find information for work or classes. 4.*I have taken courses on the internet. 5. Making my own web page is something I’ve thought about doing. 6. I purchase products or services over the internet. 7. When I surf the web, I follow links to explore. 8. I use the internet to learn new things. Please estimate the number of hours you have spent designing and creating web pages: Self-Efficacy Scale Participants will mark one of the five boxes shown below, which will follow each statement. [:I Strongly Agree I: Agree El Neutral [:IDisagree [:1 Strongly Disagree 9. Compared with others in this training program, I expect to do well. 10. I’m certain I can understand the ideas taught in this course. 11. I expect to do very well in this training course. 12. Compared with others in this course, I think I’m a good trainee. 13. I’m sure I can do an excellent job on the tasks assigned in this training course. 14. I think I will perform well in this course. 15. My learning skills are excellent compared with other trainees in this course. 16.* Compared with other trainees in this course I think I know a lot about web page design. 17. I know that I will be able to learn the material for this training course. *Items dropped from scales 109 Mastery/Performance Orientation Scale Participants will mark one of the five boxes shown below, which will follow each statement. [3 Strongly Agree [:1 Agree El Neutral [:lDisagree E] Strongly Disagree 18. The opportunity to do challenging work is important to me. 19. I do my best when I’m working on a fairly difficult task. 20. I try hard to improve on my past performance. 21. When I have difficulty solving a problem, I enjoy trying different approaches to see which one will work. 22. The opportunity to learn new things is important to me. 23. The opportunity to extend the range of my abilities is important to me. 24. I prefer to work on tasks that force me to learn new things. 25. When I fail to complete a difficult task, I plan to try harder the next time I work on it. 26. The things I enjoy the most are the things I do the best. 27. I feel smart when I can do something better than most other people. 28. I like to be fairly confident that I can successfully perform a task before I attempt it. 29. I am happiest at work when I perform tasks on which I know I won’t make any errors. 30. I feel smart when I do something without making any mistakes. 31. I prefer to do things that I can do well rather than things that I do poorly. 32. The opinions others have about how well I can do certain things are important to me. 33. I like to work on tasks that I have done well on in the past. Metacognitive Skills Scale Participants will mark one of the five boxes shown below, which will follow each statement. E] Strongly Agree E] Agree [:1 Neutral [:IDisagree [:1 Strongly Disagree 34. I ask myself questions to make sure I know the material I have been studying. 35. I work on practice exercises and answer end of chapter questions even when I don’t haveto. 36. *When I am learning, I try to determine which concepts I don’t understand well. 37. Before I begin studying I think about the things I will need to do to learn. 38. I often find that I have been reading for a class but I don’t know what it is all about. 39. When an instructor is talking, I think of other things and don’t really listen to what is said. 40. When I’m reading, I stop once in a while and go over what I have read. *ltem dropped from scale 110 Final Self-Efficacy Scales Participants will mark one of the five boxes shown below, which will follow each statement. D Strongly Agree [:1 Agree E] Neutral CIDisagree I: Strongly Disagree 41. I can meet the challenges of creating a basic web page. 42. I am confident in my understanding of how different elements of web page design are related. 43. I can create a web page on my own, without help. 44. I am certain that I can manage the requirements of web page design, even when problems occur. 45. I believe I can handle more difficult elements of webpage design. 46. I am confident that I can meet the challenges of creating web pages that are more complex. Please rate your confidence in your ability to complete the following tasks: Create a topic of content Create a title Change background color Change text colors Create a table Insert a horizontal line Insert a graphic Insert a hyperlink Create meta tags Set background texture Format the text Insert scrolling text Format text using text headings Align text with graphics *Insert placeholders for graphics Create thumbnail images Create linked graphics Create bookmarks *ltem dropped from scale 111 (0%-100% confident) (0%-100% confident) (0%-100% confident) (0%-100% confident) (0%- 100% confident) (0%-100% confident) (O%-100% confident) (0%-100% confident) (0%-100% confident) (0%-100% confident) (0%-100% confident) (0%-100% confident) (0%-100% confident) (0%-100% confident) (0%-100% confident) (0%-100% confident) (0%-100% confident) (0%-100% confident) Performance Rating Checklist One point scored for each item present on submitted web page Create a topic of content Create a title Change background color Change text colors Create a table Insert a horizontal line Insert a graphic Insert a hyperlink Create multiple pages Create meta tags Set background texture Format the text Insert scrolling text Format text using text headings Format tables Align text with graphics *Insert placeholders for graphics Create thumbnail images Create linked graphics Create bookmarks *ltem dropped from scale 112 APPENDIX D Hypotheses and Analyses Hyp Independent Variable(s) Dependent Variable Result 1 2a 2b 2c + Mastery orientation - Prior experience + Mastery orientation - Prior experience + Performance orientation + Mastery orientation - Self-efficacy - Prior experience + Mastery orientation - Prior experience - Self-efficacy + MC skills MC monitoring X MC skills + Mastery orientation + Prior experience +Self—efficacy + Mastery orientation - Performance orientation - Prior experience +Self—efficacy - MC skills MC control X MC skills Breadth of seeking Descriptive feedback seeking Evaluative feedback seeking Guidance feedback seeking Frequency of feedback seeking Depth of feedback seeking Complexity of information seeking Frequency of information seeking Depth of information seeking Not significant (ns) [IS ['18 Significant, positive effect of prior experience ns ns 113 (table continues) Hyp Independent Variable(s) Dependent Variable Result 8 + Mastery orientation Complexity of ns + Prior experience practice seeking +Self-efficacy 9 + Mastery orientation Frequency of ns - Performance orientation practice seeking - Prior experience +Self-efficacy - MC skills 10 MC control X MC skills Depth of practice ns seeking 1 l + Breadth of seeking Learning Significant, + Complexity of information seeking positive effect of + Frequency of information seeking breadth of seeking 12 MC monitoring X MC control Learning ns 13 - Frequency of feedback seeking Efficiency ns - Frequency of information seeking - Frequency of practice seeking 14 + Self-efficacy (covariate) Final self-efficacy Significant + Breadth of seeking covariate + Complexity of information seeking Significant, + Frequency of information seeking positive effect of + Complexity of practice seeking complexity of + Frequency of practice seeking information seeking 15 + Breadth of seeking Performance Significant, + Descriptive feedback seeking negative effect of - Evaluative feedback seeking breadth of seeking + Guidance feedback seeking Significant, + Frequency of feedback seeking positive effect of + Complexity of practice seeking complexity of + Frequency of practice seeking practice seeking 16 MC monitoring X MC control Performance ns 114 IIIIIIIIIIIIIIIIIIIIIIIIIIIIIII llll'l’lll‘“lllllll‘l'l'l'll