IMPROVING THIRD GENERATION LEARNING: THE EFFECTS OF PEER FEEDBACK TRAINING ON QUALITY FEEDBACK, TRAINEE CHARACTERISTICS, AND PERFORMANCE By Adam John Massman A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Psychology 2012 ABSTRACT IMPROVING THIRD GENERATION LEARNING: THE EFFECTS OF PEER FEEDBACK TRAINING ON TRAINING CHARACTERISTICS AND PERFORMANCE By Adam John Massman As organizations build technology enabled, social learning environments, Kraiger (2008) suggested that organizations are on the brink of the next generation of learning. Third generation learning is a peer-to-peer learning environment where peers sustain the learning culture typically within an online learning environment. One of the major assumptions of this learning environment is that the learners sufficiently understand how and when to give each other feedback to sustain effective learning experiences (Sadler, 1989). Research on peer feedback has shown that, while the benefits of peer feedback are positive, peers lack adequate skills to give proper peer feedback. Crafting quality peer feedback within these environments may affect the entire learning process from trainee characteristics (e.g. London & Smither, 1995) and training outcomes (e.g. Nicol & Macfarlane-Dick, 2006). Thus, this dissertation crafted an intervention to increase peer feedback effectiveness based on five principles of quality feedback. Previous researchers have attempted to increase peer feedback quality through training (e.g. Sluijsmans et al., 2001); however, these researchers have had mixed results, mostly because they did not rely on traditional feedback research on constructing quality feedback. In this dissertation, learners who received peer feedback training were compared to participants who did not receive peer feedback training. A participant’s feedback environment and feedback orientation were measured to determine how these can moderate peer feedback quality and trainee characteristics, respectively. Trainee characteristics were assessed and training outcomes were assessed to determine the impact of quality peer feedback, ultimately hypothesized as a result of peer feedback training. Results indicated that participants who received peer feedback training resulted in higher quality feedback, higher trainee characteristics, and training outcomes. Higher quality feedback resulted in higher knowledge. A participant’s feedback environment and feedback orientation did not moderate peer feedback quality and trainee characteristics, respectively. Self-efficacy and motivation mediated the relationship between quality and outcomes. Implications for training are discussed. Copyright by ADAM JOHN MASSMAN 2012 ACKNOWLEDGEMENTS The road to the completion of my dissertation is one that I will remember for my entire life. This road was filled with many obstacles to overcome, but also filled with a wealth of wonderful people who helped me get to the finish line. For these people, I am eternally grateful. For intellectual support, I give the most gratitude to my dissertation chair, Dr. J. Kevin Ford. His willingness to push me to my intellectual limits and help me be a better scientist is something that defines him as a person. I could not ask for a better chair, as his compassion for his students with his commitment to organizational research is a unique combination of strengths. I offer additional thanks to my three other committee members, Dr. Ann Marie Ryan, Dr. John Schaubroeck, and Dr. Steve Yelon. These three people have taught me how to think through organizational research in new ways. For that and much more, I am eternally grateful. I offer final thanks to Dr. J. Brodie Gregory and Dr. A Silke McCance for your intellectual contributions and your execution of the brutal ratings calibration. For emotional support, there are endless amounts of people who I could list here. Ultimately, these individuals know who they are, but I felt it necessary to list people who were instrumental to carry me to the end. To Alan Johnson and Lauren Robles, your constant encouragement during the high stress times is something that I will never forget. To my managers, L. Flanagan and R. Gibby, I cannot express how much your embracing of my career and professional goals has facilitated the ultimate completion of my degree. To my mother, I cannot offer up enough thanks for your prayers and encouragement. As I said before, I cannot list all the wonderful people that have helped me get this far – but I hope that I can one day return the favor to help each person listed live their life dreams to their fullest. v TABLE OF CONTENTS LIST OF TABLES ......................................................................................................................... ix LIST OF FIGURES ........................................................................................................................ x INTRODUCTION .......................................................................................................................... 1 Training Intervention History ..................................................................................................... 5 First Generation Instructional Design ..................................................................................... 6 Second Generation Instructional Design ................................................................................ 8 Third Generation Instructional Design ................................................................................. 11 Theoretical Framework for Peer Feedback ............................................................................... 14 Constructivism ...................................................................................................................... 14 Cooperative Learning............................................................................................................ 15 Zones of Proximal Development .......................................................................................... 17 An Introduction to Feedback Theory and Research ................................................................. 18 Timing ................................................................................................................................... 20 Sign of Feedback................................................................................................................... 23 Frequency of Feedback ......................................................................................................... 25 Specificity of the Feedback ................................................................................................... 27 The Type of Feedback .......................................................................................................... 29 The Source of the Feedback .................................................................................................. 33 Peer Feedback ........................................................................................................................... 37 Five Tracks of Peer Feedback ............................................................................................... 38 Training Outcomes for Peer Feedback ................................................................................. 41 Trainee Characteristics Affected by Peer Feedback ............................................................. 44 Variables Impacting Receptivity, Acceptance, and Outcomes of Peer Feedback ................ 48 Summary of Peer Feedback .................................................................................................. 52 Considerations for Peer Feedback & Peer Feedback Training ................................................. 53 Reliability of Peer Feedback ................................................................................................. 53 Understanding How to Give Proper Feedback & Training .................................................. 56 Framework for Improving Feedback in Learning Environments ............................................. 59 Framework for Current Study ................................................................................................... 62 The Theoretical and Practical Need for This Study .............................................................. 62 A Conceptual Model ............................................................................................................. 63 HYPOTHESES ............................................................................................................................. 67 Feedback-Giver Hypotheses ..................................................................................................... 67 Feedback Quality .................................................................................................................. 67 Feedback Environment ......................................................................................................... 73 Feedback-Receiver Hypotheses ................................................................................................ 75 Receiver Self-Efficacy .......................................................................................................... 75 Receiver Motivation.............................................................................................................. 76 Receiver Feedback Orientation ............................................................................................. 78 Training Outcomes................................................................................................................ 79 vi METHODS ................................................................................................................................... 82 Participants ............................................................................................................................ 82 Design ................................................................................................................................... 82 Task ....................................................................................................................................... 83 Procedure .............................................................................................................................. 84 MEASURES ............................................................................................................................. 89 Feedback Environment Scale ................................................................................................ 89 Feedback Orientation Scale .................................................................................................. 90 Self-Efficacy Scale................................................................................................................ 91 Motivation Scale ................................................................................................................... 91 Knowledge Assessment (Knowledge Outcome) .................................................................. 92 Adult Learning Task (Behavioral Outcome) ........................................................................ 92 Feedback Utility .................................................................................................................... 93 Training Utility ..................................................................................................................... 93 Feedback Quality .................................................................................................................. 95 Analytic strategy ....................................................................................................................... 97 RESULTS ..................................................................................................................................... 99 Descriptive Data........................................................................................................................ 99 Model & Hypotheses Investigation .......................................................................................... 99 Hypothesis 1........................................................................................................................ 100 Hypothesis 2........................................................................................................................ 101 Hypothesis 3........................................................................................................................ 101 Hypothesis 4........................................................................................................................ 102 Hypothesis 5........................................................................................................................ 102 Hypothesis 6........................................................................................................................ 103 Hypothesis 7........................................................................................................................ 103 Hypothesis 8........................................................................................................................ 104 Hypothesis 9........................................................................................................................ 104 Hypothesis 10...................................................................................................................... 105 Additional Post-Experimentation Analyses ............................................................................ 105 The Effect of Condition on Trainee Characteristics and Outcomes ................................... 105 The Effect of Trainee Characteristics on Outcomes ........................................................... 107 Effect of Peer Partner .......................................................................................................... 108 Effect of Time ..................................................................................................................... 110 DISCUSSION ............................................................................................................................. 112 Summary of the Results .......................................................................................................... 112 The Effect of Condition Assignment .................................................................................. 112 The Effects of Quality Feedback ........................................................................................ 115 Interactive Effects of Individual Traits ............................................................................... 118 Mediating Effects ................................................................................................................ 119 Conclusions ......................................................................................................................... 119 Implications for Practice & Directions for Future Research .................................................. 120 Third Generation Learning ................................................................................................. 120 vii Training Effectiveness ........................................................................................................ 123 Peer Feedback Training ...................................................................................................... 126 Feedback Research.............................................................................................................. 128 Study Limitations .................................................................................................................... 130 PRACTICAL IMPLICATIONS & CONCLUSIONS ................................................................ 132 APPENDICES ............................................................................................................................ 134 Appendix A: Consent Form .................................................................................................... 135 Appendix B: Peer Feedback Training Module ....................................................................... 137 Appendix C: The Feedback Environment Scale ..................................................................... 141 Appendix D: The Feedback Orientation Scale ....................................................................... 142 Appendix E: Self-Efficacy Scale ............................................................................................ 143 Appendix F: Motivation Scale ................................................................................................ 144 Appendix G: Knowledge Assessment .................................................................................... 145 Appendix H: Adult Learning Task ......................................................................................... 149 Appendix I: Feedback Utility.................................................................................................. 151 Appendix J: Training Utility ................................................................................................... 152 Appendix K: Debrief Letter .................................................................................................... 153 Appendix L: Figures ............................................................................................................... 155 Appendix M: Tables ............................................................................................................... 162 REFERENCES…………………………………………………………………………………170   viii LIST OF TABLES Table 1. Training Lessons, Feedback Exercises, And Relation To Coures Objectives .............. 162 Table 2. Expert Peer Feedback Grading Rubric ......................................................................... 163 Table 3. Overall Means, Standard Deviations, And Group Means ............................................ 165 Table 4. Correlations And Reliabilities For Variables................................................................ 166 Table 5. Correlations For Outcomes And Feedback Elements ................................................... 168 ix LIST OF FIGURES Figure 1: A Conceptual Model ................................................................................................... 155 Figure 2: Interaction Of Feedback Environment And Feedback Training On Feedback Quality ......................................................................................................... 156 Figure 3: Interaction Of Feedback Quality And Feedback Orientation On Trainee Characteristics................................................................................................. 157 Figure 4. Procedural Diagram Of Study ..................................................................................... 158 Figure 5. Peer Feedback Quality Rating Over Time................................................................... 159 Figure 6. Peer Feedback Elements Over Time ........................................................................... 160 Figure 7. Resulting Conceptual Model ....................................................................................... 161 x INTRODUCTION In modern business, which is less dependent on tradition and more dependent on evolution, training has evolved as our understanding of learning has evolved. As an example, classical conditioning inspired much of the stimulus-response based training that was prevalent during the 1950s and 1960s (Kraiger, 2008). Years later, Bandura (1977)’s research on social learning inspired behavioral modeling, which has penetrated corporate training (Decker & Nathan, 1985). Similarly, Kraiger and Ford (2006) noted that training has moved from a scientific management era to a more participative era. Thus, the world of training continues to evolve in strategies to enhance learning. Kraiger (2008) has suggested that the next evolution in learning, which is inspired by technological advances, constructivist theory, and adult learning theory, is called third generation learning. Third generation learning is a mode of training that “places greater emphasis on the learner forming understanding largely through a process of social negotiation” (p. 454) which is ideally delivered through web-based networked training (Arbaugh, 2008). Of the many components that enable third generation learning, social negotiation through peers is one of the primary sources in which a learner can extract meaning from training and engage in on the job transfer because learning happens from collective knowledge exchange, feedback exchanges, and storytelling. Kraiger’s theory is not new or revolutionary. For example, Moore (1989) predicted that peer-to-peer learning would emerge as a critical consideration for distance education. He further suggested that peer-to-peer interactions foster learning in the training context, as well as help develop other critical skills such as communication and leadership. Thus, the emphasis on the 1 effect that peer feedback has on the recipient continues to be at the center of third generation learning many years later. Drawing off the fundamentals of Ilgen, Fisher, and Taylor (1979) model that notes that the source of the feedback is a component of the effectiveness of feedback, third generation learning emphasizes the peer as a primary source of feedback, which results in improved learning. This makes sense given work has become more project-oriented and less linear, involving more interactions with peers to improve skills. Through these peer feedback interactions, learners can “interpret, clarify, and validate their understanding through sustained dialogue (i.e. two way communications) and negotiation” (Garrison, 1993, p. 202). For nearly half a century, educational settings have used this type of peer-to-peer feedback to guide students’ learning (Sluijsmans, Brand-Gruwel, & Van Merriënbor, 2002). Peer feedback is grounded in the theory that individuals in the same situation can assist each other to perfect performance. Peer feedback is messaging that is given by other people in the same field or setting to assist their fellow colleagues achieve their end state. Professors commonly use this to enhance the quality of assignments or experiences. The health education system has been on the forefront of creating systems around peerto-peer learning. Coined as “PAL,” peer assisted learning, this collaborative learning strategy includes peer-to-peer learning, peer-to-peer teaching, peer-to-peer mentoring, and peer leadership (Henning, Weidner, & Marty, 2008). In fact, some national surveys suggest that peer training happens in over 50% of cases of health training (Sluijsmans, Brand-Gruwel, Van Merriënbor, & Bastiaens, 2003). While the education system has been researching this effect, the corporate world has only begun to venture into third generation learning. The use of peer feedback in corporate training is 2 relatively unexplored and has unknown outcomes (Van den Bossche, Segers, & Janssen, 2010). The closest iterations of peer feedback corporations have adopted are 360-degree feedback assessments (Westerman & Rosse, 1997), which will be discussed in the dissertation. There are many concerns when it comes to embracing peer feedback in corporate learning environments. First, there is the question as to whether peer feedback inside a training environment is as useful as instructor feedback (Falckikov & Goldfinch, 2000). Second, peers are competitive within organizations (Edmondson, 2002), which may prevent trainees to cooperatively learn for fear of losing a competitive knowledge or skill. Third, many students and teachers are concerned with the validity/reliability of peer assessment in education and learning (Falckikov & Goldfinch, 2000). It is still premature to make final judgment on the validity of the fears listed, as researchers are still understanding the nature and effect of peer feedback in learning. Early research suggests that peer feedback may be accurate and beneficial as it may increase selfassessment, reflection, increased accountability, better problem solving skills, and increased confidence (e.g. Erikson, 1987; Papinczak, Young, Groves, & Haynes, 2007; Vickerman, 2008; Hanrahan & Isaacs, 2001; Ladyshewsky & Gotjamos, 1997, respectively). However, other studies have suggested that while the peer feedback was accurate, it was not corrective. That is, the undesired behaviors did not change after the feedback was given. With these conflicting reports on the effect of peer feedback, it is important that corporations ensure peer feedback is valuable, especially as organizations begin adopting third generation learning and more webbased training. Thus, this study attempts to understand how peer feedback training can improve the effectiveness of third generation learning strategies. With the future of training leaning towards 3 peer-to-peer learning, peer feedback interactions need to be as effective as possible (Kraiger, 2008). Thus, if this study can assist the effectiveness of third generation learning through one facet, the return on investment for this type of training will evolve training further. The importance of this study is two-fold. First, third generation learning is gaining wide popularity. Many corporations believe that social learning through online learning platforms will enhance the learning environment and bring a new way of learning (Allen & Naughton, 2011). The use of “Learning 2.0” learning platforms is growing with the assumption that social learning will automatically engage peer-to-peer learning. Learners are given these tools without much understanding of how to engage with others. Thus, the need to equip learners with the proper training for using these tools becomes of critical importance. Second, training researchers have not determined how to improve web-based training. Many studies have concluded that webbased training is similar to classroom in many aspects (e.g. Hylton, 2006) and in some cases can be superior (Sitzmann, Kraiger, Stewart, & Wisher, 2006). This proposed intervention may be a viable option for improving the effectiveness of web-based training based on the assumption that feedback during the learning process is critical to a learner’s success. Decades of research have shown that feedback is critical to the learning environment (e.g. Komaki, Heinzmann, & Lawson, 1980). For example, Kluger and DeNisi (1996) found that feedback motivated and focused attention to increase learning and performance. Thus, the effect of peer feedback – if done correctly – may benefit the learner in a variety of learning outcomes in addition to those listed previously (e.g. Guiford, 2001; Topping, Smith, Swanson, & Elliot, 2000; Xiao & Lucking, 2008). For example, peers have more interaction with fellow peers during the training event than with the instructor, there would be more opportunities for a learner to correct performance. Additionally, the frequency of feedback is increased given that the peers can 4 interact with other peers at any time, not just at scheduled assessment or evaluation periods. However, there are several assumptions with which third generation learning is built upon that may not be accurate: (1) peer feedback happens regularly in social learning environments without prompting, (2) peer feedback is constructive, and (3) peers know how to deliver quality feedback. Because of the importance of feedback and the effective construction of quality peer feedback, this study proposes that, if peers are deliberately trained on how to deliver high quality feedback using traditional pillars of feedback research, then the learning inside third generation training will be higher. The intervention selectively focuses on five elements of traditional feedback research and analyzes the outcomes. The following sections illustrate the background to the intervention through: (1) a review of the research on the generations of learning, (2) a theoretical background to support peer feedback in learning, (3) an introduction to feedback theory and research, (4) the research on peer feedback, (5) the review of considerations for peer feedback and the limited research on peer feedback training, (6) a discussion of the model to test the proposed intervention, and, finally (7), a discussion of the hypotheses. Training Intervention History Training research continues to evolve our understanding on how to improve training experiences. Starting with Tyler (1949) and Gagne (1962)’s work on instructional principles, current training researchers are uncovering how to improve training delivery and design. For example, Cannon-Bowers, Salas, Tannenbaum, and Mathieu (1995) noted that the context in which the employee learns the information affects the ability for the knowledge transfer to occur successfully. More recently, Heimbeck, Frese, Sonnentag, and Keith (2003) and Joung, Hesketh, 5 and Neal (2006) proposed that the work environment and individual characteristics are key components to the success of training interventions. A review of training interventions history is presented to understand the importance of third generation learning and lay the foundation for enhancing the quality of peer feedback. First Generation Instructional Design The first generation of instructional design, which is the “analysis of performance requirements, the design of training content, and the selection of the training methods in order to train workers to do their jobs more effectively” (Kraiger, 2008, p. 454-455), has its grounding in the work of Tyler (1949) and Gagne (1962). Gagne synthesized much of his ideas from the research conducted between 1950-1960 in military bases that showed that current instructional design principles were not informing the performance outcomes for training as well as effective design of training. Gagne wrote: “If I were faced with the problem of improving training, I should not look for much help from the well-known learning principles such as reinforcement, distribution of practice, response familiarity… [Instead], I should look…at the technique of task analysis… the principles of component task achievement, intratask transfer, and the sequence of subtask learning to find those ideas of greatest usefulness in the design of training” (p.90). Gagne was emphasizing the importance of delivering effective instructional design through task performance analysis and structure of the design of the delivery of the training. From this research, Gagne (1965) published his book, The Conditions of Learning, in which he identified the mental conditions of learning. These were based on the information-processing model of mental events that happen when learners are presented with stimuli. Subsequently, he created a nine-step process called “The Events of Instruction” which address conditions of learning. They are as follows: (1) gain the attention of the learner through stimuli activation, (2) inform learners 6 of objectives of the learning to create expectations of the learning, (3) stimulate recall of prior learning to activate short term memory, (4) present the content for the learner, (5) provide learning guidance to encode for long term memory retrieval, (6) elicit performance and practice while answering questions related to learning objectives, (7) provide feedback for reinforcement, (8) assess performance, and finally (9) enhance retention through generalizing the learned skills to new situations. Through these, he tried to make these principles more relevant for impacting learning. Kraiger and Ford (2006) noted that these constitute the primary theoretical approach of most effective training. Of importance, this research lead to what researchers now refer to as the Instructional System Design (ISD) Model (Goldstein & Ford, 2002) which was developed in the 1970s (Goldstein, 1974). The model cascades across several popular training design models such as the Dick and Carey Systems Approach (Dick, Carey, & Carey, 2001), the Kemp Model (Morrison, Ross, & Kemp, 2004), and the Three Phase Design Model (Sims & Jones, 2003). These models all generally follow the ADDIE approach: analysis, design, development, implement, and evaluation (Dick, Carey, & Carey, 2001). As such, the need is first understood, and then the determination if training can alleviate the situation is decided. Following these steps, training objectives are developed, and training content is generated. After delivery of the training, evaluation of the training is given. The underlying assumptions of these models and designs is that the responsibility of treating the “problem”, which is a deficit in understanding, knowledge or behaviors necessary to complete a job, is in the instructor’s hands. The instructor defines the content, elicits performance, and gives the feedback on performance (Kraiger, 2008). Gagne (1962) even noted that the “first event” of instruction is making sure the learner is attending the training. This idea 7 in itself has been found to be problematic. Baldwin, Magjuka, and Loher (1991) discovered that denying a learner’s choice of training lowered pre-training motivation to learn and actually decreased learning transfer compared to those who either were allowed their choice or were not allowed to choose at all. Participants who were given the choice of training had the highest levels of motivation and learning transfer. Current instructional design models still emphasize the role of instructor (Kraiger, 2008). Goldstein and Ford (2002) discussed the qualities of a good instructor and the role of a quality instructor, which included providing feedback and making important design and practice decisions. Noe and Colquitt (2002) also emphasized the role of the instructor in providing feedback to the learner. Kraiger (2008) lumped this instructor-centric version of training together, calling it “first generation training,” and noted that most interventions between 1960-1990 are first generation: “The most popular models of instructional design, those that have driven training design for over 4 decades, place the learner in a passive role, tasked only with the absorbing that information identified through prior analysis” (p. 456). He notes that the organization takes too much liability for the learner, leading to ineffective outcomes. Second Generation Instructional Design A shift began in the late 1980s that placed more emphasis on learner-based approaches to learning. For example, Stevens and Collins (1980) proposed that learning happens when learners add, delve, and generalize components of an initial model, and then, using situated learning contexts help link new knowledge to relevant contents. From this perspective, the learner becomes more center stage. Kraiger (2008) noted that second generation learning is categorized by specific characteristics: (1) the task relevant models and skills are learner focused – not 8 instructor focused and (2) activities inspire exploration and “sense making” rather than linear training that followed principles. The instructor becomes more of a facilitator of learning, rather than a dictator of learning. Second generation learning theory relies on constructivism (see discussion below), which is consistent with adult learning theory. Knowles, Holton, and Swanson (2005) proposed six learning principles for adult learning: (1) recognizing the necessity to learn something before starting to learn it, (2) accepting responsibility for own decisions and learning, (3) recognizing learners enter a learning situation with previous work experience(s), (4) acknowledging that adults learn things eagerly, which they know and apply to real life situations, (5) staying life centered in their orientation to learning, and (6) responding to external motivators for learning. Using these principles in learning, the learner can “see education as a process of developing increased competence to achieve their full potential in life… [being] to apply whatever knowledge and skill they gain today to live more effectively tomorrow” (Knowles, 1980, pg. 43). Thus, the focus of second-generation learning is ensuring that the employees’ need to learn is addressed. This results in a higher emphasis on enhancing motivation, utilizing different backgrounds, and allowing for self-discovery. The difference between an approach to learning in first and second-generation learning is illustrated in this new-hire training example. In first generation learning, a training program identifies commonly performed tasks for a new hire and the critical skills to accomplish new hire tasks effectively. The goal is to enhance the new hire’s skills across the same set of tasks, regardless if he had prior knowledge or skills or if he did not need a certain set of skills within in the early part of his career. Thus, the organization takes responsibility for the new hire’s learning and skills. In second-generation learning, the new hire’s background and experience would be 9 considered before designing a training program for the new hire. Based on the experiences of the new hire and his needs for the job, skills that the new hire already had would not be trained; and likewise, skills that were not relevant to the first ninety days on the job would be removed. Thus, the focus is on the learner. The shift to second-generation learning has been accompanied by the surge of web-based instruction, which has been deemed as the “hypermedia based instructional program that utilizes the attributes of the World Wide Web to create meaningful learning environments where learning is fostered and supported” (Khan, 1997, p. 6). This shift has caused the rise of the “learner controlled” training environment. Learner controlled training creates a self-drive through various facets such as pace control, sequence of learning modules, the selection of the content modules, and ability to contribute through their own experiences (Kraiger & Jerden, 2007). However, the results of learner control have been relatively inconsistent and referred to as a “double edged sword” (Scheiter & Gerjets, 2007). For example, Kraiger and Jerden (2007) reported in a meta-analysis study that the effect size for learner control is small and noted that individual moderators influence learner control. Conversely, Fisher, Wasserman, and Orvis (2010) noted that when there was a high level of learner control in a training course participants reported higher satisfaction. However, there was no significant difference in learning outcomes. Some researchers have proposed ways to improve second-generation learning. For example, Bell and Kozlowski (2002) proposed adaptive guidance in computer based instruction to allow the learner to have some control in the situation while still being guided through diagnostic feedback. Others have proposed that learners need to grow their awareness of their own gaps in their learning (Williams, 1993). 10 Thus, second generation learning is a step forward in matching theory about learner centric principles with training design. However, second generation learning still focuses on the organization guiding the need for learning. Third generation learning is a unique evolution of learning and training, as it moves further towards a focus on peer-to-peer learning. Third Generation Instructional Design Third generation learning is a model that moves away from organizational implementation of feedback and learning towards peer construction of feedback and learning. The key characteristics of third generation learning are social negotiation and knowledge exchange (Kraiger, 2008). Third generation learning also relies on constructivism theory (discussed in the section) but places the individual much more at the social and cultural center of the learning process. Although first generation learning may be required for certain types of training such as factfinding, proponents of third generation learning contend that most learning happens through social negotiation. For example, a supervisor instructs what tasks a new employee should know through social negotiation, given the new hire’s previous knowledge and ability to demonstrate the needed skills without training. Lave and Wenger (1991) note that understanding norms and practices within a social context is of critical importance to move learning forward, which can be learned not only from the instructor but also by peers doing similar tasks. Similarly, cognitive constructivism theory confirms this notion that learners not only learn from instructors but also from other peers. Simply put, the “goal of instruction should be to create an interactive learning environment in which training participants learn from instructors, participants learn from each other, and the instructor learns from participants” (Kraiger, 2008, p. 461). 11 Thus, third generation learning is training within the context of social environments, which is well suited for online learning. Moore (1989) highlighted that there are three sources of interaction with learning: learner-to-content, learner-to-instructor, and learner-to-learner. First and second-generation learning have focused on learner-to-content and learner-to-instructor. Third generation highlights the learner-to-learner interactions. Learner-to-learner or peer-to-peer instruction is the “exchanges among training participants in an effort to extract meaning” (p. 461) through completing projects socially, preparing for assessments and examinations in social networks, and providing feedback to each other in social contexts (Kraiger, 2008). Peer-to-peer knowledge is not a new concept. As noted earlier, higher educational instructors have seen the value of peer feedback for some time in learning environments (Xiao, 2010; Sluijsmans, Brand-Gruwel, & Van Merriënbor, 2002). These types of interactions help learners interpret, clarify, and validated the training and learning they have experienced (Garrison, 1993). Peer-to-peer instruction and feedback as defined in third generation learning is most suitable for online training for several reasons (Arbaugh, 2008). First, because of the peer focus, the instructor has a decreased physical presence in the training. In an appropriate technologyenabled environment, learners do not actually need the instructor to be online and present in the course for learning to happen. Instead, peers can share knowledge, provide feedback, and improve their skills with or without the instructor. In many ways, the instructor acts as the facilitator. Second, there are multiple media formats available for the learner to engage and grow their skills and knowledge. Peers may meet face-to-face if they chose but they can also engage via email, chat rooms, file sharing, wikis, blogs, discussion boards, picture sharing, etc. 12 There are many benefits to this type of interaction to promote learning. First, this type of learning creates knowledge that is situated for the peer. Second, this type of interaction may translate into to a higher number of social and technical mentors (Bryant, 2005). Third, this type of learning may drive motivation to learn and develop skills. Fourth, learners process difficult information together, help others apply information in new ways, and construct knowledge with peers. Fifth, there may be less fear of making errors with other fellow peers rather than with an instructor. Finally, discussion boards and chat blogs remove participants’ titles and socially confined markers so that everyone feels as equal learners (Kraiger, 2008). However, the effectiveness of third generation learning is based on the relatively untested assumptions that all peers will know how to engage other peers. Vrasidas and McIsaac (1999) noted that there are certain conditions under which online learning will happen more effectively and not all technologies or learners are suited for online learning with social learning components. In fact, Bedwell and Salas (2008) question whether third generation learning is possible to execute because there is a need for the instructor to equip students with the skills to share knowledge and provide feedback. In summary, the effectiveness of third generation learning is relatively untested, yet has many promising ideas around the three sources of learning. One of the critical components that third generation learning relies on is the aspect of peer-to-peer interactions. These are cornerstone to third generation learning, and skeptics have been curious as to whether it can truly happen. While there are many aspects of peer-to-peer interactions that could be studied, the focus of giving peer-to-peer feedback is one that needs critical attention. Peer feedback lays the foundation for guiding thinking and helping learners construct their understanding of the content being engaged. Thus, this dissertation focuses on the aspect of peer feedback. 13 Theoretical Framework for Peer Feedback Aspects around feedback and assessment have focused heavily on the supervisor or the manager of the employee being evaluated. Thus, we turn to the educational literature to understand how peer feedback can influence the learner from a theoretical perspective. Researchers have contended that it is “very difficult” to articulate the underlying rationale for using peer feedback through one theoretical perspective (Xiao, 2010). Thus, there are several theories that can come into play when discussing peer feedback. Constructivism Third generation learning places much attention at the learner’s specific role in growing knowledge and building skills. This is consistent with the approach that educational theorists have recently uncovered as a driver for successful classrooms (Xiao, 2010). Piaget (1969) noted that a learner’s knowledge is specific to the culture in which he or she interacts; so much so, the learner can only absorb new knowledge by others within the same culture. Having a learnercentered approach to learning defines constructivism. Lave and Wenger (1991) discuss how constructivism places learners at the center of the learning process. The learner internalizes knowledge through the social and cultural context by interactions with others. Thus, constructivism may be in conflict with traditional forms of learning in which information is pushed upon individuals regardless of the context or environment. Consistent with situated cognition theory, learners acquire knowledge and learn new behaviors through interactions and practices within a culture (Lemke, 1997; Lave & Wenger, 1991). 14 Recent theorists who focus on constructivist perspectives state that the mind produces its own reality based on the events that occur in life rather than a tool to reproduce external reality (Jonassen, 1997). Learners analyze different interpretations of information that is presented and then form their own abstracts of reality (O’Loughlin, 1992). Analogous to cognitive maps in which learners acquire, code, store, recall, and decode events through psychological transformations (Richardson, Schifter & Simon, 1992; Toleman, 1948), learning is a “formation of abstract concepts to represent reality” (Leidner & Jarvenpaa, 1995, p. 267). Thus, the learnercentric approach is critical to the formation of this reality. From this perspective, individuals learn more effectively as they learn by experiencing knowledge for themselves versus being told or instructed. As such, instructors serve more as a mediator for learning progress. Educators are encouraged to use class time for projects where the learner can construct their own (Leidner & Jarvenpaa, 1995) and use journaling to understanding learning (Hawkins, 1993). In peer feedback, constructivism becomes a critical component of understanding how the learner engages with the feedback. Constructivism requires the learner to exchange, share, and negotiate knowledge while also drawing on expertise of more knowledgeable individuals (Liu, Lin, Chiu, & Yuan, 2001). In such a way, as a learner engages in third generation learning, a learner must not only receive information but also exchange the knowledge he has with other peers and experts. With all the exchanges of knowledge, the learner then draws his or her own conclusions on which information to bring to his or her learning environment. Cooperative Learning As an offspring of constructivist theory, cooperative learning (also called collaborative learning) contends that learning emerges through interaction with other individuals, rather than 15 objects in his or her environment (Slavin, 1990). Learning occurs as individuals “exercise, verify, solidify, and improve their mental models” (Leidner & Jarvenpaa, 1995). Because learners are improving their own mental models, their learning leads them to new understandings (Whipple, 1987). Compared to traditional forms of learning where training is viewed as linear, collaborative learning is seen as organically leading to new ideas. These new ideas are dependent on the interaction of the learner with other individuals. Flynn (1992) called this “branching” and “concentricity” as there is an implicit goal in cooperative learning that the learner provides communication, listening skills, and feedback during the learning process. There are several assumptions that cooperative learning relies upon. First, the more is shared the more is learned. If no one is sharing their knowledge, then there are no new interactions, which do not lead to new knowledge being created. Second, active participation is required for the learning to take place. Passive learners will not assist in the learning process. Third, ideal environments are necessary for the learning to occur. Researchers have suggested that smaller, intimate groups and settings are ideal so all learners are comfortable to contribute and share. Finally, learners should ensure that students promote each other’s success through assisting and encouraging each other. This relates to sharing of bad information, causing learners to suffer, which is why the role of the instructor can play an integral role in ensuring wrong information is thwarted if the group does not reject the false information first (Johnson, Johnson, & Smith, 1998). The role of the instructor is much different in cooperative learning. Instead of pushing information onto the learner, the instructor acts a facilitator among peers during their interactions and ensures the feedback received between peers maximizes the learning experience. Instructors 16 should give feedback to help facilitate the learning process but act as catalysts to enable peer-topeer feedback (Leidner & Jarvenpaa, 1995). Thus, an important component of cooperative learning is peer feedback. Peers should be directing their feedback to each other to facilitate the creation of new knowledge. The assumptions are the same – that is, that there is active participation and those ideal environments exist. Zones of Proximal Development Vygotsky (1971) introduced the zones of proximal development, which is the difference between what a learner can do without help and what a learner can do with help. Educators should teach children through experiences that are within certain development zones, which advance their own knowledge. One of the critical components of these zones is scaffolding. Scaffolding is the process in which a teacher or peer helps the learner in their appropriate zone. As the learner becomes more competent, the “scaffolding” is removed much like in construction of a building (Palincsar, 2003). Through this, learning is not a secluded activity in which the learner engages in learning on his or her own but is in fact reliant on social interaction (Vygotsky, 1971). Scaffolding acts much like peer feedback. While there is a role in which instructors play in scaffolding, interaction with other peers is critical. When a building is being constructed, there is not just one support – support comes from all around. In a similar way, third generation learning relies on not only instructor feedback but also the feedback from peers who are engaging in the content at the same time (Liu et al., 2001). 17 With several theories supporting the notion of third generation learning and the benefits of peer-to-peer interactions, the role of feedback in learning environments is now examined. As training environments rely on collaborative environments with peers providing feedback to support their learning “buildings,” the role of providing high quality feedback becomes imperative. To examine what makes quality feedback, an analysis of feedback theory and research is examined, focusing on dominant themes in the feedback literature. An Introduction to Feedback Theory and Research Latham and Locke (1991) noted that, “Few concepts in psychology have been written about more uncritically and incorrectly than that of feedback” (p. 224). Kluger and DeNisi (1996) noted that of 23,663 observations of feedback interventions on performance over one third of these interventions resulted in decreased performance while others have resulted in increased performance. These inconsistent results have been noted by others (Ilgen, et al., 1979; Balcazar, Hopkins, & Suarez, 1985-86). Given the broad implications of the feedback, it makes sense that feedback is also a critical component of learning. Thorndike (1927) provided evidence that feedback is an important component when it comes to learning, but the importance continues to be echoed (e.g. Gagne, Briggs, & Wager, 1992; Ilgen, Fisher, & Taylor, 1979; Komaki, Heinzmann, & Lawson, 1980; Trowbridge & Cason, 1932;). In fact, Gagne (e.g. Gagne & Driscoll, 1988; Gagne, Briggs, & Wager, 1992) noted that feedback is a critical component of instructional design. As such, the components necessary to improve the quality of feedback are now discussed to help drive the conceptual background for delivering high quality peer feedback. The importance of feedback can also be seen in the popularity of feedback research: 71% of all articles published in the 18 Journal of Organizational Behavior Management analyzed the effect of feedback in some manner (Nolan, Jarema, & Austin, 1999). Thus, to ensure peer feedback is of quality, the next section reviews the critical components of feedback. Before reviewing the components, there are two important considerations when reviewing the feedback research. First, there is a distinction needed between intrinsic and extrinsic feedback. Intrinsic feedback occurs naturally in the environment whereas extrinsic feedback is a specific intervention made by someone or something outside the training intervention. This dissertation focuses on feedback only inside training and learning environments. Second, feedback research has explored many venues to understand what drives high quality feedback. For example, Goldstein and Ford (2002) noted that the conditions of the task being trained with the expected learning outcomes are important; Smith-Jentsch, Zeisig, Acton, & McPherson (1998) concluded that there are differences when delivering feedback to individuals or teams; and presentation mode of the feedback as a critical component to high quality feedback was concluded by Baddeley and Logie (1999). Thus, the feedback research is broad and cannot all be covered in the following discussion. As a starting point to investigate the critical components of feedback, the seminal feedback model, Ilgen, Fisher, and Taylor (1979), was reviewed. These researchers proposed that the multidimensional nature of feedback influences the behavior of individuals and organizations because feedback is a “special case of the general communications process in which some sender… conveys a message to a recipient” (p. 350). The way the message is perceived and responded to depends on three primary components: (1) the feedback recipient’s characteristics, which includes the receiver’s motivation and their goals, (2) the nature of the message, which includes the timing, frequency and sign of the message, and (3) the source of the feedback, 19 which includes how trustworthy the source is and who the source is. These various components will ultimately lead to a response by the feedback recipient. Given this framework and the advancements made since the 1970s, the following is a review of overarching topics that feedback researchers investigate to drive an understanding of how to enhance the quality of peer feedback. The following themes are explored: (1) timing of the feedback, (2) the sign of the feedback, (3) the frequency of the feedback, (4) the specificity of the feedback, (5) the type of feedback, and, finally (6), the source of the feedback. Timing Labeled as one of the three dimensions of the feedback stimulus in the Ilgen model (1979), the impact of the timing of feedback delivery on valued outcomes is of continued debate among feedback researchers. Researchers have argued from a theoretical perspective that the consequences or rewards for behaviors should be linked to cues that are part of the environment (Mayer, 2001; Guthrie, 1935). Thus, timely feedback can be justified from a theoretical perspective. Unsurprisingly then, meta-analysis research has concluded that immediate feedback results in better post-test performance versus delayed feedback results. For example, BangertDrowns, Kulik, Kulik, and Morgan (1991) analyzed 58 effect sizes and noted that there was benefit for immediate feedback over delayed feedback for performance on achievement tests. Similarly, Azevedo and Bernard (1995) compared immediate feedback to delayed feedback in computer instruction training, finding that immediate feedback was superior resulting in an effect size of d=.8 for immediate versus d=.35 for delayed. There are many reasons why immediate feedback can be beneficial for the learner. One reason is illustrated by Anderson, Corbett, Koedinger, and Pelleteir (1995) who found that 20 immediate feedback assisted learners to stay on track during a computer based LISP computer language course. Similarly, Gibson (2000) noted that in dynamic learning environments where the learners participate, share, and collaborate with other learners, immediate feedback is much more critical. Corbett, Koedinger, and Anderson (1997) supported this idea when they recommended that feedback should be delivered in a timely manner to ensure that the information and context is relevant for the feedback to drive meaningful action. This approach is consistent with control theory which notes that only way to gauge progress toward the end state is through immediate feedback so that the learner can immediately change behavior aligned to the learners’ goals (Carver & Scheier, 1981; Lord & Hanges, 1987). Corbett, Anderson, and Patterson (1990) noted that the goal of the learner (as it relates to control theory) may be more quickly attained with immediate feedback. Because of the immediacy, the learner can self-adjust his behavior his behavior instantly. This type of result was found in football training, when researchers found that immediate, task specific feedback benefited the trainees better than delayed feedback (Rogers, Rousseau, & Fisk, 1999). Additionally, Anderson, Conrad, and Corbett (1989) found that because of this effect, students were able to proceed through their learning more quickly without negative effects to learning outcomes. However, delayed feedback may not always be less beneficial. For example, Kulik and Kulik (1988) noted cognitively demanding tasks favored immediate feedback whereas mundane tasks such as memorization favored delayed feedback. More dated research found that delayed feedback is superior to immediate feedback (e.g Kulhavy & Anderson, 1972; Markowitz & Renner, 1966). For example, Brackbill, Bravos, and Starr (1962) found that delayed feedback participants had higher post-test performance and retention scores compared to those who 21 received immediate feedback. Several researchers have hypothesized why delayed feedback is superior to immediate feedback. For example, Kulhavy and Anderson (1972) proposed the “preservation interference hypothesis,” which noted that the simultaneous occurrence of an incorrect behavior interferes with the ability to process the correct behavior given in the feedback. Similarly, Winstein and Schmidt (1990) stipulated that the results of Brackbill and colleagues can be best explained because delayed feedback is delivered with higher quality than more regular feedback. Others such as Chi, Bassok, Lewis, Reimann and Glaser (1989) theorized that immediate feedback prevented learning how to detect errors. While there may be some conflicting research on the timing of feedback, it is important to note that the research supporting delayed feedback is methodologically not as sound as the research that supports immediate feedback. Bolton (2006) outlined at least five primary reasons why research claiming delayed feedback is superior to immediate feedback is flawed: (1) the research occurred in unrealistic training settings, (2) there was unequal times during the trainings for immediate versus delayed feedback groups, (3) there was unequal feedback delivery time between immediate and delayed feedback groups, (4) there was inconsistent use of the post-test evaluations, and (5) most were not adult learners. Researchers have also categorized delayed and immediate feedback into other categories such as process versus outcome. Outcome feedback is feedback that indicates results and performance and it is typically given at the end of course or time period. Process feedback helps provide more diagnostic information about the performance of the individual throughout the course or time span (Earley, Northcraft, Lee, & Lituchy, 1990). It is critical in helping shape the future direction of the individual as the individual can receive feedback and alter behaviors 22 before the time is over. Process feedback is more informative than outcome feedback, especially when it is negative (Medvedeff, Gregory, & Levy, 2008). Thus, research on feedback shows greater support for timely, immediate feedback. As will be discussed later, the type of feedback or the topic of training may also interact the effect of timeliness of the feedback, especially in non-motor skill trainings (Bolton, 2006). Regardless, more timely feedback is consistent with control theory. Interestingly, however, the role of timely feedback becomes more complicated when considering the sign or content of the feedback message as being positive or negative in tone. The effect of the sign of the feedback is now discussed. Sign of Feedback Identified as a second dimension of the feedback stimulus that links the feedback giver and the reception of the feedback (Ilgen et al., 1979), the sign of the feedback refers to negative or positive relevance to attaining some goal (DeNisi & Kluger, 2000; Kluger & DeNisi, 1996). Much like the timeliness of feedback, the effect of sign as it relates to delivering quality feedback has been debated among researchers. Kluger and DeNisi (1996) noted that there was a great deal of variation in the effects of sign. While feedback sign has not been found to impact motivation (Kluger & DeNisi, 1996) in a consistent manner, Medvedeff et al. (2008) argued that feedback recipients’ motivation will be reinforced by positive feedback, resulting in enhanced future performance. Ilgen (1971) explored the effect of comparing positive feedback versus negative feedback. He found that participants who received positive feedback on a knowledge task had higher training utility. This type of result is consistent with current researchers. Vancouver and Tischner 23 (2004) theorized that positive feedback is consistent with the “goal of the positive self.” They noted that the fully formed self-concept is generally stable and generally positive, based on previous research which stated the desire to think positively about oneself is one of the strongest human motivators (Baumeister, 1998; Baumeister, 1999). Similarly, Steele (1975) found that, when there are threats to the self-concept, the individual often makes behavioral attempts to restore the positive self-concept. This can be done through reaffirmations of the self, establishing feelings of overall goodness, alleviating the discrepancy through unrelated events, etc. These self-concept discrepancies from negative feedback can result in lower performance. Vancouver and Tischner (2004) noted that negative feedback results in a shift of attention to the self, which may decrease task performance. The researchers found that overall feedback sign did relate positively to performance but only for those who had a cognitively intense task and concluded negative feedback has a positive impact on outcomes. In fact, DeNisi and Kluger (2000) suggested that those goals driven by what someone “ought” to possess (Higgins, 1997) based on the presence or absence of negative outcomes, are more suitable for negative feedback. Benefiting from negative feedback is consistent with control theory (Carver & Scheier, 1998; Medvedeff et al., 2008). Control theory notes that there is an actual state of behavior and then there is a goal for performance. Negative feedback, which is written in a critical way and productive way (not general negativity), can help the individual work towards that desired state. Medvedeff et al. (2008) found that negative feedback identified specific information that was used to change poor performance. Similarly, Steelman and Rutkowski (2004) noted that recipients of unfavorable feedback appreciated the negative feedback because it helped them identify how to improve performance. This is consistent with previous research which noted that 24 negative feedback can lead to increased effort to attain the desired state (Bandura & Cervone, 1983, 1986; Podsakoff & Farth, 1989). Negative feedback may have many beneficial outcomes beyond performance as well. For example, Podsakoff and Farth (1989) asked participants to list as many objects as possible that could be named based on an adjective given. Participants who received negative feedback set higher goals to perform better in the future versus those who received positive feedback on their performance. Similarly, Matsui, Okada and Kakuyama (1982) determined that participants who receive more negative feedback prompted learners to increase effort and performed at higher levels. Finally, feedback sign may be related to other important training variables, motivation and self-efficacy: Medvedeff et al. (2008) suggested positive feedback results in a betterment of self-efficacy, Idson and Higgins (2000) found that positive feedback resulted in positive outcomes. However, negative feedback may result in positive motivation (Matsui et al., 1982). A further discussion on the effect of sign as it relates to trainee characteristics is described in further detail below. In summary, the research is mixed on the sign of feedback, as there are many benefits for both types of feedback, although negative feedback allows for alignment with control theory research. The mixed results of specific signed feedback are similar to the results of frequency of feedback, which is now discussed. Frequency of Feedback Noted as a third dimension of the feedback stimulus that influences the extent to which the recipient can apply the feedback in the Ilgen model (1979), frequency of feedback is less 25 researched than the other facets of feedback. Leivo (2001) suggested that the lack of research on frequency of feedback may be related to a “common sense” belief that frequent feedback results in higher levels of performance (see Ilgen et al., 1979, p. 354-355). In fact, Alvero, Bucklin, and Austin (2001) and Balcazar, Hopkins, and Suarez (1985-86) both assumed frequent feedback was better than less frequent in the design of their study. Few comparisons have been made in the literature and the ones that have been made have lead to conflicting results (Kang, Oah, & Dickinson, 2003). Chhokar and Wallin (1984) asked participants to attend a six-week safety training program that consisted of a goal setting phase. Participants were divided into two groups: one was given feedback every week whereas the other was given feedback every other week. Participants in the courses had equal performance levels in demonstrating safety behaviors. Similar results were found by Leivo (2001) who gave employees feedback on the effectiveness of their housekeeping behaviors. Feedback was given once a month with some employees whereas the other employees received feedback every three months. The results revealed employees displayed the same level of housekeeping effectiveness. However, research has also found frequency is not always a moot point. For example, Alavosius and Sulzer-Azaroff (1990) divide participants into three groups during safety behavior training: no feedback, continuous/“dense” feedback (feedback given many times a day), and intermittent feedback (feedback given after approximately every three weeks). Results indicated that there was substantial improvement made after feedback was introduced, supporting the previously discussed proposition that immediate feedback is more desirable. Additionally, continuous feedback resulted in quicker learning of the behaviors needed versus the intermittent feedback. However, after mastery was obtained on the desired behaviors for both groups, 26 behavior was maintained equally over time in a performance follow up that occurred several months later. Researchers have suggested that frequency alone may not be a driver of feedback quality. Balcazar et al. (1985-1986) and Alvero et al. (2001) found that the majority of studies on feedback frequency used frequency in a combination with other procedures to drive performance. Kang, Oah, and Dickinson (2003) supported this suggestion when they placed frequency feedback in partnership with an incentive. In Kang et al.’s study, they not only found that participants who received feedback more frequently were more productive, but they also found that the results were stronger when there was an incentive involved. Their results suggest that feedback frequency may be dependent on the consequences of performance. Thus, feedback frequency is an important component of giving and receiving feedback, but it is a component that needs further research to determine if certain types of trainings are best suited with certain frequency rates. What may matter more than frequency is the helpfulness or specificity of the feedback. Specificity of the Feedback Mentioned as a major concept for feedback effectiveness in the Ilgen model (1979), feedback specificity is the “level of information presented in feedback messages” (Goodman & Wood, 2004, pg. 1), which transforms into how helpful the feedback is to the learner. Feedback researchers have determined that specific feedback is helpful in managerial skills such as developing coaching skills (e.g. Gomez-Mejia, Balkin, & Cardy, 2001; Kreitner & Kinicki, 2001) and performance appraisal skills (e.g. Cascio, 1988). 27 As feedback becomes more specific, there can be an increased focus on particular behaviors that need to be corrected or rewarded (Adams, 1987; Annett, 1969). Cognitively, feedback specificity decreases the amount of information that has to be processed such as error diagnosis, encoding, and retrieval (Schmidt, 1991). In other words, feedback specificity puts less work on the receiver of the feedback because the behaviors that need to be changed, altered, or rewarded are specifically prescribed. On the other hand, feedback specificity’s benefits of cognitive demands are not aligned with the guidance hypothesis (Salmoni, Schmidt, & Walter, 1984; Goodman & Wood, 2004). The guidance hypothesis notes that feedback high in specificity may not always result in higher learning. Yoder (2009) illustrates an example through a pupil of chess. In high specificity feedback environments, the trainer would watch the learner, and instruct the student how to play the game of chess in order to win the game. A trainer in a low specificity feedback environment would have a less active role in giving specific advice on how to play the game. Undoubtedly, if this were the early stage of learning how to play chess, the student in the high specificity environment would win the game. However, when the student played the game again, the student in the low specificity feedback situation may fair better because the student had generalized the knowledge through information processing. Goodman and Wood (2004) were concerned that feedback interventions impair learning (see Bjork, 1994; Schmidt & Bjork, 1992), that frequent, immediate feedback can undermine learning during transfer tasks (Goodman, 1998) and that feedback interventions that are less frequent and have delayed feedback lead to better learning, but not better practice (Goodman, 1998; Schooler & Anderson, 1990). Goodman and Wood summarized the complexity of feedback specificity when they noted, “Increasing feedback specificity is beneficial for learning 28 what to do when things are going well, yet detrimental to learning what to do when things are going poorly.” Thus, they examined the effects for themselves. The researchers found that feedback specificity results are dependent on what is being learned and these effects are mediated through opportunities to learn and respond to different conditions during the practice time. While increasing specificity of feedback during practice increased performance, the more specific feedback was beneficial in learning to respond to good performance whereas it was not beneficial for learning how to respond to poor performance. Goodman, Wood, and Chen (2011) followed up on their study and found similar results. They found that high specificity feedback enhanced performance of the participant. However, they did note that high specificity feedback was negatively related to exploration in the task-decision making process. The benefits of providing specific feedback are found in other studies. Ilgen et al. (1979) and Kluger and DeNisi (1996) both found that there was a positive relationship with feedback specificity and short-term performance. Kopelman (1986) speculated that specific feedback informs the individual how far away they are from reaching their specific goal. If the specific feedback can be detailed with strategies and more details on how to perform at peak, then the performance outcomes can be larger for specific feedback (Kluger & DeNisi, 1996). In summary, the specificity of feedback is debated, with some leaning towards providing high specificity feedback. However, these previous four variables may all be dependent on the type of feedback given, which is now covered. The Type of Feedback Feedback reduces the discrepancy between the current self and the desired self; so much so, Ilgen and colleagues (1979) noted “in all cases, feedback plays an essential role in the goal 29 process.” Quality feedback helps the learner achieve the desired state or outcome through the following questions: (1) Where am I going or what are the goals?, (2) How am I going?, and (3) Where to next or what activities need to be undertaken to make better progress? (Hattie & Timperley, 2007). Feedback givers have a critical role in answering the last two questions through the type of feedback they give(Locke & Latham, 1984; Sadler, 1989). Traditional feedback theorists typically focus on two primary types of feedback, directive and facilitative (Black & William, 1998). Directive feedback focuses on just what needs to be fixed, whereas facilitative feedback offers suggestions to help learners guide their own reconceptualization of the situation. Recent researchers however have suggested there are four types of feedback: (1) task-specific, (2) process-specific, (3) self-regulatory, and (4) personspecific (Hattie & Timperley, 2007). Task-specific feedback. Task-specific feedback informs the learner how well a particular task is performed. This feedback helps the learner understand correct answers and how to acquire certain information (Lysakowski & Walberg, 1982). An example would be, “You should include more information about this theory in your answer.” This is perhaps one of the most common types of feedback given, as Airasian (1997) reports that about 90% of teacher feedback is at this level. Researchers have found that this type of feedback is relatively effective in driving desired behavioral outcomes: Lysakowski and Walberg (1982) reported d = 1.13 for learning outcomes in their meta-analysis, Walberg (1982) reported d = .82 for school performance, and Tenenbaum and Goldring (1989) reported d = .74 in motor skill learning. However, researchers have cautioned that this type of feedback is typically delivered with person-specific feedback. For example, an instructor might say, “Good job! However, you should include more information 30 about this theory in your answer.” This type of interaction is ill advised because it mixes positive and negative feedback into one statement (Bennett & Kell, 1989). Under some circumstances, highly specific task-specific feedback can hinder the ability to generalize to other tasks (Thompson, 1998). For example, Kluger and DeNisi (1996) found that when there was specific information given about the details of a task, the cognitive processing levels needed to generalize to task accomplishment is depleted and results in failure to complete future tasks. Process-specific feedback. Process-specific feedback improves the process used to reach an end result or complete the desired task. Balzer, Doherty, and O’Connor (1989) noted that this type of feedback is focused on the environment relationships, the relations of the learner, and the connections between the environment and the learner. For example, an instructor may say, “The presentation you are giving might be more effective if you utilize the strategies and processes that you learned yesterday in your presentation class.” In such a way, the process is the focus. Researchers have suggested that process level feedback is more effective than taskspecific feedback because it enhances a deeper learning, especially when in the partnership with other learner characteristics (Balzer et al., 1989; Earley et al., 1990). For example, goal setting behaviors or feedback seeking orientation may enhance the outcomes of process-specific feedback. Earley et al. (1990) noted that process-specific feedback improves confidence and selfefficacy. Self-regulatory feedback. The third type of feedback focuses on the self-regulatory processes including self-evaluation and confidence to move forward on the task. Self-regulation is the “interplay between commitment, control, and confidence” (Hattie & Timplerley, 2007). During this interplay, learners actively monitor their progress towards their learning goal. As a 31 result, learners regulate action to reach their goal. Related to control theory, as an individual sets goals, his or her commitment and confidence to reach that goal is related to self-regulation. Zimmerman (2000) described self-regulation as the involvement of “self-generated thoughts, feelings, and actions that are planned and cyclically adapted to the attainment of personal goals.” Self-regulatory feedback can take many forms. There are at least six major components of self-regulation as it relates to quality feedback: (1) the ability to create internal feedback, (2) the ability to self-assess, (3) the willingness to invest effort in seeking and dealing with feedback information to achieve the goals, (4) the degree of certainty in the correctness in the feedback, (5) the internal diagnosis of task failure or success, and (6) the propensity to seek help (Hattie & Timperley, 2007). Hence, one example would be, “You already know how to deliver great visuals for your presentations. Make sure you’ve executed the visuals to the level you typically deliver in this presentation.” Feedback that activates self-regulation is powerful because it allows for deeper information processing and mastery of tasks. Butler and Winne (1995) argued that for selfregulation, “feedback is an inherent catalyst.” Because of this focus, self-regulatory feedback will encourage the students to continue on their task. Ashford and Cummings (1983) found that feedback is psychologically reaffirming as it helps learners reach their desired end state. Similarly, Kluger and DeNisi (1996) noted that feedback will be enhanced in its effectiveness if it can tap into self-regulatory power and drive action, which will also impact self-esteem. However, all this may be dependent on the feedback seeking orientation of the individual, which is discussed later. Person-specific feedback. The last type of feedback is person-specific feedback, which is more directed to the person. This type of feedback is not about the task, process, or learner 32 characteristics, and instead, is about the summative evaluations of the person. For example, an instructor would say, “You are such an active learner!” or “You are very intelligent in class.” This type of feedback includes the highly popular types of feedback such as, “Good job!” and “Way to go, Joe!” Nothing about their specific behaviors or knowledge is demonstrated (Brophy, 1981). Wilkinson (1981) found in his meta-analysis of teacher praise and person-specific feedback that there was little if any relationship to student achievement, resulting in an effect size of d = .12. Kluger and DeNisi (1998) reported a similar effect size, d = .09. In fact, Kluger and DeNisi found not giving praise at all resulted in much higher achievement, d = .34. Hattie and Timperley (2007) noted that this does not mean that all praise-oriented feedback is bad. For example, if the feedback giver says, “You are awesome because you have correctly applied the five factors of effective visuals in presentations in your 15 slides on the presentation given yesterday,” then the learner is understanding why he has achieved the desired end goal specifically. In summary, these four types of feedback offer feedback givers a framework for the possible results of their delivery. As noted, the type of feedback can interact with various learner characteristics such as self-efficacy and motivation. However, all these types of feedback may look different when it comes to the final variable discussed, the source, which lends itself to the essential component of third generation learning and the focus of this study. The Source of the Feedback According to Ilgen et al.’s (1979) model, the source of the feedback should influence the degree to which the content of the feedback is perceived and accepted by the target employees. 33 In fact, the “source may be the most important influence on the extent to which recipients accept their feedback” (p. 356). Because different sources of feedback inheritably hold different perspectives and characteristics, feedback receivers will evaluate and react to feedback accordingly. An overview of various sources of feedback is presented. Multi-source feedback. Multi-source feedback has been evaluated in the role of assessment such as 360-degree feedback situations. The 360-degree feedback gives perspectives from multiple sources including but not limited to supervisors, peers, direct reports, and external customers (Nowack, 1992, 2005). In general, this type of feedback helps the individual understand his or her level of skills as it relates to the job requirements (Tornow & London, 1998). Hazucha, Hezlett, and Schneider (1993) noted that 360-degree feedback may be viewed as more credible and result in greater behavioral change because it contain multiple opinions. However, results are inconsistent regarding the usefulness of 360-degree feedback. For example, Atwater, Waldman, Atwater and Carter (2000) reported that at least fifty percent of participants who received 360-degree feedback improved their performance whereas other studies report that receivers of 360-degree feedback have either no change or negative change (Ghorpade, 2000). The research on 360-degree feedback may be complicated by the nature of the feedback, as it is normally anonymous. Similarly, when 360-degree feedback collapses across all sources, it becomes more muddled as conflicting feedback cancels out each other. Regardless, 360-degree feedback research can yield important implications for peer-to-peer feedback, especially with concerns on leniency. This research is presented later. Manager feedback. The research on manager feedback is broad, but there are some insights that may inform feedback given by instructors or those who are considered “experts” when it comes to learning. A consistent theme in manager feedback is that relaying negative 34 feedback is something not enjoyed (Fisher, 1979). For example, both Benedict and Levine (1988) and Ilgen and Knowlton (1980) both found that supervisors avoid, delay, or distort negative feedback. Larson (1986) further detailed this finding when he discovered that managers tended to give feedback less during times of poor performance than compared to those times when an employee was meeting expectations. To further complicate matters, Larson also found that managers gave positive feedback even when the manager deserved negative feedback. Referred to as the “mum effect” (Tesser & Rosen, 1975), managers commonly do not deliver negative feedback because of personal fear of repercussion. Larson (1986) speculated that this is due to fear of rejection of negative feedback and projection of a negative image of the manager. Argyris (1991) found that employees of managers who gave negative feedback were perceived to be biased or insensitive. This research is interesting, given that feedback that comes from a credible source such as an expert on a job (e.g. manager) is more generally accepted (Ilgen et al., 1979; Larson, 1986; London & Smither, 1995). This credibility comes from the supervisor’s status as higher in organizational hierarchy, decision-making power, and perceived experience. Bloom and Hautaluoma (1987) illustrated this effect: when individuals received negative feedback from a credible source in their study, employees were more likely to make behavior changes compared to employees who received negative feedback from a less credible source. Similarly, Steelman and Rutkowski (2004) found source credibility to be one of the primary motivators to improve job performance when delivering feedback. Similarly, Steelman, Levy, and Snell (2004) emphasized source credibility when they included source credibility as a factor in a positive feedback environment. Given this focus on source credibility, it would make sense that 35 instructors would be seen as credible sources of feedback, thus making the feedback more effective and receptive by the receiver of the feedback. However, this is not always the case. Instructor feedback. The impact of instructor feedback on the student is “surprisingly” sparse and overly “complex” (Higgins, Hartley, & Skelton, 2002; Carless, 2006). However, recent literature may help instructors understand the role of feedback and learning. Ackerman and Gross (2010) report that students and instructors both agree that feedback is an important function in the classroom. Ramsden (2003) noted that the teacher’s ability to construct meaningful feedback on a student’s work is a critical component of effective teaching. Researchers have noted that students learn more efficiently when the instructor of the course is an active feedback giver (Hounsell, 2003). Feedback from instructors is generally perceived not only as a valued part of a class but also a critical component of learning. For example, Hyland (2000) reported that approximately 90% of students believed feedback from instructors was essential to identifying strengths and weaknesses for a student’s future work. So much so, it is an expectation for assignments in the classroom (Higgins et al., 2002), unless the students are experiencing academic performance issues (Jaasma & Koper, 1999). The impact of the instructor feedback on learning is complex, especially in the eyes of the student. For example, Higgins et al. (2002) found that feedback considered extremely useful by instructors was perceived as not useful by students. This was confirmed by Carless (2006). The range of reasons for why the feedback was seen as not useful ranged from being too general, too vague, too detailed, too technical, too biased, or too subjective. Additionally, beneficial or quality feedback from the instructor’s perspective was interpreted by students as being too harsh and discouraging. 36 Ackerman and Gross (2010) theorized that quantity of instructor feedback is a critical component of instructor-student feedback. They noted that students may interpret non-positive feedback as less “socially normative” and more reflective of instructor assessments of performance. With this, high levels of feedback will cause students to believe he or she is not performing well. The researchers found support for their theory in their study with students in an upper division marketing class. With manager and instructor feedback analyzed, the next section now turns its focus to the role of peer feedback in great detail. Peer Feedback As noted before, the source of feedback has been given considerable attention. And more recently, researchers have turned their attention to the role of peer feedback. Traditional feedback has focused on interactions between expert to learner, manager to employee, or instructor to student. The problem with the peer feedback literature is that it is mostly descriptive in nature. While researchers have been discussing the effects of peer feedback for over 25 years (Topping et al., 2000), there are few scientific approaches to evaluating its effectiveness. The following section highlights the unique aspects of peer feedback by (1) reviewing literature that falls to the major themes of feedback research highlighted above, (2) discussing the outcomes of peer feedback, (3) illustrating what research has been done to evaluate the trainee characteristics may be affected by peer feedback, (4) reviewing variables that affect the receptivity and outcomes of receiving peer feedback, and finally (5) discussing the additional complexities of peer feedback. 37 Five Tracks of Peer Feedback Timing of the feedback. The timing of feedback in the peer feedback literature uses different labels than traditional feedback. Instead of delayed and immediate feedback, peer feedback researchers have utilized the research done with formative versus summative feedback (Topping et al., 2000). Summative peer assessment focuses on accumulating feedback at the very end of a course or training session and providing the learner with all the feedback at end of the course. This type of feedback is consistent with “outcome feedback” versus process feedback. Formative feedback involves a wider audience, the other learners, key stakeholders (perhaps managers), and instructors. It focuses on improving the learning “while it’s happening to maximize success, rather than aiming to determine success or failure only after the event” (Topping et al., 2000). Formative feedback is similar to process feedback, which emphasizes the process rather than the outcome. Topping and Ehly (1998) suggested that formative feedback could have several impacts on the learner. Simply stated, “formative assessment seems likely to be more helpful if it yields rich and detailed qualitative feedback information about strengths and weakness, not merely a mark or a grade” (Topping et al., 2000, p. 150). As such, formative peer assessment can enable earlier error identification because formative peer assessment involves questioning through understanding. In such a way, as feedback is received and given, there is reorganization of the information given to the peer when the information is partnered with identification of gaps and explanations. Additionally, formative assessment may increase post hoc reflection. This is critical as post hoc reflection may help in the transferability of the new behavior or knowledge in new situations. Because formative feedback is immediate and individuated compared to other 38 forms of feedback, formative feedback has cognitive and meta-cognitive benefits, which can help with the post-transferring of the knowledge or behaviors. The sign of the feedback. The sign of peer feedback has been less researched but there is some research to suggest that the sign does interact with peer feedback. For example, Stone and colleagues (1984) examined the effects of feedback expertise and perceptions of feedback accuracy. They found that when the rater had low expertise, the feedback receiver does not judge the feedback as accurate. Thus, for a peer whose expertise may not be as high as an instructor, peers may not judge the feedback to be accurate, causing less action to be taken to bring he or she closer to the learning goals. Similarly, the results suggest that negative feedback is considered more credible after positive feedback is received. Of note, the researchers did not specifically say the low expertise feedback giver was a peer; instead, they noted that the low expertise rater was a graduate student in contrast to a Ph.D. in management. Another example of research related to sign and peer feedback relates to fear of giving negative feedback. Research suggests that peers are more likely to “over-mark” – that is, be more lenient – because they have anxiety giving fellow peers negative feedback (Cheng & Warren, 1999; Falchikov, 1995; Falchikov & Goldfinch, 2000; Orsmond, Merry, & Reiling, 1996). However, while negative feedback helps learners reach their desired end state as noted in control theory, Stone and colleagues (1984) found that positive feedback elevates self-perceived confidence. The frequency of feedback. The quality of peer feedback may be dependent on the frequency (Archer, Kent, & Mote, 1956; Christina & Bjork, 1991; Goodman, 1998; Schmidt, 1991; Schooler & Anderson, 1990; Winstein & Schmidt, 1990). For example, Lin, Liu, and Yuan (2001a) found many students did not improve until after two rounds of peer feedback in an on- 39 line peer assessment study. Tsai, Lin, and Yuan (2002) and others have found similar findings, but recent research has found the frequency at which the peer feedback has been given is significantly related to performance (Lin, Liu, & Yuan, 2001b; Tsai et al., 2002). It is important to note that these experimental designs included feedback requirements to evaluate feedback, which may have implications on the perceived value and credibility of the feedback (Falchikov & Goldfinch, 2000). The specificity of the feedback. Lin et al. (2001a) evaluated the differences between two different types of peer feedback, specific versus holistic. Using a web-based course, the researchers found that students with high executive thinking styles (those who adhere to existing rules and procedures) who received holistic and specific peer feedback and students with low executive thinking who received specific peer feedback improved their performance in the course. However, students who had a low executive thinking style who received holistic peer feedback did not improve their performance. Similarly, Xiao and Lucking (2008) looked at the type of peer feedback that can be given to various peers. They compared a numerical type of feedback such as performance ratings versus a qualitative feedback method. Using 232 preservice teachers, the researchers found that participants in the group that used numerical plus qualitative feedback showed greater training outcomes. Thus, the quality of information provided during feedback matters. The type of feedback. The research on the type of feedback in peer feedback research is relatively unexplored except as it pertains to over-marking and overly-praising feedback (e.g. Cheng & Warren, 1999; Falchikov, 1995). In general, because peers lack the skills necessary to deliver quality feedback (Sluijsmans et al., 2001), the type of feedback that peers typically give other peers is person-specific feedback. A purposeful intervention to ensure appropriate types of 40 feedback are delivered to fellow peers may be necessary to ensure that peers do not focus on the least effective type of feedback. Training Outcomes for Peer Feedback Researchers have placed the effects of peer feedback within categories (Xiao, 2010): performance as a learning outcome, training utility as a learning outcome, and the awareness of the learning process. Performance as a Learning Outcome. Peer feedback can be an important component to improving skill and performance (Xiao, 2010). For example, Richer (1992) analyzed the effects of peer feedback versus teacher based feedback on writing proficiency in a class of 87 students. The study showed significantly better writing proficiency for the peer feedback group compared to the teacher feedback group. Similarly, Plutsky and Wilson (2004) used a quasi-experimental study to compared three methods for teaching writing. The researchers found that peer feedback was beneficial to improving learning outcomes. The similar result has been found in language courses: Paulus (1999) found that peer feedback was as effective as instructor feedback. Chaundron (1983) reported that peer feedback was more cost effective than instructor feedback. However, the research on peer feedback has not been positive across all studies. For example, Birkeland (1986) compared three kinds of feedback: self, peer, and teacher feedback. In a class of 76 adult learners, the researchers found there were no significant differences between the teacher feedback group and the self-feedback group, between the self-feedback group and the peer feedback group, or between the teacher feedback and peer feedback group. This same effect was found 18 years later when Li and Steckelberg (2004) found that peer and 41 self-feedback on performance in an online web based class showed no significant difference on performance outcomes. Training Utility as a Learning Outcome. Learners’ attitudes towards receiving peer feedback are also positive. For example, Li and Stecklelberg (2004) reported that students had high levels of satisfaction towards the peer process. Similarly, Liu et al. noted that nearly 70% of participants preferred using peer feedback and viewed the peer-to-peer interactions as effective as the instructors’ feedback. Similarly, in the quasi experiment by Xiao and Lucking (2008), the researchers found that participants in the numeric plus qualitative feedback study reported higher levels of satisfaction with the course versus those who had just the numeric data. In addition, nearly 80% of all participants showed a positive attitude towards peer feedback and found it to be a beneficial activity. The results, however, are not universally consistent. Saito and Fujita (2004) researched the effects of attitudes towards peer-to-peer feedback among 61 business majors. The researchers found that peer feedback was not a significant predictor of student attitude towards peer assessment. The reasons for this may come from Topping et al. (2000) which showed students viewed peer feedback as time-consuming, intellectually challenging, and socially uncomfortable, especially if the group has familiarity. Chaundron (1983) used an experimental design to evaluate the quality of peer feedback versus teacher feedback in a course that was for English as a second language (ESL). Students were assessed on content, organization, vocabulary, language, and mechanics in a writing composition paper. In a training utility evaluation, students rated the student and teacher 42 feedback equally positive but found peer feedback to be more cost-effective, as instructor feedback required an investment of time of the paid instructor. It is important to note that these are student perceptions of accuracy; thus, students who purposefully gave inaccurate feedback would rate the feedback as accurate to deceive their fellow peers. However, there is some research to suggest that this accuracy exists beyond perceptions. Jacobs and Zhang (1989) found that, in an ESL class, teacher and peer feedback was evaluated as equal by a comparative analysis of what errors were found within the exercises. Awareness of the Learning Process. Making learners aware of the learning process during the training or learning intervention is an important and critical component of effective training as it makes the learner a more active participant in the training intervention. The majority of research that has looked at the learning process with peer interventions has been primarily investigated in writing situations. For example, Liu and colleagues (2001) evaluated the perceptions of students of peer feedback and assessment of third-year computer science majors. During their evaluations of the course, the participants not only mentioned that peer feedback was an effective method of learning, but they also noted that by reading other peer’s work, it allowed them to gain awareness of their own personal strengths and weaknesses. This is consistent with Robinson (1999) who suggested that when students are reviewing others thoughts, works, and opinions in a course – whether summatively or formatively – this process triggers the learner to reflect on his or her own opinions, work, and thoughts. Thus, the feedback process becomes an awareness trigger in itself. The processes that peer feedback triggers can also assist with the students’ abilities to understand course objectives and the criteria of the learning tasks. Because peer feedback has 43 implied elements of peer assessment, the act of negotiating and understanding why a learner received elements of feedback results in the learner becoming more aware of the learning objectives (Falchikov, 1995; Orsmond, Merry, & Reiling, 2000). For example, Guilford (2001) found that participants learned more about peer reviewing and technical skill writing in his undergraduate course than through traditional, instructor-led means of teaching such skills. Similarly, Venables and Summit (2003) discovered that learners found that the reading of others’ works in itself increased their personal interest in learning. The staff noted additionally that the feedback given from peers to other peers tended to be more detailed than that of the instructor. Trainee Characteristics Affected by Peer Feedback Researchers have noted that training effectiveness is grounded in the trainee’s ability to first master the information and then transfer it to the natural environment (Ford, Quiñones, Sego, & Sorra, 1992; Noe, 1986). The ability to master and transfer is affected by trainee characteristics. Topping (2000) suggested that peer feedback could affect trainee characteristics. For example, peer assessment increases motivation to learn because there is an “enhanced sense of ownership” and personal responsibility. The following section evaluates trainee characteristics that are affected by feedback given to the learner: self-efficacy and motivation. Self-Efficacy. Self-efficacy, the judgment of “one’s capability to organize and execute the courses of action require to produce given attainments” (Bandura, 1997, p. 3), has played a role in the area of training whether it be in computer training (Martocchio & Webster, 1992) or interpersonal skills (Gist, Stevens, & Bavetta, 1991). Translated to a training context, selfefficacy is a trainee’s beliefs on how well he or she is at the task being trained or how well he or she will learn the information. Messages such as feedback received before or during the training 44 – either positive or negative – will help trainees compare their performance to the established standard (Bandura, 1977). These messages will affect a trainee’s belief on how they will be able to perform on a task in the future (Bandura, 1991), and then these messages translate into the cognitive representation of different factors such as ability and motivation that will affect future performance (Bandura, 1986). Self-efficacy has been applied to all different aspects of work performance: higher selfefficacy has led to higher job performance (Stajkovic & Luthans, 1998), persistence in achieving a goal in the job has been underwritten by higher self-efficacy scores, and the choice of tasks has been mediated by self-efficacy (Gist & Mitchell, 1992). Students who are higher in self-efficacy outperform individuals with lower self-efficacy in nearly all types of performance tests. Additionally, these students had more persistence in the face of failure than those with lower self-efficacy (Schunk, 1989). More researchers are suggesting increasing self-efficacy should be one of the goals of training and more attention needs to be placed on how an individual arrives at their level of selfefficacy (Quiñones, 1995). In addition to Bandura (1982)’s four categories of determinates for self-efficacy (enactive mastery, vicarious experiences, verbal persuasion, and physiological arousal), Gist and Mitchell (1992) outlined three mechanisms by which a person can arrive to their pre-training self-efficacy. In the first step, an individual assesses the task and his or her relationship to the task. Secondly, the individual makes inferences about his or her past performance. Finally, the individual then evaluates the resources available to complete the task. With these three factors, the individual then arrives at his or her self-efficacy. Feedback is one mechanism in which a peer can help evaluate his current performance and thus form his or her self-efficacy. 45 Several researchers have analyzed the relationship between sign of feedback and selfefficacy. For example, Medvedeff et al. (2008) proposed self-efficacy would be reinforced by positive feedback, which would in turn result in better performance. Krueger and Dickson (1994) recorded self-efficacy levels when placed in negative and positive feedback situations. They found that self-efficacy was lower in the negative feedback situation. The effect of peer feedback on self-efficacy is less understood. Research has suggested, however, that peers typically do have differing opinions of themselves compared to their peers’ evaluation of them. For example, Yu and Murphy (1993) found that employees maintain favorable views of themselves compared to ratings provided by others for performance. The discrepancies between the self and the peer evaluation may have significant implications on the self-efficacy of the employee (London & Smither, 1995). Coupled with the propensity of peers to give elevated positive feedback, peer feedback may create a false sense of confidence, and may lead to higher self-efficacy. Stone and colleagues (1984) resonated a related thought: selfefficacy may be related to the rater’s credibility, especially in combination of other individual differences such as locus of control. Conversely, however, because peer feedback is less descriptive and falls into person specific feedback, self-esteem may be unaffected or be lowered by peer feedback because it does not relate to their desired end state (Kluger & DeNisi, 1996). Motivation. Much like self-efficacy, the trainee’s motivation has a significant impact on the participants’ ability to engage in the knowledge transfer activity. Noe (1986) noted that the motivation to learn is a “trainee’s specific desire to learn the content of the training program.” In an attempt to find a direct link between motivation and training outcomes, Facteau, Dobbins, Russell, Ladd, and Kudish (1995) found that trainees comprised of government managers with 46 high motivation to learn reported higher transfer than those who had lower motivation. Mathieu, Martineau, and Tannenbaum (1993) have also confirmed that trainees with a high motivation improved training effectiveness. In addition, pre-training motivation has been found to mediate other characteristics that would affect the training and transfer outcomes (Mathieu & Martineau, 1997), and that pre-training motivation resulted in higher levels of physical and academic selfefficacy (Tannenbaum, Mathieu, & Cannon, 1991). Additionally, researchers have suggested that motivation to learn is related to attitudes about the training. Martocchio (1992) suggested that a trainee’s pre-training attitudes predicted the ability to learn in a computer-training task such that participants who believed the training would benefit their learning had higher learning outcomes compared to those who did not see the training as beneficial. Similar to implicit personality theory (see Schneider, 1973), participants who are told that they are low on a certain skill may be less motivated to accelerate in a course because of these preconceived messages they have received about their abilities, which would affect the outcomes of the training intervention. In regards to feedback, motivation may be related to feedback received during the training or education session. Consistent with previous research on motivation, enhanced ownership and responsibility can lead to higher interest in the training. Van-Dijk and Kluger (2004) hypothesized that feedback interacts with regulatory focus to affect motivation, especially when sign comes into play. They suggested that negative feedback under prevention-focused individuals – those individuals who are sensitive to punishments that result from poor performance (Higgins, 1998) – elevates motivation. The researchers found support for their theory and also found that promotion focused individuals who received positive feedback had higher motivation. 47 As with self-efficacy, the sign of the feedback has been found to have an impact on motivation (e.g. Rutherford, Corbin & Chase, 1992). For example, Whitehead and Corbin (1991) looked at evaluative feedback in students on a novel fitness test. The researchers found participants in the negative signed feedback group reported lowered levels of motivation than those in the positive or no feedback condition. This is consistent with more recent research that noted that individuals who receive “success” feedback will persist longer on a task compared to those who received failure feedback, especially when partnered with process feedback versus task feedback (Medvedeff, Gregory, & Levy, 2008). Research as it pertains specifically to the effect of peer feedback on motivation is nearly nonexistent. However, researchers have proposed some preliminary ideas. Topping and colleagues (2000) suggests that formative peer feedback could have a motivating effect on individuals. Chhokar and Wallin (1984) found some preliminary research that supports Topping’s notion. When they looked at the feedback received from supervisors versus peers, they found the feedback from supervisors to be less motivating because it was used for personnel decisions or appraisal. However, participants perceived peer feedback to be more intrinsically motivating because it would be for the betterment of the team or workgroup. Similar to selfefficacy, if the peer feedback is non-person specific, employees may feel more motivated (Hattie & Timperley, 2007). Variables Impacting Receptivity, Acceptance, and Outcomes of Peer Feedback As noted in the previous section, training effectiveness is reliant on the trainee’s ability to transfer skills to his or her natural environment. There may be some unchangeable components in the natural environment that would affect the receptivity of the feedback. Similarly, although 48 under researched (Fedor, 1991), there may be some feedback traits of the learner that inhibit or enhance a learner’s ability to accept feedback. This section discusses two variables that affect the feedback received: feedback orientation and the feedback environment. Feedback Orientation. How receptive an individual is to feedback is an important consideration for evaluating feedback effectiveness. If an individual is resistant to feedback, then the feedback may be rejected and result in failure to act upon the feedback received. Feedback receptivity may be influenced by a variety of factors such as contextual variability, the feedback sign and the style of the feedback (Ryan, Brutus, Greguras, & Hakel, 2000). Ryan and colleagues (2000) proposed that there were at least five components of a receiver that would define their feedback receptivity: (1) self-awareness, one’s knowledge of strengths and weaknesses, (2) age of the feedback receiver, (3) demographic similarity between the giver and the receiver, (4) acquaintance, how familiar the two parties of feedback are, and (5) self-esteem, a person’s sense of self-worth. The researchers found partial support for all five of these components. Other researchers have investigated feedback receptivity as one overall trait within an individual that may contain many dimensions, referred to as a feedback orientation (Ashford & Cummings, 1983). Feedback orientation refers to an “individual’s overall receptivity to feedback” (London & Smither, 2002, pg. 81). Various researchers have defined this trait in similar ways. London and Smither (2002) proposed that feedback orientation includes six dimensions: (1) the general liking of feedback which includes a positive attitude towards feedback and less anxiety towards performance evaluation; (2) a behavioral propensity to actually seek out feedback; (3) a cognitive propensity to process feedback purposefully and consciously; (4) a public awareness of oneself in the view of others; (5) a belief that feedback offers insights not 49 found elsewhere that can cause individuals to become more effective and achieve desired end states; and, finally, (6) personal feeling of accountability as it relates to acting on feedback (see Ashford & Black, 1996; Ashford & Tsui, 1991; Bandura, 1982; Levy, Albright, Cawley, & Williams, 1995). Linderbaum and Levy (2010) provides similar dimensions: (1) defensiveness, an individual’s propensity to dislike negative feedback, (2) utility, an individual’s personal belief that feedback is useful in obtaining future end state outcomes, (3) accountability, an individual’s tendency to feel obligated to act upon feedback, (4) social awareness, an individual’s propensity to use feedback in social situations externally, and (5) feedback self-efficacy or individual’s competency level to interpret feedback correctly and then act upon it (see Ajzen & Fishbein, 1977; London & Smither, 2002; London, Smither, & Adsit, 1997, for further discussion of these components). This feedback scale was validated to show direct relationships to important components of other theoretically derived components such as self-monitoring and regulation. Regardless of the researcher’s definition, the dimensions of the feedback orientation are generally unchangeable. While an organization might be able to craft messages to drive change, an individual’s feedback orientation may inhibit or enhance the feedback messages. As Ilgen and colleagues note (1979), feedback must be accepted in order to change behaviors, which is why it is a critical to consider when evaluating the effect of peer feedback. Regardless of how high the quality of the feedback given, if the individual is not high on feedback orientation, there will be little effect of the feedback. Source is an important factor here as well. Given that peers will be receiving feedback from other peers, the feedback orientation may heighten or inhibit the effect of the peer feedback. 50 The Feedback Environment. The feedback environment can also impact the outcomes and processes of feedback. This relationship is similar to the one found in learning contexts, where the learning environment can influence training effectiveness (Eddy, Glad, Wilkins, Painter, Zavala, & Sedlacek,1967). Researchers have examined environmental factors such as choice of training (Baldwin, Magjuka, & Loher, 1991), organizational support (Tannenbaum, 1997), climate (Goldstein, 1986), managerial support (Cohen, 1990), and task opportunity (Ford, Quiñones, Sego, & Sorra, 1992). These work environment factors have direct or indirect outcomes on training such as improved learning transfer or increased self-efficacy (Tracey, Hinkin, Tannenbaum, & Mathieu, 2001). An organizational environment where individuals continually give and receive feedback – whether informal or formal – is important to improving job performance (London & Smither, 2002). Various conditions can foster this feedback environment such as performance management policies, continuous learning environments, and purposeful feedback systems. Cederblom (1982) noted that the more frequent the feedback, the more likely the feedback is to be accepted. A feedback environment can be created through various organizational practices (London & Smither, 2002). For example, organizations can enhance the quality of feedback through purposeful training of feedback, understanding performance baselines, and providing timely clear feedback. Alternatively, organizations can emphasize the importance of feedback. This involves the giving and receiving of feedback at all levels to understand behavioral performance standards and behaviors that change due to feedback are rewarded. London and Smither (1999) demonstrated that organizational support for feedback enhanced continuous learning. Other effects of a supportive feedback environment typically are implied from the research on learning environments. For example, Tracey, Tannenbaum, and 51 Kavanagh (1995) found that supervisor training was more likely to transfer to the job when the environment supported continuous learning. The need for organizational support of diagnostic feedback is also an important factor when looking at the effectiveness of peer feedback. If employees are not encouraged to give feedback on the job, peer feedback may feel awkward, forced, or may result in inflation, especially in the absence of training. In fact, Schelfhout, Dochy, and Janssens (2004) noted that for true learning environments to exist, the feedback culture must include feedback from the peers. Similarly, Saks and Haccount (2008) argue that the context may have more influence for third generation learning because of the importance of the level of organizational support for peer-to-peer interaction. Thus, if employees perceive a poor feedback environment, the quality of peer feedback may ultimately be diminished or the peer feedback may be dismissed as irrelevant. Summary of Peer Feedback In summary, there are many considerations when it comes to peer feedback. The frequency, sign, timeliness, type, and specificity of feedback are critical – if not more important – when a peer is delivering and receiving the feedback. As with other training and learning environments, trainee characteristics such as self-efficacy and motivation may enhance the outcomes and processes that happen during the learning. Finally, the feedback orientation and environment can influence the acceptance and quality of peer feedback. Thus, ensuring quality peer feedback and enhancing peer feedback environments become critical components in third generation learning. These issues coupled with evolving concerns on peer feedback emphasize the need for structured interventions for ensuring quality peer feedback. What follows is the research on other concerns about peer feedback. 52 Considerations for Peer Feedback & Peer Feedback Training There are some unique aspects of peer feedback that are typically not addressed in traditional feedback or training research. Centered on the reliability of peer feedback and skills necessary to give peer feedback, these two issues are discussed in detail in the section that follows. The secondary issue is where this dissertation adds its research contribution. Reliability of Peer Feedback While the adoption of peer feedback seems to be increasing in educational settings, researchers and educators are still concerned about the quality, validity, and reliability of peer feedback. Falchikov and Goldfinch (2000) summarizes these fears: “Fears of teachers about the lack of reliability or validity of peer…[feedback] may restrict its use and thus deprive many students of its learning benefits” (pg. 288). On the other hand, researchers such as Fedor and Bettenhausen (1989) believe that peer feedback holds psychometric superiority to other sources of feedback because peers have frequent and diverse contact with the targeted employee, which supervisors and instructors typically do not see. Thus, there is a need to understand if peer feedback is a reliable and valid method to improve learning. Topping (1998) defined validity and reliability of peer feedback. Validity of peer feedback refers to how strong the peer’s feedback and assessment correlate with feedback or assessment given by a professional or expert. Reliability refers to the consistency of the feedback and judgment statements by the peer to another peer that is in agreement across several peers about a level of performance of a fellow peer (Cho, Schunn, & Wilson, 2006; Topping, 1998). While some studies have looked at the relationship between peer and student feedback and have 53 claimed to investigate the reliability of peer feedback, almost all studies have only focused on the validity of peer feedback (Topping, 1998). Falchikov (1995) shed light on the validity of peer feedback. He asked his students to conduct presentations in a biological science course. The students jointly agreed to a composite checklist of assessment criteria for the presentation. Peer grading was done anonymously, but both students and staff gave feedback personally to the presenters. In their comparative analysis, the researchers found that aggregate peer scoring was similar to the instructors’ marks. Of note, peers and instructors seemed to vary on the presentation strengths. However, for weaknesses, the instructors and the peers were in close agreement. Since Falchikov (1995), few other studies have attempted to validate peer feedback. Topping and colleagues (2000)’s review attempted to understand what research had been done around this area. In this review, there were five studies that investigated the validity of peer feedback and assessment: Falchikov (1986), Stefani (1994), Mowl and Pain (1995), Cheng and Warren (1999), and Saito and Fujita (2004). All studies had between 48-67 undergraduate participants and validity scores were computed by averaging means of numerical peer assessment used for feedback. There was a wide range of results. For example, Mowl and Pain (1995) and Cheng and Warren (1999) reported lower validity scores with correlations between peer feedback and instructor feedback to be low (e.g. r = .22-.29). However, Saito and Fujita (2004) found much higher relationships (e.g. r = .72-.89). Other studies computed the validity coefficients differently, which resulted in different judgments. For example, Haaga (1993) used Pearson’s product moment correlation between pairs of students and graduate students which resulted in r = .55. 54 One of the most recent and comprehensive evaluations was done by Cho and colleagues (2006). They investigated the validity and reliability of peer feedback and assessment with 708 students across 16 courses over a three-year period in web-based environments. The classes varied in both undergraduate and graduate level instruction crossing multiple disciplines such as cognitive psychology, health psychology, leisure studies, and history. The results showed that there was a relatively strong relationship between instructor and peer feedback (r = .5-.7). There was no statistical support for graduate peer feedback to be of higher quality than undergraduate feedback, which could indicate that age or experience does not necessarily equip students of the skills necessary to give quality peer feedback. Reliability of peer feedback was high, r = .78, indicating a high level of agreement among peers. The researchers also found that the more peers in a group make the validity and reliability higher. Xiao and Lucking (2008) confirmed that finding, when they found that twenty peers resulted in higher reliability than three peers. Cho and colleagues (2006) method for evaluating the reliability of peer feedback is similar to the approach researchers have employed evaluating the reliability of 360-degree evaluations. This type of feedback tool stacks peer feedback alongside supervisor feedback. Demonstrated, Harris and Schaubroeck (1988) evaluated seventy students of mutli-source feedback systems and found that there was agreement between peers and supervisors (ρ = .62). Similarly, Conway and Huffcutt (1997) found a correlation between supervisor and peer ratings of r = .34. Finally, van Hooft and colleagues (2006) also found that there are certain dimensions that may lend to stronger inter-rater agreement than others. For example, human skills such as flexibility resulted in non-significant relationships between peer and supervisor ratings, whereas technical skills correlated above r= .3. However, they also noted that the internal construct 55 validity is not desirable, concluding that that “multi-source performance ratings reflect rater characteristics more than performance of the rates” (p. 78). While the results of 360-degree ratings inter-rater agreement may not be as high as peersupervisor feedback, there is significant agreement between peers and supervisor. More importantly, even though they did not conclude 360-degree assessments should be used for official evaluation methods, van Hooft and colleagues (2006) did note that disagreement among raters is “not problematic” when used for developmental purposes, which is what peer feedback would be used for, and researchers agree that training is needed prior to delivering 360-degree assessments (see Antonioni, 1996; Yukl & Lepsinger, 1995). Thus, it makes sense that training is needed for delivering peer feedback in third generation training environments. In summary, the inter-rater agreement among studies for comparing peer feedback to instructor feedback appears to be promising. Similarly, researchers investigating the reliability of multi-source ratings are finding significant relationships between peer and expert ratings. So much so, some researchers have suggested that practitioners “may be well advised to seek, and perhaps weight more heavily, information from peers” (Greguras & Robie, 1998, p. 964). These research findings on 360-degree feedback assessments give third generation learning instructors more cause to deliver peer feedback training prior to entering an environment that requires the delivery of quality and accurate feedback. Understanding How to Give Proper Feedback & Training Having the skills to give peer feedback is a critical component of delivering peer feedback. One of the common problems with peer feedback is the tendency to “over-mark” (Cheng & Warren, 1999; Falchikov, 1995; Falckikov & Goldfinch, 2000; Orsmond, Merry, & 56 Reiling,1996). Over-marking is being too lenient on your fellow peers. Examples of overmarking include the following: “Good Job!”, “Sounds good”, or “Nice.” It is positively worded feedback with nothing actionable to either reinforce a positive specific behavior or change an incorrect behavior. Reasons such as fear, discomfort, or anxiety are possible explanations of over-marking (e.g. Villanova, Bernardin, Dahmus, & Sims, 1993). Sluijsmans and colleagues (2001) suggested that over-marking could happen because peers lack the skills and abilities to discriminate between levels of performance. While studies that emphasize over-marking are traditionally in education literature, the tendency to over-mark – typically referred to as leniency in organizational performance research – has been in performance ratings assessment (Bernardin, Cooke, & Villanova, 2000). This effect is debated within the 360-degree, mutli-rater assessment feedback literature. Ng, Koh, Ang, Kennedy, and Chan (2011) highlight these “inconsistent findings” (p. 1034), noting that there may be just as many studies that find that peer ratings tend to be more lenient (e.g. Springer, 1953) as there are studies that show that peer ratings are not more lenient than other sources of ratings (e.g. Wohlers, Hall, & London, 1993). Ng and colleagues (2011) noted that these differences in studies did not account for the differences in power distance or individualcollectivist orientations. With those taken into account, peers do exhibit higher rates of leniency compared to subordinates, especially with individuals who are highly collectivist who wish to avoid damaging a relationship. Given, the discrepancies in ratings, the researchers call for practitioners to consider the culture of the organization and to require raters to attend training on how to rate properly. Similar to a call to action by Ng and colleagues (2011), other researchers have found that peers lack the skills and/or motivation they need to be honest to other peers when making 57 feedback statements in light of personal preferences, gender bias, racial stereotypes, and personal relationships (Carson & Nelson, 1996; Ghorpade & Lackritz, 2001; MacLeod, 1999; Nilson, 2003; Zhao, 1998). As such, there can even be hostility towards peer feedback because the peers being evaluated believe their feedback is in itself not accurate because it came from a peer that may have perceived biases or pre-judgments (Hanrahan & Isaacs, 2001). This becomes even more troublesome in the face of friendships. MacLeod (1999) and Schaffer (1996) both reported that students who had to give feedback to classmates who were also friends found it extremely difficult to deliver negative feedback because they thought it would damage their relationship. Zhao (1998) echoed social familiarity concerns when the researcher discovered that faceto-face peer feedback caused anxiety and fear of social rejection by fellow peers who were delivering negative feedback. This type of anxiety may lead to incorrect feedback. For example, MacLeod (1999) found that students who gave face-to-face peer feedback provided less accurate feedback because of interpersonal relationships. MacLeod noted that students who used computers or non-personal means gave feedback that was more honest than face-to-face feedback. Researchers have suggested that less familiar colleagues in peer feedback settings may be more appropriate. Social familiarity is a central issue of many researchers looking at whether peers have the proper skills to deliver feedback to fellow peers. Falchikov (1995) noted that social embarrassment could be an issue when it came to peer feedback in small groups of peers who were highly familiar with each other. From this perspective, peer feedback may be best when the group is large, with relatively unfamiliar contacts. In Falchikov (1995)’s experiment, the researcher stipulated that part of his success of the results of peer feedback was the relative lack of experience with each other. 58 Researchers have studied ways to help peers make better peer judgments. For example, Falckikov and Goldfinch (2000) suggested that peers are not be equipped with the dimensions of understanding to provide useful feedback. Hence, some type of purposeful training may be required for peers to deliver high quality peer feedback. Other researchers have attempted to fix deficiencies in skills and understanding through making peer feedback anonymous, which may help increase critical feedback. Similar to high specificity feedback (see above), researchers in peer feedback refer to critical feedback as feedback that assists the peer actually improve and identifies specific behaviors and examples of failure or success and helps improve performance (Lu & Bol, 2007; Zhu; 1995). To illustrate this, Lu and Boi (2007) conducted an experiment in which ninety-two college students gave critical feedback to other peers over two semesters. In one condition, students were given identifiable feedback where the peer knew who wrote the feedback and in the other condition, the feedback was completely anonymous. The researchers found that students in the anonymous peer review received significantly more critical feedback than those in the identifiable peer feedback group. Consequently, the performance of the students in the anonymous peer feedback group was much better than those in the identifiable feedback group. This study is consistent with Zhu (1995) who argued that peer feedback must be critical feedback to drive actionable and meaningful behaviors. Similarly, Kerr, Park, and Domazlicky (1995) found that those who are more critical of others’ works are more critical of their own work. Framework for Improving Feedback in Learning Environments Researchers have made a call to improve cooperative learning, (e.g. Sluijsmans et al., 2001, Sluijsmans, et al., 2002, Sluijsmans, et al., 2004). Training researchers have discovered quality feedback is a driver of higher training outcomes (e.g. Goodman, 1998; Schmidt, 1991) 59 and can be one of the ways to improve cooperative learning environments such as third generation learning. If trainers can unleash the power of quality feedback in training environments, training can be much more effective and have a higher return on investment (Black & William, 1998). Research presented pointed out at least five major factors that lead to more quality feedback: timely feedback, mixed signs, specific feedback, less person specific feedback, more frequent feedback. Nicol and Macfarlane-Dick (2006) argued that good feedback when focused on the learner should ultimately “help clarify what good performance is” and “deliver high quality information to students about their learning” (pg. 205). Thus, researchers such as Boud (2000) and Yorke (2003) have made the definitive call to focus “much more effort” into improving the quality of feedback messages. Kraiger (2008) and Bedwell and Salas (2008) note that the assumptions of third generation learning rely on the peer’s ability to exchange knowledge and give feedback to other students effectively. Few studies have been conducted to evaluate if the skills naturally exist in learners and/or if training would improve the effectiveness of peer-to-peer interactions. The results of studies from the education field indicate that training has potential to improve the quality of feedback to drive action (see Falckikov, 1995; Orsmond et al., 1996; Topping, 1998; Xiao & Lucking, 2008). However, the majority of literature on peer assessment has focused on negotiating about performance criteria (Falckikov, 1995; Mehrens, Popham, & Ryan, 1998; Orsmond et al., 1996, 2000). While this is an important skill, there are other skills (Xiao, 2010; Sluijsmans, et al., 2002) such as defining assessment criteria for delivering constructive feedback, ascertaining the 60 performance of the peers (what are the strengths and what are the weaknesses), and providing and delivering the feedback in the most productive, corrective way. Zhu (1994) conducted an experiment on the effect of peer training for peer feedback. In a university course with 169 participants, there were teacher student conferences in which the teacher trained students on how to develop and conduct peer feedback for revisions of papers. Because of the training, peers gave more higher quality feedback, resulting in enhanced attitudes towards peer revision and ultimately engaged peer revision tasks more actively. However, the writing quality of both groups (the group that received training versus the group that did not receive training) was statistically the same. In a similar study, Sluijsmans and colleagues (2002) used ninety-three second-year preservice teachers to investigate the effects of peer evaluation training. The training focused on defining criteria for performance, giving quality feedback, and writing feedback reports. The researchers found that those who were in the training were more likely to use the feedback and give more constructive feedback than those who were not in the training. Additionally, treatment condition participants received higher scores on their outcome indicators – in this case, lesson planning. Finally, the participants in the training had higher training utility to the control group. Sluijsmans and colleagues (2004) used ninety-three pre-service teachers to evaluate peer assessment training for learning. The results of the training showed that those teachers who received the training were more capable in evaluating peers using criteria than those who did not participate in the training, but there was no difference in outcome performance scores between the two groups. However, of note, there was little emphasis on constructing quality feedback within the intervention. 61 In summary, few studies have been able to capture effective interventions on how to improve feedback inside the training environment. These studies emphasize the importance of training; yet they also reveal the mixed results that peer feedback training can evoke on key learning outcomes. Ensuring that the feedback is of high quality is of utmost importance. Peers may be hesitant to provide proper feedback and provide vague person-specific feedback. Without quality feedback, learners inside third generation learning environments will struggle to grow and develop. Thus, this study focused on improving peer feedback during learning through a peer feedback intervention, utilizing traditional feedback research to help participants understand how, when, and why to give productive, constructive feedback. Framework for Current Study The Theoretical and Practical Need for This Study The current study attempts to address three significant gaps in the literature. First, researchers have been consistently calling for trainers to enhance and improve the ability to give quality peer feedback (Fallows & Chandramohan, 2001; Hanrahan & Issacs, 2001; Sluijsmans et al., 2004). Yet, one of the most cited or referenced researchers on peer assessment training, Sluijsmans and colleagues (2001, 2004) focuses primarily on ability for peers to judge the peer, and neglects to inform peers about how to give appropriate feedback based on sign, frequency, specificity, type, and timeliness. Thus, rigorous and purposeful interventions grounded in the decades of feedback training are an important direction for moving peer feedback research forward. Second, while there is a significant amount of research on the effect of feedback from managers or the organizations, there is limited data on the extent to which peer feedback can 62 impact learning within organizational training. In addition, we need studies that focus on potential mediating factors such as self-efficacy and motivation to learn between feedback provided and learning outcomes (e.g. London & Smither, 1995). Finally, the research on peer-to-peer feedback has barely begun understanding the importance of environmental factors and individual traits that can affect training outcomes. More specifically, what is the impact of organizational support and feedback orientation on the quality of peer feedback and the acceptance of peer feedback by the learner? Coupled with the complexities of peer feedback, feedback environments are important to analyze as organizations begin to adapt peer-to-peer feedback training interventions (e.g. Saks & Haccount, 2008). A Conceptual Model Given the reliance on peer feedback and peer-to-peer interaction for third generation learning, it is important to understand how peer feedback impacts learning outcomes. Kraiger (2008) has proposed third generation learning will create superior learning outcomes. This study examines one facet of this third generation learning model, the effect of peer-to-peer feedback. Many feedback-training programs for peers have resulted in minimal results, at best, and have not resulted in the desired outcomes. Thus, this study proposes a specific, unique training intervention based on feedback research, shown in Figure 1. This study examines how a purposeful, newly devised intervention to increase quality of peer feedback based on feedback components may improve the quality of peer-to-peer feedback, impact trainee motivation and confidence, and ultimately learning outcomes in a workplacetraining program. As such, the model proposed is divided into four parts. First, the model begins with the manipulation, which is a specialized training on how to improve quality of peer 63 feedback in third generation learning environments. It is proposed that the training intervention will impact the quality of feedback, and this intervention will then interact with the employee’s feedback environment, which will impact the quality of the peer feedback. Second, the peer feedback training leads to more quality peer feedback based on issues of specificity, frequency, timeliness, sign, and type. Third, quality peer feedback will lead to higher levels of desired trainee characteristics that affect training outcomes; high quality peer feedback will lead to higher self-efficacy and higher motivation. However, an individual’s feedback orientation may interact with these, as individuals who are lower on feedback orientation may fail to value peer feedback. Finally, training outcomes will be affected by elevated motivation and self-efficacy that occur due to more quality peer feedback. Training outcomes will be evaluated through feedback utility, training utility, knowledge outcomes, and behavioral outcomes. The study makes four assumptions. First, it assumes other aspects of third generation learning are not being evaluated (i.e., the interaction between instructor to the students, the interaction between students to the instructor, etc.). The effects of instructors on students have been documented (e.g. Carless, 2006; Higgins, Hartley, & Skelton, 2002). Second, like Sluijsmans and colleagues (2004), this study assumes that the training of feedback skills will have a positive effect on the skills and behaviors being developed in the course if the training is conducive to the feedback training intervention. Sluijsmans et al. noted that the peer feedback training should be embedded in existing course material so that the students will learn how to grow their skills through the course. While Sluijsmans went as far as to specifically not train on peer feedback uniquely in a specific module, this study begins the training session with the feedback training manipulation and then ensures the course content provides ample behavioral demonstration of those skills learned. Thus, this study assumes a 64 purposeful manipulation of peer feedback training prior to the training session will be more effective than building it into the course. Third, the presence of multiple sources of feedback is a current reality for organizations. The complexities of receiving multiple sources of feedback can be seen when analyzing the understood outcomes of 360-degree feedback (e.g. Atwater, Waldman, Atwater, & Carter, 2000). Because there is a need to understand the dynamics and effects of peer-to-peer feedback, the study focuses on the directive feedback between two peers. After understanding the effect of peer feedback training in this specific setting, the focus can move to multiple peer feedback. Finally, in keeping in alignment with current research, the study acknowledges that differences in training content may result in different effects of the peer feedback training. Given third generation learning is most conducive for non-motor skills (Kraiger, 2008), the study uses a management skill training for its focal subject. Additionally, the study focused on online learning environments only, which is the typical environment for third generation learning to happen (Kraiger, 2008). Given that online learning has a diminished presence of the instructor, peers are hypothesized to be able to exchange feedback and knowledge more freely. This study focuses primarily on the results of peer feedback, which is seen through the feedback recipient. While the feedback giver is a critical component of the model, to deliver the results of feedback giving, the study focuses primarily on outcomes through the feedback recipient with the exception of the feedback environment, which may interact with the effectiveness of the manipulation. As demonstrated in the figure, participants are placed into a training course and then selected to participate in the control (non-peer feedback training) or manipulation group (peer feedback training). The peer feedback training engages behaviors and knowledge around timeliness, sign, type, specificity, and frequency. As a result of the 65 manipulation, the quality of the feedback will be affected. The quality of the feedback will affect the feedback receiver’s motivation and self-efficacy, which may interact with the receiver’s feedback orientation to enhance or inhibit the receptivity of the feedback. Receiver motivation and self-efficacy will directly affect the training outcomes and will mediate the effects of the peer feedback training on training outcomes. Similarly, peer feedback training will also increase training outcomes directly: a behavior outcome, training utility, feedback utility, and a knowledge outcome. 66 HYPOTHESES The hypotheses are presented in two major groups: those that affect the feedback-giver and those that affect the feedback-receiver. The following section starts with the feedback-giver hypothesis and then moves into feedback-receiver. Feedback-Giver Hypotheses Feedback Quality A considerable amount of research has suggested that high quality feedback can lead to higher transfer outcomes (Nicol & Macfarlane-Dick, 2006). For example, Black and William (1998) completed a meta-analysis of over 250 studies on feedback that had been done since 1988 in the educational domain, which included high fidelity teaching situations and a variety of assessment outcomes. The results revealed that feedback resulted in higher learning outcomes across all outcomes regardless of skill type and education level. Sadler (1989) noted that there are necessary components for students to benefit from learning interventions. First, the student must know what good performance is, how current performance relates to good performance, and finally how to close the gap between current and good performance. In the context of peer-to-peer learning environments, peers possess the ability to influence all three of those requirements. However, as Sadler notes, the feedback-giver must provide quality feedback messages. Researchers have focused on various elements of feedback. Nicol and Macfarlane-Dick (2006) noted that high quality feedback helps the student (1) clarify what good performance is, (2) facilitates the development of reflection in learning, (3) delivers high quality information about their performance, (4) encourages dialogues around the learning experience, (5) 67 encourages motivation and confidence, (6) provides opportunities to close the gap to the desired end goal, and (7) provides information to the instructor to shape the learning experience. The current study proposes that quality feedback from peers will operate by the following principles to enhance the learning experience. Rationale for the principles is illustrated for each principle in the subsequent sections. The principles are: 1. 2. 3. 4. 5. Feedback will be delivered immediately following a learning exercise. Feedback will not always be positive. Feedback will happen regularly. Feedback will be as specific as possible. Feedback will help the learner achieve their learning goal. Feedback will be delivered immediately following a learning exercise. Timeliness concerns the delayed or immediate feedback given after a learner attempts a learned behavior or skill. Researchers such as Anderson, Corbett, Koedinger, and Pelletier (1995) find support to deliver immediate feedback. In learning environments, immediate feedback is even more important, as it facilitates learning (Reid & Parsons, 1996). Brosvic, Epistein, Dihoff, and Cook (2006) placed participants into five different laboratory examinations: no feedback, Scantron form feedback, delayed feedback, end of the test feedback, 24 hour delay feedback, and immediate feedback. Participants completed a 100-item cumulative final exam, which included ten items from each laboratory examination plus fifty new items. The researchers found that immediately delivered feedback significantly enhanced the results of the cumulative final exam. Other researchers have recommended the immediacy of feedback in learning contexts. In their fifty-three-study meta-analysis, Kulik and Kulik (1988) found that, although delayed feedback was superior in laboratory studies, immediate feedback was better in studies conducted 68 in actual classrooms. Butler, Karpicke, and Roediger (2007) also argued that delayed feedback may also reduce the level of attention a learner has as compared to immediate feedback. The empirical research specific to peer feedback is too sparse to draw definitive conclusions on the effectiveness of delayed to immediate feedback. However, Topping and colleagues (2000) noted that there are two types of peer feedback: formative feedback, which is more continuous feedback, and summative feedback, which delivers summative, post-experience feedback. Early researchers have found that formative feedback is more effective for peer-topeer feedback (Topping, et al., 2000). Feedback will not be always positive. Kluger and DeNisi (1996) emphasized that there are varied results of signed feedback. While some studies yield beneficial results of positive feedback (e.g. Ilgen, 1971), other studies have shown that negative feedback to be more beneficial (Steelman & Rutkowski, 2004). Additionally, feedback sign can influence a variety of other factors such as goal setting, effort, self-efficacy, and motivation (see Idson & Higgins, 2000; Matsui, Okada, &Kakuyama, 1982; Medvedeff et al., 2008; Podsakoff & Farth, 1989). In a learning context, feedback is a critical component to ensure learners meet their end goals (e.g. Bandura, 1997; Locke & Latham, 2002). Negative feedback assists the learning process (e.g. Brown, 2001; Cron, Slocum, VandeWalle, & Fu, 2005) because negative feedback reduces the discrepancy between the current state and the end goal (Mansell, 2005), whereas positive feedback only helps the learner stay at the current form. Given that peer feedback results in “over-marking” (e.g. Cheng & Warren, 1999; Falchikov, 1995; Falckikov & Goldfinch, 2000) or inflated feedback due the nature of the source (e.g. Ng et al., 2011), peer feedback training may have some of the most important impact on sign balance. For those who receive peer feedback training, there will be more negative feedback 69 given than those who do not receive peer feedback training because the natural tendency of peers will be to over-mark. Feedback will happen regularly. Feedback should happen on a regular basis. It is generally believed that the more frequent the feedback, the better (Ilgen, Fisher, & Taylor, 1979) in learning environments. Researchers such as Alavosius and Sulzer-Azaroff (1990) found support for more frequent feedback during learning. Researchers who have found that frequent feedback diminishes performance suggest that in these cases the feedback itself served as a distraction from the task at hand (DeNisi & Kluger, 2000; Kluger & DeNisi, 2005). Since the training is already third generation enabled, feedback is designed to be part of the training environment and thus should not act as a distraction (see Salvati, Gosselin, & Morin, 2003). Interestingly, frequency may be more important when coupled with other effects (e.g. Kang, Oah, & Dickinson, 2003). One of those factors could be in the presence of peers. Frequency of peer feedback has received more attention than other feedback variables. Lin et al. (2001a) found little impact of peer feedback influencing results of training outcomes when there were only two rounds of peer feedback. However, Tsai et al. (2002) found that more frequent feedback allows for the peers to evaluate source credibility and thus higher feedback frequency from peer feedback may be required for peers to react and alter behaviors. Feedback will be as specific as possible. The notion that high specificity feedback is beneficial for the learner is consistent with control theory (Kopelman, 1986). For example, Goodman, Wood, and Hendrick (2004) found highly specific feedback was better for training environments compared to on the job feedback; and Goodman et al. (2011) found that high specificity feedback is good for enhancing performance and highly specific feedback has been a recommended practice for learning (Larson, 1998). This principle is a close partner of immediate 70 feedback, as specific feedback is more optimal when it is delivered as immediate as possible (Goodman et al., 2004). Known as “microtraining” (Daniels, 1997; Ivey, 1971), supervisory training that focuses on specific feedback has lead to higher outcomes. In this model, learners are given a specific skill to work on and spend a significant amount of time practicing the skill. Trainees are given very detailed feedback on how to improve their performance. Miller, Morrill, and Uhleman (1970) noted that this high level of specificity takes the mystery out of learning, and reduces anxiety because they know exactly how to perform better in the future. In the domain of peer feedback, there is research to suggest that high specificity feedback could be beneficial. For example, Lin et al. (2001a) found that students with particular qualities (high executive thinking) benefited from specific feedback and thus improved their performance in the course. Similarly, Xia and Lucking (2008) found that learners who receive more descriptive peer feedback such as numeric plus qualitative feedback showed greater training outcomes. However, peers will not generally exhibit this behavior (Liu et al., 2001) and thus the training is needed to deliver specific feedback. Feedback will help the learner achieve their learning goal. As learners attempt to move themselves closer to their ultimate end state, effective learners utilize quality feedback to reach that end state (Nicole & Macfarlane-Dick, 2006). However, the type of feedback given affects a learner’s ability to reach the desired end state. Hattie and Timperely (2007) determined that there were four major types of feedback. First, task-specific feedback focuses on delivering tailored and specific information regarding the task at hand. The second type, process specific feedback, emphasizes the process used to get to the end result. Self-regulatory feedback, the third type, 71 focuses on self-evaluation and confidence to move forward on a task. Finally, person-specific feedback focuses less on the task and more on the person. Each type of feedback affects effectiveness of feedback differently. For example, taskspecific feedback has a variety of positive outcomes: reducing cognitive load (Sweller, Van Merriënboer, & Paas, 1998), assisting novice learners in grasping information in complex problem solving (Moreno, 2004), and or correcting procedural errors (Ilgen et al., 1979). Likewise, process-specific feedback can help with deeper learning (Balzer et al., 1989), and selfregulatory feedback can assist in other related processes such as help seeking. However, personspecific feedback has much less positive outcomes, as the majority of it cannot help the learner reach the desired end state (Hattie & Timperley, 2007). As it relates to peer feedback, there is little research to suggest that peers are effective at delivering any type other than person-specific feedback. For example, Papinczak, Young, Groves, and Haynes (2007) found, that while the majority of peer feedback was very well received, it very difficult for peers to give specific feedback about the task at hand (task-specific feedback). However, there is some evidence that process-specific feedback may be provided by peers. Guilford (2001) found that the participants learned through the process of receiving peer feedback that they became more focused in their delivery of feedback (Nicol & Macfarlane-Dick, 2006). While the research has not specifically examined the differences in feedback given by peers, it appears that the most common type of feedback that peers give other peers would fit into the person-specific feedback. Many researchers have found that the major problem with peer feedback is the tendency to “over-mark” (Cheng & Warren, 1999; Falchikov, 1995; Falckikov & Goldfinch, 2000; Orsmond et al., 1996). Thus, the principle focuses on delivering feedback that 72 relates to helping the learner achieve their end goal, lowering the frequency of person-specific feedback. In summary, the current study proposes that quality feedback from peers will operate by the following principles to enhance the learning experience: that feedback will be delivered immediately, may not be all positive, will happen regularly, will be as specific as possible, and help achieve a learning goal. These principles, which are the elements in the training, will help generate quality feedback. Hypothesis 1: Individuals attending a peer feedback training program will provide higher quality feedback (i.e. as per the five characteristics) to learners than individuals who do not receive the training. Feedback Environment Organizations that empower employees to give and receive formal and informal feedback allow for employees to improve job performance (London & Smither, 2002). This type of environment is similar to that of supportive learning environments, where employees are allowed to foster their skills that they learn in training (Tracey, Tannenbaum, and Kavanagh, 1995). Thus, the organizational value of feedback is an important component of understanding how quality peer feedback training will have an impact on feedback providers. There are several components of a feedback supportive environment. Researchers have recognized that the organization must be supportive from two main sources: supervisors and peers (Cromwell & Kolb, 2004; Lim & Morris, 2006; Nijman, Nijhof, Wognum, & Veldkamp, 2006). Russ-Eft (2002) notes that while supervisors provide environments for the reinforcement of learned skills, peers provide support of learning and development. Schelfhout, Dochy, and Janssens (2004) emphasized that learning environments only exist if the feedback culture 73 includes feedback from the peers. Saks and Haccount (2008) echoed this sentiment when they noted that organizational contexts may have more impact on implementation of third generation learning strategy because of this organizational support for peer-to-peer interaction throughout the workday. As such, organizations may already enable a third generation learning strategy. London and Smither (2002) illustrated three major strategies in which organizations can enable this type of organizational support. In these environments, peer feedback training may be less needed because (1) the organization has embedded feedback into the employee’s daily routine or (2) they deliver quality feedback regularly. If the organization supports the frequent exchange of feedback, peers may reinforce the skills that they have and thus motivate them to give high quality in the training programs that are third generation enabled. Thus, the relationship between feedback environment and feedback training may be interactive. Those who already have a high feedback environment may have less behavioral change than those who have low feedback environments. Similarly, the perceptions of usefulness of peer feedback training may be lower than those who perceive high organizational support for feedback. However, someone who does not attend the peer feedback training but is in a high feedback environment may already be empowered to deliver quality peer feedback as they have been exposed to other trainings (London & Smither, 2002) and someone who has a low feedback environment may benefit more from peer feedback training because they have not been exposed to quality feedback delivery. The interactive effect of perceived feedback environment on peer feedback training is illustrated in Figure 2. Hypothesis 2: Feedback training and perceived feedback environment will interact such that those who perceive a higher feedback environment will be less affected by peer feedback training than those who have a lower feedback environment. 74 Feedback-Receiver Hypotheses Receiver Self-Efficacy Self-efficacy is the “judgment of one’s capability to organize and execute the course of action to produce given attainments” (Bandura, 1997, p. 3). Messages received during training impact the individual’s confidence to perform that task in the future (Bandura, 1977; Bandura, 1986). Positive training self-efficacy relates to a variety of training outcomes: higher training outcomes on computer skills (Gist, Schwoere, & Rosen, 1989; Martocchio & Webster, 1992), idea generation skills (Gist, 1989), interpersonal complex skills (Gist, Stevens, & Bavetta, 1991), and radar-tracking tasks (Ford, Smith, Weissbein, Gully, & Salas, 1998). Martocchio and Webster (1992) demonstrated that sustaining positive feedback throughout a training session had more benefits on the trainee performance than negative feedback. Martocchio and Dulebohn (1994) demonstrated a similar effect on eighty-six employees who received software training. Participants who received feedback that was based on factors within the trainee’s control had higher self-efficacy. More importantly, quality feedback leads to higher levels of self-efficacy of the learner, even if the feedback is not always positive. As it relates to control theory, the higher the quality of the feedback, the more likely the learner can increase his or her progress towards the end goal (Nicol & Macfarlane-Dick, 2006). Black and William (1998) noted that, if the feedback is more about the person and not the performance, then feedback has a negative effect on a person’s confidence. Thus, feedback that focuses on the end goal may help increase their self-efficacy to achieve their end goals. Peer feedback has been theorized to increase self-efficacy, given the context of peer-topeer learning and comfort zone with the peer environment (Kraiger, 2008). London and Smither (1995) noted that the peer evaluation affects the self-efficacy of the peer receiving the feedback. 75 Thus, given that quality feedback focuses also on negative feedback, it equips users with the selfefficacy to apply the learning in other contexts and warrant against “learned-helplessness” (Dweck, 1999). Hypothesis 3: Trainees who receive higher quality peer feedback will have higher selfefficacy than those who receive lower quality feedback. Looking at the impact of the peer feedback training, the quality of peer feedback may mediate the relationship between self-efficacy and peer feedback training. Mediation between variables explains how “external physical events take on internal psychological significance” (Baron & Kenny, 1986, p. 1176). Peer feedback training does not in itself increase self-efficacy (although, it could yield higher self-confidence to give peer feedback but that is not in scope for this study). However, there may be an indirect relationship when considering high quality feedback delivery as a mediating variable between peer feedback training and self-efficacy. Hypothesis 4: Quality of the peer feedback will mediate the relationship between peer feedback training and self-efficacy. Receiver Motivation Motivation to learn is the “trainee’s specific desire to learn the content of the training program” (Noe, 1986, p. 743). If the trainee is not motivated to learn the information, there is little or no chance the learner will be able to apply the knowledge or skills learned. For example, Facteau, Dobbins, Russell, Ladd, and Kudisch (1995) found that trainees higher on motivation to learn had higher learning outcomes than those lower on motivation to learn. Similarly, Mathieu, Martineau, and Tannenbaum (1993) found that motivation to learn affected training outcomes and Martocchio (1992) found that motivation influenced attitudes towards training. 76 Given the critical role of motivation, quality feedback’s impact on motivation is an additional hypothesized benefit in this learning environment. For example, Butler (1988) demonstrated that including feedback in learning environments improved a student’s motivation compared to environments where just grades were given without feedback. In fact, environments without feedback have a negative impact on motivation and lessen the ability for the student to turn into a lifelong learner (Harlen & Crick, 2003). Similarly, researchers have found that more immediate feedback, which is one of the principles of the peer feedback, increases motivation (Brosvic et al., 2006) and Van-Dijk and Kluger (2004) found support that negative feedback, another principle of peer feedback, may elevate motivation. Translated to a peer feedback environment, Kraiger (2008) proposed that the social environment will promote an equal environment for peers to express their feedback and learning and will make the learning environment more tangible. As participants learn to give more appropriate peer feedback, the motivation to learn by the feedback recipient will increase because the feedback will become even more relevant for the learner. Similarly, Topping and colleagues (1998, 2000) and Chhokar and Wallin (1984) found support that peer feedback increased motivation when compared to supervisor feedback because feedback from supervisors felt more performance management based. Hypothesis 5: Trainees who receive higher quality peer feedback will have higher motivation to learn than those who receive lower quality feedback. Looking at the impact of the peer feedback training, the quality of peer feedback may mediate the relationship between motivation and peer feedback training in a similar way as selfefficacy. Because the training focuses on delivering high quality feedback, the participants who 77 receive quality feedback because of the training will have higher motivation, as it assists them to achieve the desired end state of their behaviors. Hypothesis 6: Quality of the peer feedback will mediate the relationship between peer feedback training and motivation to learn. Receiver Feedback Orientation The idea of “learning organizations,” which is consistent with many of the foundations of third generation learning strategy, centers on continuous self-improvement and development of employee skills that match organizational goals (Gagne, 2002; Rich, 2002). Organizations search for employees who guide their own learning and developmental plans through purposeful seeking of feedback (Ashford and Cummings, 1983). London and Smither (2002) noted that there are six dimensions of a feedback seeker: (1) the general liking of feedback which includes a positive attitude towards feedback and less anxiety towards performance evaluation; (2) a behavioral propensity to actually seek out feedback; (3) a cognitive propensity to process feedback purposefully and consciously; (4) a public awareness of oneself in the view of others; (5) a belief that feedback offers insights not found elsewhere that can cause individuals to become more effective and achieve desired end states; and, finally, (6) personal feeling of accountability as it relates to acting on feedback. There are various levels of the feedback orientation trait, as some individuals do not want to seek or act upon feedback (Levy et al., 1995; Northcraft & Ashford, 1990). Given that feedback orientation is connected to performance (Renn & Fedor, 2001), it makes sense that people who tend to seek feedback would likely be more receptive to high quality peer feedback, and thus, be more motivated to act upon the feedback given. 78 Thus, feedback orientation may also interact with the quality of the peer feedback to affect the receiver’s self-efficacy and motivation. Individuals who are high on feedback orientation who receive high quality feedback from peers may have elevated self-efficacy and motivation. For those who are low in feedback orientation, the high quality feedback delivered may not increase self-efficacy or motivation as much as the high feedback orientation individuals. Ultimately, this relationship may negate the overall value of focusing on improving feedback in peer-to-peer learning environments if organizations have mostly low feedback orientation employees. If this hypothesized relationship exists, it could mean that organizations would have to consider individual differences in creating training programs (see Nease, Mudgett, & Quiñones, 1999). The proposed interaction is illustrated in Figure 3. Hypothesis 7: Feedback orientation will interact with the level of feedback quality such that high feedback orientation individuals who receive high quality feedback are more likely to have higher self-efficacy and motivation. Training Outcomes Training effectiveness and outcomes have been measured in several different ways: reactions to the training, length of retention, behavioral changes, and results on a task (Kirkpatrick, 1960). The ultimate goal of this study was to increase training effectiveness in third generation learning environments through focusing on improving the quality of peer feedback through a purposeful intervention. Thus, the final set of hypotheses analyzed the effects of improved feedback. Researchers have found that high quality feedback will increase performance outcomes (e.g. Gomez-Mejia, Balkin, & Cardy, 2001; Kreitner & Kinicki, 2001). Similarly, research has found that quality feedback leads to higher training and learning outcomes (e.g. Black & William, 79 1998; Goodman, 1998; Schmidt, 1991) because quality feedback helps the learner reach their learning and performance goals. High quality feedback will assist the learner in clarifying how to deliver quality performance and deliver adequate information about their performance (Nicol & Macfarlane-Dick, 2006). These will all help learners reach higher training outcomes. Hypothesis 8: Higher quality feedback will lead to higher (A) training utility, (B) knowledge outcomes, (C) behavioral outcomes, and (D) feedback utility. While high quality feedback may explain the relationship between peer feedback training and training outcomes, trainee characteristics may also explain the results of the training outcomes through mediating effects. Mediating effects explain “how external physical events take on internal psychological significance” (Baron & Kenny, 1986, p. 1176). As such, selfefficacy and motivation may mediate the relationship between high quality peer feedback and training outcomes (Blume et al., 2010). Self-efficacy influences training outcomes (e.g. Frayne & Latham, 1987; Gist, 1989). Very little research – if any – has demonstrated a direct link between feedback quality and training outcomes through the mediating effects of self-efficacy. However, researchers have found mediating effects of self-efficacy on training outcomes (e.g. Gist, Schwoere, & Rosen, 1989; Martocchio & Webster, 1992). For example, Gomersall and Myers (1966) demonstrated that self-efficacy fully mediated the relationship of a training intervention and training outcomes. This is consistent with researchers who test and find significant results for self-efficacy mediating learning outcomes (e.g. Blume et al., 2010; Tracey et al., 2001). Thus, this study hypothesized self-efficacy may mediate the relationship between peer feedback training and training outcomes. 80 Hypothesis 9: Self-efficacy will mediate the relationships between high quality peer feedback to (A) training utility, (B) knowledge outcomes, (C) behavioral outcomes, and (D) feedback utility. Similarly, motivation has been found to affect training outcomes (e.g. Tannenbaum et al., 1991). The importance of motivation on learning outcomes is demonstrated in the Quiñones (1995) study. He found evidence for correlational relationships between motivation and learning outcomes. The role of motivation mediating relationships between feedback and learning outcomes has not be adequately studied. However, researchers have found mediating effects of motivation on learning outcomes (Blume et al., 2010; Colquitt, LePine, & Noe, 2000). Similarly, Barrick, Stewart, and Piotrowski (2002) found that motivation mediated the relationship between personality and job performance in sales associates. Thus, this study hypothesized motivation may mediate the relationship between peer feedback training and training outcomes. Hypothesis 10: Motivation will mediate the relationships between high quality peer feedback to (A) training utility, (B) knowledge outcomes, (C) behavioral outcomes, and (D) feedback utility. 81 METHODS Participants Participants were employees at an international, Fortune 500 company. Participants were asked to evaluate a new method of learning for a course that would increase an HR professional’s capability to train others. The training was part of a pilot program testing Blackboard and was part of the company’s management skills course offerings for HR professionals. The training was voluntary in nature, as participants enrolled in the training course when it was beneficial for their career. Participants received no award, direct or indirect, for participating in the study. Participants were part of the human resources function with limited experience in training others. Nine participants dropped the course during the experiment. Of the remaining 203 voluntary participants (20% male; 43% female; others did not wish to identify), 12% were Caucasian/White, 3% were Hispanic/Latin, 10% were Asian/Pacific Islander, 1% were Black/African America, and 74% identified as other or wished to not disclose their ethnicity. The tenure was fairly even: 10% were less than one year to the company, 10% were 1-3 years, 14% were 4-5 years, 12% were 6-10 years, 10% were 11-15 years, and 10% were more than 15 years. Age was not available for the participants. Design The study used a one-way experimental design with two (2) levels of a between subjects variable. The manipulated variable involved a pre-training intervention in which participants learned how to give high quality peer feedback. The participants were assigned to either condition (peer feedback training or no peer feedback training). 82 Task The focus of the study was on evaluating the effects of peer feedback training in online learning on subsequent training characteristics and performance outcomes of a training intervention. To assess these outcomes, this study utilized the criteria Quiñones (1995) used for selecting a training intervention for evaluating between subject differences in real-world situations: (1) participants must have a limited prior knowledge of the task, otherwise it would create a ceiling effect, (2) the task must be difficult enough to allow for variation in performance outcomes, and (3) the task must be complex enough to ensure that the participant is paying attention to task and training. The study added two additional conditions: (4) while the task should be complex, it should be within comprehension of the targeted employee as so much that feedback is necessary to complete course objectives, and (5) the training had to be a designed in such a way that peer-to-peer interaction was essential to the overall course objectives. The course selected was an “Introduction to Instructional Design and Training Others.” This course was delivered to global human resource managers who had chosen to advance their skills in training others. The course focused on how to deliver effective training by incorporating adult learning theory principles. Participants who enroll in this type of course typically are brand-new or novice trainers and have (1) never taught a course and/or (2) seeking further development of their training skills based on the career goals aligned by the participant and his or her manager. The course is divided into five lessons and delivered through web-based instruction. The course lasts for approximately five weeks with one module delivered weekly. Class sizes ranged from 20-50 participants per class; thus, there were six classes administered all together. The learner spent approximately 2-4 hours per module. This course met the requirements set forth above because (1) the course allowed for participants to effectively partner with one other 83 trainee to explore the effects of peer-to-peer feedback and (2) the course learning objectives were designed to rely heavily on peer-to-peer interaction. An outline of the lesson overview, feedback exercises, and relationship to course objectives are described in Table 1. The primary instructor for the course was the experimenter, and secondary instructors were informed about the nature of the peer training add-on included in the beginning of the course. Employees were identified only with learner identification numbers (e.g. Learner001), as to prevent experimenter bias. Procedure A summary of the procedure is demonstrated in Figure 4. Participants enrolled for the course using the standard learning management system of the company. Marketing to potential learners described a new online learning platform that utilizes peer-to-peer interaction for a course on how to teach using adult learning strategy and how to utilize online training to create dynamic learning experiences. While the nature of the marketing campaign and focal training topic may have drawn users who are already eager to give and receive peer feedback, the company had noted that the effects of similar peer-to-peer learning initiatives were not overwhelming positive, which is not unlike the results of other companies who have experimented with internal peer-to-peer sites and found mixed engagement rates (e.g. DiMicco, Millen, Geyer, Dugan, Brownholtz, & Muller, 2008; Priya, 2011). Thus, the study did not purposefully exclude people based on their motivation to give feedback, share knowledge in online learning environments, or experience using peer-to-peer learning tools. As part of the marketing campaign, the consent form was included (Appendix A). The informed consent reminded employees that the collection of employee data for research purposes was voluntary and would not affect their current or future employment with the company. 84 Participants who wished to enter the training read through the consent form and then clicked a link to enroll in the course. Because random sampling would have required flexibility in course scheduling, participants enrolled in the course that best fit their particular work schedule. This type of sampling resulted in the first one hundred participants in the training receiving the manipulation intervention and the second one hundred participants receiving the control condition. The course content and options for application and practice were identical in nature for both groups. The difference between the two courses is that one condition contained a pre-training on how to deliver high quality peer feedback while the control condition was given an presentation on HR history at the company. When the course began, the peers were placed into two partner teams using the Blackboard “team” technology. With this technology and the precise design of the course, interactions occurred between the two peers. The experimenter downloaded all interactions. The introduction to the course informed the participants of the nature of the training curriculum. They were introduced to the online learning system (Blackboard), which illustrated technologically how to give peer feedback using the various exercises of the course asynchronously. Participants were then told to complete two assessments: Feedback Environment Scale and Feedback Orientation Scale. After a brief introduction to the technology platform, those participants in the control condition were assigned a peer partner. Control condition participants were then presented with a history of training interventions that existed at the company and a history of the HR function to ensure similar allocation of time requirements for the two conditions. 85 Those participants who were in the manipulation condition were introduced to peer feedback training and also assigned to a partner to allow for isolated investigation of the peer feedback giver and receiver interactions. The peer feedback training consisted of a forty-five minute training session on how to deliver high quality peer feedback (See Appendix B for outline of training course used on peer feedback) across three mini-lessons. This was delivered in a live web-training environment (Cisco’s WebEx). The methodology used was similar to Xio (2010): (1) understanding the rationale of social learning, (2) understanding the aspects of peer feedback, and (3) how to deliver peer high quality feedback. Both the feedback giver and the feedback receiver received this training (the implication of this is discussed in the limitations section). In the first lesson, participants were introduced to the idea of social learning where peer feedback becomes a critical component of social learning. Participants were given examples of social learning and how it can benefit the individual. Following the examples, participants were given an unsupervised 5-question multiple-choice quiz to solidify their understanding. In the second lesson, participants were introduced to the value of feedback and what research has been done to understand how to give effective feedback. Participants were given a 5-minute overview of each element of feedback: (1) sign balance, (2) frequency, (3) timing, and (4) type, and (5) frequency. The skills selected were based on the study by Xiao (2010), but focused more on traditional feedback theory (e.g. Ilgen et al., 1979). This lead to the discussion of the five principles of giving peer feedback in their upcoming course: 1. 2. 3. 4. 5. Feedback will be delivered immediately following a learning exercise. Feedback will not always be positive. Feedback will happen regularly. Feedback will be as specific as possible. Feedback will help the learner achieve their learning goal. 86 Each component of high quality feedback was illustrated by demonstrations of both good and bad examples (Baldwin, 1992). Following this conversation, participants were asked informally to list three reasons why feedback is important to achieving their goals. This prompt internalized the reasoning behind delivering high quality feedback (Toll, O’Malley, & Katulak, 2007). In the final section, principles were summarized, and participants practiced delivering high quality peer feedback. Participants were asked to give two examples of delivering peer feedback based on the training they had just received on feedback. One feedback statement had to be positive, and one feedback statement had to be negative. Participants demonstrated their ability to deliver feedback, and the instructor recorded their ability to demonstrate quality feedback statements before leaving the intervention and entering the new trainer’s training program. Following the peer training or training history conditions, participants were given an orientation of the course, given their login information for the course, and told that the exercises in each course were set up as social learning exercises such that peers would post their exercises in their online environment for the other peer to provide feedback. There was no limit or set rule for the amount of feedback interactions that would happen between the peers. After this pretraining intervention, participants began the course. During the course, participants completed five, on-demand modules. Each module lasted for approximately one week, making the course last five weeks all together. Participants were allowed to complete the course anytime asynchronously. Although not required due to business priority changes and vacation schedules, participants were encouraged to follow the course syllabus timing, which noted each participant should complete each lesson by the end of the week. The first module was an orientation to the course, the second module was a discussion on 87 various models of designing training (e.g. 4D), the third module was a presentation of adult learning strategies to engage learners, the fourth module discussed online training environments, and the fifth was an introduction to teaching in the Blackboard training environment. The only difference in the course for the control and the manipulation group was module one. Consistent with Xiao (2010), the instructor in the peer feedback training course provided feedback on the peer feedback given in module one only. For every module, participants were allowed to give feedback to each other on their exercise in an electronic two-way discussion forum facilitated by the Blackboard (version 9.1) environment. The exercises typically involved an on-the-job relational example. These feedback interactions were presented at the end of each module in a two-member team discussion board. For example, participants were asked to describe a time when they had attended training and to evaluate the effectiveness of the training as it relates to the incorporation of adult learning concepts. A peer was then allowed to provide feedback on that example given. The other peer was allowed to continue the conversation with the feedback receiver if he or she wished. The instructor had access to these feedback sessions. All together, there were five feedback interactions, one for each lesson. The layout of course content and exercises is presented in Figure 4. Prior to the last lesson, participants completed a Self-Efficacy Scale and Motivation Scale to evaluate the learners’ confidence and motivation to transfer the skills learned. At the end of the course, participants were given four exercises/assessments. First, there was a Knowledge Assessment. Second, the Adult Learning Task asked participants to design a short module to demonstrate mastery in incorporating adult learning principles into their own training program. Upon completion of this exercise, a Peer Perceived Feedback Quality was given. Finally, within 88 24 hours of completing the entire course, trainees were given the corporate Training Utility assessment to complete. Following collection of the data, participants were debriefed and told the details of the study (Appendix I). The researcher emphasized there were no potential adverse impacts of their participation in one course over the other and their participation had no bearing on their current or future employment at the corporation, nor their performance ratings for the current or future fiscal year. Control participants were offered the ability to receive the feedback intervention to improve their peer feedback if they so desired. MEASURES The measures are listed in accordance to when the participant completed the assessment. Results of these mesures are detailed in the “Results” section. Feedback Environment Scale Rosen (2006)’s fourteen-question version of Steelman et al.’s (2004) original 63-question Feedback Environment Scale was used to assess the employee’s perceptions of the feedback in their organization. Learners evaluated perceptions of feedback environment from coworkers and supervisors (Steelman et al., 2004). In the current study, participants indicated their level of agreement to statements regarding their manager and coworker’s support of a high quality feedback environment (Whitaker, Dahling, & Levy, 2007). A sample item was: “My supervisor gives me useful feedback about my job performance.” Questions were anchored on a five-point scale from 1, which was strongly disagree, to 5, which was strong agree. Gregory (2010) found an overall reliability of α = .95. 89 The twenty-eight questions in the measure were analyzed using a confirmatory principal component analysis with varimax (orthogonal) rotation (Kim & Mueller, 1978). Kaiser-MayerOlkin’s measure resulted in .84, above the .5 recommendation for factor analysis (Kaiser, 1974). The analyses yielded two factors explaining 47% of the variance for the entire set of variables. Factor 1 was strongly associated with coworker support (questions fifteen through twenty-eight) and explained 32% of the variance. The second factor was strongly associated with manager support (questions one through fourteen) and accounted for 15% of the variance. After examination of the communalities, questions two and six were low, indicating that the factor model did not work well for these indicators (Garson, 2011). The two questions (question six and twenty-two) were dropped to determine the impact of the removal. When the two questions were discarded, internal consistency remained the same, α = .92, and were thus retained. Both factors were evaluated separately in proceeding analyses; however, because the separate factors did not alter outcomes, the full scale was used as the measure of feedback environment. A copy of the scale can be found in Appendix C. Feedback Orientation Scale Linderbaum and Levy (2010)’s twenty-question assessment of feedback receptivity and feedback orientation was used. Participants were asked to indicate their level of agreement to statements regarding their feedback orientation. An example question was, “Feedback contributes to my success at work.” Questions were anchored on a five-point scale from 1, which was strongly disagree, to 5, which was strong agree. Linderbaum and Levy (2010) found an internal consistency of α = .91. The observed reliability for this study was α = .88. A copy of the scale can be found in Appendix D. 90 Self-Efficacy Scale Quiñones’ ten-question, self-efficacy scale was used. The measure was adapted from Hattrup’s original test (1992) to determine a person’s expectation regarding their future performance on the Training Assessment Test. Participants indicated their level of agreement to statements regarding their self-efficacy. An example question was, “I think I can eventually reach a high level of performance applying adult learning teaching.” Questions were anchored on a five-point scale from 1, which was strongly disagree, to 5, which was strong agree. Quiñones’ observed reliability was α = .90. The observed reliability for this study was α = .78. Individual items were evaluated to determine if dropping one item resulted in a higher alpha level; the scale’s reliability improved to α = .80 when item six was removed. However, when correlations were re-evaluated with the item dropped, statistical significance did not change. Thus, the entire scale was preserved. A copy of this measure can be found in Appendix E. Motivation Scale Quiñones (1995)’s ten-item scale was used to measure motivation. Participants were asked to indicate their level of agreement to statements regarding their motivation. An example question is, “I am motivated to use the skills emphasized in this training program.” Questions were anchored on a five-point scale from 1, which was strongly disagree, to 5, which was strong agree. The observed reliability for Quiñones was α = .93. Individual items were evaluated to determine if dropping one item resulted in a higher alpha level; this pattern did not emerge. The observed reliability for this study was α = .79. A copy of this measure can be found in Appendix F. 91 Knowledge Assessment (Knowledge Outcome) The training course included a twenty-item knowledge test anchored in five multiplechoice questions, where participants were asked about the principles they learned during the training. The assessment purposefully oversampled some lessons, as a result of organizational priorities. Similarly, the knowledge test was a mixture of knowledge and behavioral questions as way to emphasize messages that the organization deemed critical. The organizational course owner was the author of the knowledge assessment. A split-half reliability check was performed for the knowledge check as consistent with the approach previous training researchers have used in evaluating reliability of knowledge checks (e.g. Quiñones, 1995). Split-half reliability resulted in r = .66. A copy of the measure can be found in Appendix G. Adult Learning Task (Behavioral Outcome) Participants were assessed on their ability to deliver a course module with adult learning content. Following completion of the course, participants are asked to create one module of their course they are planning on creating in the Blackboard environment using the adult learning principles and best practices learned within the course (Blackboard, 2011). A course facilitator rated their performance. Participants were required to get “Satisfactory” in order to pass the course. The corporation does not distinguish levels of performance above “Satisfactory,” but the researchers collected the raw data to distinguish high to low performance. Participants were rated on their ability to deliver a module that emphasizes the seven points illustrated in the course. The scale ranged was from 0 to 25. The behavior measure showed positive skew; transforming the data using a logarithmic transformation did not produce in any substantial differences in outcomes; thus, the original, untransformed data was preserved (Howell, 2007; Tabachnick & Fidell, 2007). 92 To estimate reliability, a course facilitator scored twenty-five other assignments to estimate inter-rater agreement between the course facilitators. A one-way random intraclass correlation coefficient showed moderate agreement, r = .69. The rubric for grading is illustrated in Appendix H. Feedback Utility One facet of Steelman et al.’s (2004) Feedback Environment Scale was used to assess the employee’s perceptions of the feedback utility during the course (Rosen, 2006). The short form only includes the utility of feedback, which looks at the employee’s individual beliefs about the feedback usefulness. In the current study, participants were asked to indicate their level of agreement to five statements regarding their perceptions of feedback given to them during the course. Questions were reworded to emphasize the feedback received just during the training from their peer. A sample item was: “Feedback I received during this training will contribute to my success at training.” Questions were anchored on a five-point scale from 1, which was strongly disagree, to 5, which was strong agree. Linderbaum and Levy (2010) found an overall acceptable subscale reliability, α = .88 for utility. The observed reliability for this study was α = .81. A copy of this scale can be found in Appendix I. Training Utility The organization uses a corporate training utility metric that is immediately delivered at the close of any course. Authored by the Knowledge Advisors, Metrics That Matter assessed the employee’s perceptions of the instructor and the usefulness of the course. Participants were asked to indicate their level of agreement to ten statements. Questions were anchored on a fivepoint scale from 1, which was strongly disagree, to 5, which was strong agree. An example question was, “This training was a worthwhile investment in my career development.” Questions 93 were anchored on a five-point scale from 1, which was strongly disagree, to 5, which was strong agree. An internal pre-pilot found observed reliability to be α = .79. The observed reliability for this study was α = .72. To improve reliability, the scale was further inspected to determine if factors would make up the structure of the scale. Kaiser-Mayer-Olkin’s measure resulted in .65, above the .5 recommendation for factor analysis (Kaiser, 1974). A principal components exploratory factor analysis with varimax (orthogonal) rotation revealed the presence of three factors explaining 58% of the variance for the entire data set. The first factor was strongly associated with questions eight through ten, accounting for 32% of the variance. This factor focused to the applicability of the training. The second factor was highly associated with items three through seven, accounting for 16% of the variance. This factor focused on the training content. The final factor was associated with questions one and two. This conceptually makes sense, as items one and two directly related to instructor perceptions. Given instructor perception questions (1) related to items that would be more typically observed in a face-to-face instruction and (2) were not something specifically examined in this study, a principal components confirmatory factor analysis with varimax (orthogonal) rotation examined if the instructor factor would separate itself from the training perceptions. The presence of two factors explaining 48% of the variance for the entire data set. The first factor was strongly associated with questions three through ten, accounting for 32% of the variance. The second factor was highly associated with items one and two, accounting for 16% of the variance. Thus, these two items were removed from the training utility scale, the reliability increased from α = .73 to α = .76. Both factors were evaluated separately in proceeding 94 analyses; however, because the separate factors did not alter outcomes, the full scale was used as the measure of training utility. A copy of the measure can be found in Appendix I. Feedback Quality Feedback quality was evaluated in three ways: (1) feedback quality ratings and (2) feedback elements, and (3) feedback frequency. Quality Ratings. The feedback statements given were evaluated for quality by four judges – a mix of graduate and post-graduate clinical and industrial-organizational psychologists. Frequency of feedback was measured separately. Rather than having separate measures for each principle, raters were asked to rate the overall quality of each feedback statement made. Osburn and Mumford (2006) employed this strategy when training employees on elements of being creative. Raters were given references to example solutions to assist raters understand varying levels of quality of peer feedback in the same fashion that Redmond, Mumford, and Teach (1993) developed benchmark rating scales for appraising plan quality and originality. To develop quality ratings scales, the experiment focused on the principles outlined in the training course (see above for five elements with supporting references), which were derived from understanding what drives effective feedback (e.g. Locke & Latham, 1984; Nicol & Macfarlane-Dick, 2006; Sadler, 1989; Yorke, 2003). Based on the examples present in feedback articles (e.g. Hattie & Timperley, 2007) and their congruence with the principles, definitions of high, medium, and low quality levels of feedback were developed on a scale of one to five. A feedback researcher also reviewed the ratings for consistency with feedback literature. Table 2 illustrates the nature of these rating scales. Prior to evaluating feedback quality, the raters participated in a training program to familiarize them with the nature of rating scales, which was a similar methods found by other 95 feedback researchers (Nicol & Macfarlane-dick, 2006; Sluijsmans et al., 2002; Xiao, 2010). They were asked to apply these rating scales on a variety of sample feedback sets. The experimenter asked raters to practice until the raters reached 75% agreement on sample feedback statements with the expert rating. After which, judges were allowed to work on the judging of peer feedback statements (Osburn & Mumford, 2006). Each rater evaluated approximately 500 feedback statements, resulting in two ratings for each individual feedback statement. A one-way random intraclass correlation coefficient showed high agreement for the raters. The final agreement rating between judges was .88 (95% CI .84-.91), which is above the acceptable cut off of .7 for intraclass correlation coefficients (Bartko, 1976; Salvaggio et. al, 2007). The average of all five feedback statements was used as the quality rating score for each feedback giver. Feedback Elements. Feedback raters were also asked to dissect the presence of the individual feedback elements with 1s (presence exists) and (0s). Prior to evaluating feedback elements, the raters were trained on the nature of the elements. They were asked to apply these rating scales on a variety of sample feedback statements. The experimenter asked raters to practice until the raters reached 75% agreement on sample feedback statements with the expert rating. After which, judges were allowed to work on the judging of feedback elements. The individual elements were: 1. 2. 3. 4. 5. The feedback statement goes beyond praise for actions well done. The feedback statement refers to receivers’ specific words or actions. The feedback statement tells what to do to improve. The feedback statement refers to specific behaviors to take. The feedback statement gives reasons for recommended actions to take. Each rater evaluated approximately 500 feedback statements, resulting in two ratings for each individual feedback statement. Once elements were assessed, a sum of all five was calculated, with 0 meaning no element of the feedback elements existed and 5 meaning all elements existed. 96   The average of all five-feedback element summations was used as the feedback element score for each feedback giver. A one-way random intraclass correlation coefficient showed moderate agreement, .67 (95% CI .56-.75). Additional investigation was run to determine if dropping a feedback element would increase inter-rater agreement. Raters had more disagreement on the first feedback element than the other four. When the feedback element was dropped, agreement between raters increased to .71 (95% CI .68-.79). However, when that element was dropped, the correlations between feedback element and other measures did not change significantly. Thus, the entire feedback score was kept. The feedback elements’ reliability, while not high, was acceptable (Anastasia, 1998; Salvaggio et. al, 2007). Feedback frequency. Feedback frequency was assessed as the number of feedback interactions the feedback giver gave during the entire course across all modules. Analytic strategy The analytic strategy employed for this study was based on the casual relationship presented in the model and Figure 1. Individuals in the two conditions were used for analysis (n = 203). Data was entered into SPSS 20.0. There was one independent variable with two levels: (1) control/non-manipulation and (2) manipulation/peer feedback training. For all scales unless otherwise noted in the methods section above, Cronbach’s coefficient alpha was used to estimate the internal consistency of the scales internal validity checks (Crokcer & Algina, 1986). This study employed a similar analytic strategy as that of other feedback researchers (e.g. Lipnevich & Smith, 2009). Correlations between all variables were conducted. Multivariate analysis of variance (MANOVA) was employed to examine the effects of the manipulation on 97 the other scales present in the model (feedback quality, receiver characteristics, and training outcomes) to reduce inflation of Type I error and increase the accuracy of finding differences in groups with multiple dependent variables as present in this study (Field, 2000; Grice & Iwasaki, 2007). MANOVA was used to evaluate if there were significant differences between the peer feedback training learners and the control group learners on the feedback quality, receiver characteristics, and training outcomes. Box’s test of equality of covariance matrices was run to evaluate whether the variances among the dependent variables were the same for all factors (Green, Salking, & Akey, 2000). When MANOVA results yielded significant results, univariate analysis of variance (ANOVA) was conducted on the respective outcome variables. Homogeneity of variances assumption was checked through Levene statistics. Eta-squared was calculated to determine effect size (Aguinis, Mazurkiewicz, & Heggestad, 2009). Finally, the model presented had several mediating relationships between peer training to training outcomes and quality of peer feedback to training outcomes. A mediation test (James & Brett, 1984) was used to check this proposed relationship, when conditions were present for the possibility of a mediating relationship. Mediating relationships exist when the variable (in this case, motivation to learn, self-efficacy, and quality of feedback) adds prediction of the dependent variable. A regression analysis was run for the specific independent variable (peer training and no peer training) predicting training outcomes (four specific outcomes: feedback utility, training reactions, knowledge outcomes, and behavioral outcomes). A secondary regression was run between the mediating variables and the training outcomes. These two regression analyses output the raw regression coefficients and standard errors needed to run the Sobel test of mediation. The same procedural was done to determine mediating effects for quality of peer feedback to training outcomes through trainee characteristics.   98 RESULTS The results are presented in the following order: (1) the results of descriptive data, (2) the results of the model and hypothesis testing, and (3) additional post experimentation analyses. Descriptive Data Table 3 presents the combined means, standard deviations, n-size and reliabilities. With the exception of knowledge, scales showed sufficient reliabilities to proceed with the data analyses (Anastasi, 1998). Concerning the moderate reliability for the knowledge test, there appeared to be no concern for floor or ceiling effects because the knowledge test showed sufficient range. The n-size of behavior was smaller than the rest of the measures. However, Little’s MCAR test revealed a non-significant Chi-square compared against the other training 2 outcome variables, χ (3, N = 203) = 7.75, ns. Demographic variables were not significantly associated with any variables of interest, which halted further investigation of demographics as covariates. Of the 203 participants, most chose not to reveal their demographic variables: only 26% noted their race, 66% their tenure, and 63% their gender. Model & Hypotheses Investigation For hypothesis testing, a series of multivariate analysis of variance tests were conducted in the presence of multiple related outcomes. Following Cramer and Bock (1966), MANOVA’s were used to protect against the Type I error inflation for follow up ANOVA’s and post hoc comparisons. Prior to conducting MANOVAs, Pearson’s correlations were performed between all variables in order to test the assumption of MANOVA that appropriate dependent variables are related to each other. 99 As demonstrated in Table 4, meaningful patterns of correlations were observed in relevant variables related to hypotheses (Meyers, Gampst, & Guarino, 2006). As hypothesized, condition affected trainee characteristics (r = .19 for motivation, r = .39 for self-efficacy) and training outcomes (r = .35 for feedback utility, r = .19 for training utility, and r = .25 for knowledge). To justify the use of MANOVAs, outcome variables were also correlated (e.g. r = .19 for feedback and training utility). Hypothesis 1 Hypothesis one proposed that the condition would affect feedback quality. That is, employees who attended the peer feedback training would deliver higher quality feedback to their peer than those who did not receive the training. Feedback quality was measured in three ways: quality ratings, feedback elements, and feedback frequency. A MANOVA was conducted to determine the effect of condition. Box’s M value of 74.91 was significant; as a result, the more robust test, Pillai’s Trace, was used as an alternative to more popular, Wilk’s lamda (Olson, 1979; Weiner, Freedheim, Schinka, & Velicer, 2003). A MANOVA revealed a significant multivariate main effect for condition, Pillai’s Trace = .66, F(3, 197) = 129.78, p < .01, partial 2 η = .66. Given the significance of the overall test, the univariate main effects were examined. Prior to conducting the ANOVAs, the homogeneity of variance assumption was test for the three measures. Based on the Levene’s F tests, the homogeneity assumption was met for feedback elements and frequency, but quality ratings were statistically significant (p < .05). An examination of standard deviations revealed that none of the largest standard deviations were more than four times the size of the corresponding smallest, suggesting that the ANOVA would be robust in this case (Howell, 2009). Significant main effects for condition were obtained for 2 quality ratings, F(1, 199) = 305.03, p < .01, partial η = .61, feedback elements, F(1, 199) = 100 2 196.34, p < .01, partial η = .49, and feedback frequency, F(1, 199) = 39.912, p < .01, partial η 2 = .17. These findings confirm that the feedback training condition yielded higher feedback quality (M = 2.94; SD = .67), feedback elements (M = 2.34; SD = .54), and feedback frequency (M = 4.99; SD = 1.49) compared to the control condition for feedback quality (M = 1.58; SD = .37), feedback elements (M = 1.31; SD = .49), and feedback frequency (M = 3.61; SD = 1.72). Thus, the first hypothesis was confirmed. Hypothesis 2 Hypothesis two stated that feedback training and perceived feedback environment would interact such that employees from a higher feedback work environment would be less affected by peer feedback training than those who had a lower feedback environment. Although feedback environment was not related to quality ratings, feedback elements, or feedback frequency (see Table 4), analyses were conducted to ensure the lack of interaction. Box’s M value of 196.86 associated with a p value of .02 was interpreted as non-significant based on Huberty and Petoskey (2000)’s guideline (i.e. p < .005). Thus, covariance between groups was assumed equal. A two-way MANOVA revealed a non significant multivariate effect of feedback training and feedback environment on feedback quality, Pillai’s Trace = .30, F(51, 471) = 1.04, ns. Thus, the second hypothesis was not supported. Hypothesis 3 Hypothesis three stated that trainees who received higher quality peer feedback would have higher self-efficacy than those who received lower quality feedback. Table 4 illustrates that from a correlational perspective, each feedback measure is related to self-efficacy. A multiple regression model for the three predictors of quality feedback (quality ratings, feedback elements, 101 2 and feedback frequency) produced an R = .15, F(3, 1978) = 11.38, p < .01, indicating participants who received higher quality feedback had higher self-efficacy. Quality rating was positively related to the model (β = .24, t = 4.14, p < .01) as well as feedback frequency (β = .16, t = 2.24, p < .01). Feedback elements did not contribute in the model, (β = .14, t = 1.42, ns). Thus, there is support for hypothesis three. Hypothesis 4 Hypothesis four predicted that the quality of the peer feedback would mediate the relationship between peer feedback training and self-efficacy. Given the correlations found between condition, feedback quality, and self-efficacy, a Sobel test of mediation was conducted using the regression-based methods described by Preacher and Hayes (2008). All three measures of feedback quality qualified for mediating tests given the significant correlational relationships (see Table 4). The results indicated that feedback quality significantly mediated the relationship between condition and self-efficacy, z = 5.06, p < .01 for quality rating; z = 2.71, p < .01 for feedback elements; z = 2.94 p < .01 for feedback frequency. Thus, hypothesis four was supported. Hypothesis 5 Hypothesis five proposed that trainees who received higher quality feedback would have higher motivation than those who received lower quality feedback. Table 4 summarizes the correlation results. As can be seen, only feedback quality related to motivation. A multiple regression model for the three predictors of feedback (quality ratings, feedback elements, and 2 frequency) produced R = .05, F(2, 198) = 3.93, p < .01 indicating participants who received higher quality feedback had higher motivation. Consistent with the correlations found, feedback quality was positively related to the model (β = .28, t = 2.67, p < .01). Neither feedback elements 102 (β = -.08, t = -.84, ns) nor feedback frequency (β = .05, t = .63, ns) contributed significantly to the model. Thus, although small in effect, there is support for hypothesis five. Hypothesis 6 Hypothesis six predicted that the quality of the peer feedback would mediate the relationship between peer feedback training and motivation. Given the correlations found between condition, feedback quality, and motivation, a Sobel test of mediation was conducted using the regression-based methods described by Preacher and Hayes (2008). Only feedback quality as defined by ratings qualified for mediation tests (see Table 4). The results indicated that feedback quality as defined by quality rating significantly mediated the relationship between condition and motivation, z = 3.40, p < .01. Thus, hypothesis six was partially supported. Hypothesis 7 Hypothesis seven proposed that feedback orientation would interact with the level of feedback quality such that trainees who had a high feedback orientation and who received high quality feedback would have higher self-efficacy and motivation. Feedback elements were used as the measure of feedback quality, given the lack of correlation of between feedback orientation to quality rating and frequency. Box’s M value of 17.09 associated with a p value of .98 was interpreted as non-significant based on Huberty and Petoskey (2000)’s guideline (i.e. p < .005). A two-way MANOVA revealed a non significant interactive effect of feedback orientation and feedback quality on trainee characteristics, Pillai’s Trace = 1.28, F(86, 206) = .71, ns. Thus, hypothesis seven was not supported. 103 Hypothesis 8 Hypothesis eight stipulated that higher quality feedback would lead to higher training outcomes as defined through training utility, feedback utility, knowledge outcomes and behavioral outcomes. Because MANOVAs require correlations between variables (see Table 4), the behavioral outcome measure was not tested (Meyers, Gampst, & Guarino, 2006). The only feedback component examined was quality rating due to the prevailing effect quality had in previous hypotheses. Box’s M value of 135.23 associated with a p value of .38 was interpreted as non-significant (Huberty & Petoskey, 2000). Thus, covariance between groups was assumed equal. A one-way MANOVA revealed a significant multivariate effect for feedback quality, 2 Pillai’s Trace = .61, F(99, 507) = 1.31, p < .05, partial η = .20. Given the significance of the overall test, the univariate main effects were examined. Based on the Levene’s F tests, the homogeneity assumption was not met for feedback utility or training utility (p < .05). An examination of standard deviations suggested that the ANOVA would be robust (Howell, 2009). A significant effect for quality was obtained for knowledge, F(33, 169) = 1.94, p < .01, partial η 2 = .28, indicating those participants who received higher quality feedback had a higher knowledge outcome. However, there was no significant effect of quality on training utility, F(33, 169) = .91, ns, or feedback utility, F(33, 169) = 1.30, ns. Thus, hypothesis eight was only partially confirmed. Hypothesis 9 Hypothesis nine proposed that self-efficacy would mediate the relationships between feedback quality and training outcomes. Mediation exists when (1) the initial variable (feedback quality) is correlated with the outcome (training outcomes), (2) the initial variable is correlated 104 with the mediator (self-efficacy), and (3) the potential mediator affects the outcome, (Baron & Kenny, 1986). These conditions for mediation were only satisfied for training utility and feedback utility, as self-efficacy was not related to knowledge or behavior. Both feedback ratings and feedback elements qualified for mediation tests. The results indicated that self-efficacy significantly mediated the relationship between feedback quality as defined by quality rating and training utility, z = 3.35, p < .01, and feedback utility, z = 3.25, p < .01; similarly, self-efficacy mediated the relationship between feedback quality as defined by feedback elements and feedback utility, z = 2.29, p < .05. Thus, hypothesis nine was partially supported. Hypothesis 10 Hypothesis ten proposed that motivation would mediate the relationships between feedback quality and training outcomes. Mediation exists when (1) the initial variable (feedback quality) is correlated with the outcome (training outcomes), (2) the initial variable is correlated with the mediator (motivation), and (3) the potential mediator affects the outcome, (Baron & Kenny, 1986). Conditions for mediation were not satisfied to conduct a mediation test, as none of the training outcomes were related to outcomes. Thus, hypothesis ten was not supported. Additional Post-Experimentation Analyses After evaluating the data, additional analyses were conducted to investigate the effects of other relationships based on the correlations found in Table 4. The Effect of Condition on Trainee Characteristics and Outcomes Given the significant relationships found in Table 4, follow-up tests examined if the condition had an effect on trainee characteristics of self-efficacy and motivation and training 105 outcomes. First, a one-way MANOVA was conducted to examine the effects of training condition on trainee characteristics. Box’s M value of 7.09 associated with a p value of .07 was interpreted as non-significant based on Huberty and Petoskey (2000)’s guideline (i.e. p < .005). A one-way MANOVA revealed a significant effect for condition on trainee characteristics, 2 Pillai’s Trace = .16, F(2, 200) = 19.61, p < .01, partial η = .17. Given the significance of the overall test, the univariate main effects were examined. Based on the Levene’s F tests, the homogeneity assumption was met for the two trainee characteristic scales. Significant effects for 2 condition were obtained for self-efficacy, F(1, 201) = 37.76, p < .01, partial η = .16 and 2 motivation, F(1, 201) = 1.08, p < .05, partial η = .04. These significant findings reveal that trainees who received the peer feedback training had higher motivation (M = 3.39; SD = .42) and self-efficacy (M = 3.36; SD = .48) compared to the motivation (M = 3.25; SD = .33) and selfefficacy (M = 2.98; SD = .42) of the control condition. Next, condition was evaluated in its effect on training outcomes. Because behavior outcomes were not significantly related to feedback quality (see Table 4), behavior was dropped from MANOVA tests (Meyers, Gampst, & Guarino, 2006). Box’s M value of 70.76 associated with a p value was interpreted as significant; as a result, the more robust test, Pillai’s Trace, was used as an alternative to more popular, Wilk’s lamda (Olson, 1979; Weiner, Freedheim, Schinka, & Velicer, 2003). A one-way MANOVA revealed a significant effect for condition on training 2 outcomes, Pillai’s Trace = .18, F(3, 199) = 14.76, p < .01, partial η = .18. Given the significance of the overall test, the univariate main effects were examined. Based on the Levene’s F tests, the homogeneity assumption was met for the three training outcomes. Significant effects for condition were obtained for feedback utility, F(1, 201) = 27.36, p < .01, 106 2 2 partial η = .12; training utility, F(1, 201) = 5.96, p < .05, partial η = .03, and knowledge, F(1, 2 201) = 14.08, p < .01, partial η = .07. These results confirm that those participants who were in the feedback training condition had higher training outcomes per feedback utility (M = 4.10; SD = .43), training utility (M = 4.48; SD = .46), and knowledge (M = 82.52; SD = 9.82) compared to training outcomes in the control condition per feedback utility (M = 3.62; SD = .85), training utility (M = 4.34; SD = .34), and knowledge (M = 75.99; SD = 14.74). Thus, the feedback training had a main effect on the quality of feedback, as well as other parts of the proposed model. Feedback training had an effect on the feedback receiver characteristic as well as a training outcome. These results indicate an interesting effect that feedback training can have on training processes and outcomes, which is discussed at length in the discussion section. The Effect of Trainee Characteristics on Outcomes Given the predictive relationship of self-efficacy and motivation found in hypotheses three and five and the positive correlation found in Table 4, a two-way MANOVA was conducted to examine the combined effects of trainee characteristics on the related training outcomes, feedback and training utility. Box’s M value of 44.15 associated with a p value of .82 was interpreted as non-significant (Huberty & Petoskey,2000). A two way MANOVA revealed a non significant effect of trainee characteristics, Pillai’s Trace = 1.03, F(150, 166) = .96, ns. To examine the positive relationship between self-efficacy and training outcomes, a oneway MANOVA was conducted for the independent effect of self-efficacy on training outcomes to understand possible Type 1 error inflation. The same test was not conducted for motivation, as it was not related to training outcomes (see Table 4). For self-efficacy, Box’s M value of 100.95 107 associated with a significant p value; as a result, Pillai’s Trace, was used as an alternative to Wilk’s lamda (Olson, 1979; Weiner, Freedheim, Schinka, & Velicer, 2003). A one-way MANOVA revealed a significant effect of self-efficacy on training outcomes, Pillai’s Trace 2 = .31, F(46,358) = 1.42, p < .05, partial η = .15. Given the significance of the overall test, the univariate main effects were examined. Prior to conducting the ANOVAs, the homogeneity of variance assumption was test for the three training outcome scales. Based on the Levene’s F tests, the homogeneity assumption was not met for feedback utility (p < .05). An examination of standard deviations revealed that the ANOVA would be robust in this case (Howell, 2009). Significant effects for self-efficacy was feedback utility, F(23, 179) = 1.66, p < .05, partial η 2 2 = .18 and marginally obtained for training utility, F(23,179) = 1.43, p = .10, partial η = .16. These results confirm the correlational relationships found, indicating that those participants who had higher self-efficacy had higher training outcomes. Effect of Peer Partner Given the nature of the peer feedback training, participants may have received some type of response-dependent reinforcement because feedback giving was required of both participants. Potentially, peer partners may have stimulated each other to give each other quality peer feedback (or less quality feedback). To evaluate this, the mean difference between feedback quality ratings, feedback frequency, and feedback elements was calculated between each partner (e.g. how frequent partner one gave feedback compared to partner two). Scores closer to zero would indicate peer partners had similar behaviors, either negatively or positively. A one-sample t-test evaluated if the scores were statistically different from zero, with a desired end result of non-significance to rule out the effect of response-dependent reinforcement. Zero would indicate 108 peer partner had an effect, because zero indicated no difference between the peer partner behavior. A one-sample t-test indicated that mean differences between feedback was greater than chance level of 0.0 for both feedback quality (M = .15, SD = .62), t(99) = 2.43, p < .05, and for feedback elements (M = .14, SD = .64), t(97) = 2.19, p < .05. These results indicated that both means were statistically different than the expected zero mean, which means there was no apparent effect of peer partner on feedback quality or feedback elements. However, there was a statistically non significant value for feedback frequency (M = .18, SD = 2.2), t(101) = .83, ns. This effect shows that peer partner may have influenced how many feedback statements were made. Ensuring the peer partner effect was not condition specific, differences between peer partners across the three feedback metrics were calculated (e.g. how frequent partner one gave feedback compared to partner two). Scores closer to zero would indicate peer partners had similar behaviors. These differences were used as the outcomes in a one-way MANOVA. Condition was used as the independent variable. Box’s M value of 32.37 was significant; as a result, Pillai’s Trace was used as an alternative to Wilk’s lamda (Olson, 1979; Weiner, Freedheim, Schinka, & Velicer, 2003). A one-way MANOVA revealed a significant multivariate 2 main effect for condition, Pillai’s Trace = .07, F(3, 94) = 2.8, p = .08, partial η = .06. Post-hoc tests were conducted to see if a similar effect of frequency carried over. Based on the Levene’s F tests, the homogeneity assumption was met to conduct post-hoc analyses. Peer partner frequency similarity was not significant for condition, F(1, 96) = 1.28, ns. Feedback element similarity was not significant for condition, F(1, 96) = 1.76, ns. However, peer feedback quality similarity was significant for condition, F(1, 96) = 5.62, p < .05. That is to say, in the training condition, peers were more likely have similar type of quality ratings indicating a relationship effect (ie putting in 109 as much as effort the partner does); whereas control condition the quality, peers were more likely to give feedback independently of how the partner gave the quality. These results indicate that there was some effect of the peer in the feedback training condition. Effect of Time Because the training consisted of five exercises, the effect of time was evaluated to see if feedback quality improved over time, which would reveal a practice effect. A repeated measures ANOVA evaluated the effect of time on feedback quality. First, feedback rating was evaluated. 2 Mauchly's Test of Sphericity indicated that the assumption of sphericity had been violated, χ (9) = 73.325, p < .0005, and, therefore, a Greenhouse-Geisser correction was used. There was a nonsignificant effect of time on quality rating, F(3.09, 321.04) = .67, ns. Second, feedback element was evaluated. Mauchly's Test of Sphericity indicated that the assumption of sphericity had been 2 violated, χ (9) = 327.99, p < .0005, and, therefore, a Greenhouse-Geisser correction was used. There was a significant effect of time on feedback elements, F(2.12, 405.62) = 162.58, p < .05. Resulting post hoc tests revealed that feedback elements decreased over time between exercises two and four, and then increasing in exercise five. Thus, there was a significant effect of time on feedback elements. Follow up tests evaluated whether condition related to feedback quality over time. A twoway repeated measures ANOVA was used. First, quality rating was evaluated. Mauchly's Test of 2 Sphericity indicated that the assumption of sphericity had been violated, χ (9) = 71.89, p < .0005, and, therefore, a Greenhouse-Geisser correction was used. There was a significant interaction effect of time and condition on quality rating, F(2.62, 494.18) = 5.29, p < .05. The graph of this interaction is show in Figure 5. The graph illustrates that in the quality 110 grew slightly from exercise one (M = 2.92) to exercise five (M = 3.03); however, the control condition was inconsistent over time starting with M =1.43, increasing in quality to M = 2.0 in exercise four, and then dropping back to M = 1.49. A discussion as to the effect found in lesson four is discussed later. Second, feedback elements were evaluated. Mauchly's Test of Sphericity indicated that 2 the assumption of sphericity had been violated, χ (9) = 136.37, p < .0005, and, therefore, a Greenhouse-Geisser correction was used. There was a significant interaction effect of time and condition on feedback elements, F(2.88, 405.62) = 127.45, p < .01. The graph of this interaction is show in Figure 6. The graph illustrates that elements that the two conditions were nearly a whole point different for exercises one and two, then became closely related in exercises three and four, and then diverged at the end. Thus, significant interaction of condition and time on quality was found.   111 DISCUSSION The purpose of the present study was to extend research on the ways to increase training effectiveness when trainees are placed in third generation enabled learning environments. The general question asked was to what extent and to what level of quality do participants give feedback in these peer learning dependent environments. The specific question asked was to what extent does a purposeful intervention designed to increase peer feedback quality enhance the effectiveness of third generation learning environments. The conceptual model in Figure 1 illustrates the overarching hypothesis: a purposeful intervention on peer feedback increases the quality and quantity of feedback, resulting in higher receiver characteristics of efficacy and motivation and higher levels of training outcomes. The model also proposes that feedback environment interacts with the feedback training, and that feedback orientation interacts with the quality of the feedback. Finally, a number of individual characteristics are seen as mediating the relationship between quality and training outcomes. Based on the outcomes, a revised conceptual model is introduced through this discussion. The discussion of this study is divided into three parts. First, there is a summary of the study’s results. Second, the implications of the study are presented along with discussions for future research. Finally, the limitations of the study are discussed. Summary of the Results The Effect of Condition Assignment The effect of the condition was tested in hypotheses one and two. The results revealed that the peer feedback training worked as was intended: feedback rating, feedback frequency, and feedback elements were all higher in the manipulation (feedback training) condition versus the control. In fact, the average frequency of feedback statements given in the control (M = 3.61) 112 was lower than the required, five feedback statements needed to complete the course. That is to say, participants in the control condition did not complete all required feedback giving exercises. These results may be explained by the lower motivation and self-efficacy in the control condition. Additional follow up tests revealed a direct effect of the peer feedback on training outcomes such that they had higher feedback utility, training utility, and knowledge. These results are consistent with previous research that peer feedback training can affect training processes and outcomes. For example, the present study found that peer feedback training recipients gave more feedback, which is consistent with the effect of Zhu (1994) who found that participants trained in peer feedback training engaged in peer revision more than those participants who did not receive the training. Similarly, consistent with Sluijsmans and colleagues (2002) who found that peers were more likely to use the feedback when given feedback training, the present study found that participants in the feedback training condition found the feedback more useful. However, unlike previous studies (e.g. Birkeland, 1986), the peer feedback training condition resulted in higher outcomes of knowledge. This is one of the first studies to quantify the results of quality peer feedback exclusively on a training outcome higher than level one (Kirkpatrick, 1960) in third generation learning environments. The reason for this change may be related to the nature of the knowledge test, which included questions that were more practical in nature or related to on the job application (e.g. question ten) versus reiteration of factors or theories. With this in mind, the results of the study align to constructivist theories (Leidner & Jarvenpaa, 1995) and proximal development theories (Palincsar, 2003), the foundation for third generation learning. Recapped, these theories note that, as participants interact with each other, they contextualize key learnings that make more sense for the learner. For example, as the 113 participants learned about creating a course in lesson five, the participants shared their own ideas for their courses – putting the context around the lesson objectives and theories. This may have resulted in the increased knowledge test scores. Similarly, the main effects of the peer feedback training may be a result of the content of the peer training. The peer training consisted of an emphasis on the value of social learning. This laid the expectation that the training they were entering would be filled with social learning activities. Conversely, the control group was less likely to understand the value of the social learning environment, resulting in expectations not being met. Meeting participant expectations for training has been found to enhance learning outcomes (e.g. Hilling & Tattersall, 1997; Rowold, 2007). For example, Tannenbaum and colleagues (1991) hypothesized that naval recruits who indicated their expectations were met would perform higher on the training task. The results found that self-efficacy and training motivation were highly correlated for those who had their expectations fulfilled. Second, the peer feedback training also illustrated why peer feedback is beneficial to their learning experiences. Ensuring participants see the potential benefit to their training has been found to increase motivation (Martocchio, 1992). Martocchio suggested that a trainee’s pretraining attitudes predicted the ability to learn in a computer-training task. Those participants who believed the training would be beneficial actually learned the information more than those who had a poor attitude toward the training. Thus, the present study adds more credence to the importance of incorporating peer feedback training in third generational learning environments. Not only does it increase peer feedback quality, but also it may increase trainee motivation, self-efficacy, and learning outcomes. It is also important to note that this study purposefully went against the suggestion of 114 previous researchers to isolate the peer feedback training (see Sluijsmans et al., 2002), who argued that isolated training would degrade the importance of delivering feedback consistently throughout a course. This study contradicts that proposition. The Effects of Quality Feedback Hypotheses three, five, and eight tested if quality feedback would lead to higher training characteristics and outcomes. Results indicated that participants who received higher quality feedback from peers had higher levels of motivation and self-efficacy. These results are some of the first to document quantitatively how quality peer feedback can impact trainees in third generational learning environments (Kraiger, 2008). Higher quality feedback also led to higher levels of knowledge. As demonstrated in hypothesis one, peer feedback training resulted in higher quality feedback; but the effect of that quality did not directly affect outcomes. There are several reasons why this may have happened. First, the behavioral measure may need further refinement. Coupled with moderate reliability and positive skew, the measure may not have been as clean as desired to make an accurate judgment on the effect of peer feedback training on behavioral outcomes. Given that the knowledge test did have elements of behavioral outcomes and knowledge was related to peer feedback training, this indicates the behavioral measure may not have captured the intended behavioral outcomes. Second, behavioral outcomes in peer feedback training may be less affected directly by quality feedback. This study aligns to previous studies that found that while other outcomes or characteristics improved, measures of improvement in behaviors remain elusive to find (e.g. Li & Steckelberg, 2004). Quality feedback may work through other mechanisms such as awareness of the learning process (see Falchikov, 1995; Orsmond, Merry, & Reiling, 2000). 115 Finally, the definition of quality feedback may need further revision. The study proposed that five principles would lead to better quality feedback. The study utilized three measures of feedback quality: rating, elements, and frequency. This is one of the most comprehensive measurements of feedback quality that has been seen in peer feedback literature. All three feedback quality measures did not operate the same way, nor did they relate to training outcomes in the same way. As noted, feedback rating, the expert rating of peer feedback by trained judges, accounted for the majority of the variance explained by quality in self-efficacy and motivation; and while the peer feedback quality and element were strongly correlated, the feedback element rating was lower than the feedback quality rating in nearly 75% of all cases. A closer evaluation of the feedback elements reveals some interesting avenues for future improvement of quality definitions to find training outcome relationships. The elements were based on the five principles of feedback giving. Table 5 presents correlations between each feedback element and the four training outcomes measured in this study. Briefly, each element is discussed as it related to the principles, as they in themselves may have caused the nonsignificant relationship between quality and outcome. Sign. Element one ensured participants did not always give positive feedback. The correlations reveal that element worked in the opposite direction as proposed. That is, participants who found the feedback useful received feedback that is more positive. This makes sense in light of self-regulatory theory and trainee characteristic literature (Idson & Higgons, 2000; Kluger & DeNisi, 1996). It makes even more sense in light of peer feedback literature, which notes that positive feedback elevates confidence (Stone et al., 1984). Type of Feedback. Elements two and three focuses on ensuring that the feedback given is about improving the process or task specifically – rather than person-specific feedback. Out of 116 all the elements, element two had the most relationships to training outcomes. These elements related to outcomes and in the direction intended, which is consistent with previous feedback research (e.g. Tenenbaum & Goldring; Walberg, 1982). More importantly, researchers continue to note that peers are highly susceptible to delivering person-specific feedback, which is nonactionable (i.e. Cheng & Warren, 1999; Falchikov, 1995; Sluijsmans et al., 2001). This study confirms that it is possible for peers to overcome this with proper training. Specificity. Element four and five both refer to principle four, focusing on being specific. Similar to element three, this element related to the knowledge outcome. This is one of the first studies to specifically illustrate that specific peer feedback can drive training outcomes, as others have focused on peer performance ratings (e.g. Xiao & Lucking, 2008). This finding is consistent with traditional feedback research (Ilgen et al., 1979; Kluger & DeNisi, 1996). However, this highly specific feedback could explain the non-significant relationship to the behavior outcome. Because learners need to generalize the knowledge to create their own training program (i.e. the behavior outcome), learners may have needed more general feedback (i.e., not “spoon fed”), which would ensure sufficient cognitive processing and exploration (Goodman & Wood, 2004). More research is needed on this element to understand its role in peer feedback. In summary, four of the five elements worked as intended. Further research is needed to understand the role of sign; and whether or not sign training should fit into the training program at all. The sign element worked in the opposite direction of the other elements and thus an inconsistent pattern in behavioral patterns emerged, which may have accounted for the lack of relationship between quality and training outcomes. 117 Interactive Effects of Individual Traits Hypotheses two and seven investigated whether the work environment (feedback environment) or characteristics of the trainee (feedback orientation) affected the results of the peer feedback training. Results indicated that there was a non-significant effect for the feedback environment on the effectiveness of the peer feedback training and a non-significant effect for feedback orientation on trainee characteristics. These results are surprising in light of the research to suggest otherwise. Feedback Environment. Feedback environment did not moderate the effectiveness of the peer feedback training. This was one of the first studies to include peers specifically as an element of the peer feedback environment, which is one potential reason why the measure used was not able to capture the full 360-degree feedback environment. One potential explanation for this is the relative high feedback environment that existed for all participants. Another explanation is the virtual nature of third generation training may be resistant to the effects of feedback environment differences (Arbaugh, 2008). Finally, another reason for lack of interaction is that the feedback environment judged the employee’s current work group. The training course included peers outside the learner’s workgroup. Thus, the measure may not have captured the right audience. Regardless of the explanation, however, the non-significant results do suggest an important finding in itself: organizations universally can benefit from peer feedback training interventions regardless of feedback environment on the job. Feedback Orientation. Feedback orientation did not significantly interact with the quality of the feedback and trainee characteristics. Interestingly, feedback orientation did relate to feedback elements and training characteristics. These results make sense because feedback orientation impacts someone’s motivation and confidence (Levy et al., 1995; Northcraft & 118 Ashford, 1990). In many ways, the non-significance of this interaction can be viewed as a positive effect for organizations because the results indicate relative unimportance of individual traits in increasing training effectiveness (see Nease, Mudgett, & Quiñones, 1999). This is one of the first studies to evaluate feedback orientation in peer feedback environments. As such, this study relied on a traditional measure for this variable. After reviewing the items, the scale made reference to “supervisors” and the “company” and had little reference to peers (Linderbaum & Levy, 2010). Thus, going forward, a feedback orientation scale related to peers may be needed to find an effect for feedback orientation. Mediating Effects Hypotheses four, six, nine, and ten investigated the mediating relationships of the training characteristics on quality and training outcomes. Quality of peer feedback was found to mediate the relationship between condition and self-efficacy and motivation. Additionally, self-efficacy was found to mediate the relationship between quality and two training outcomes. Motivation was found to mediate the relationship between feedback quality and condition. This is one of the first studies to specifically look at mediating trainee characteristics in peer feedback environments and supports emerging literature that illustrates the mediating effect of these characteristics in training environments (e.g. Blume et al., 2010; Tracey et al., 2001). Conclusions Considering all the relationships and effects found for peer feedback training, the presented solution to improve third generation learning appears to increase training effectiveness on many fronts. The resulting model of the study is found in Figure 7. Overall, the model illustrates that peer feedback training improves feedback quality and increases feedback quantity. Much of the initial proposed model was retained. Most significantly, differences appear with the 119 direct impact the peer feedback training had on the variables in the model and the lack of interactive effects of individual traits. Implications for Practice & Directions for Future Research The results of this study have several implications for future researchers and practitioners who wish to implement third generation learning within their environment. These relate to four areas: (1) third generation learning environments, (2) training effectiveness, (3) peer feedback training, and (4) feedback research. Third Generation Learning As noted in the introduction, third generation learning is a proposed evolution of learning that places the individual at the cultural and social center of the learning process (Kraiger, 2008). It is seen as a critical component of online learning environments (Arbaugh, 2008; Moore, 1989) and rests on many untested assumptions. Overall, this study was one of the first to evaluate the effects and possibilities of third generation learning. The resulting model (Figure 7) can act as a future basis to continue to test the effects of third generation learning. Quality feedback is just one component. Researchers should continue to evolve and improve the model to encompass other elements of third generation learning. There are two other recommendations for practitioners with some direction for future research: (1) peers need to be trained, and (2) peer feedback should be encouraged in all pieces of the course. Peers Need to Be Trained. One of the assumptions Bedwell and Salas (2008) raised was the plausibility of third generation learning working, given that students do not know how to 120 share knowledge and give feedback. This study emphasizes that this assumption cannot be overlooked: learners in these environments must be trained to give feedback. In the current study, participants who were given the third generation learning environment without proper training did not give feedback with as much frequency, nor as much quality as the manipulation group. In fact, in many instances, the feedback exercises were not completed – students simply put their answers to the exercise topic, instead of giving feedback to each other on how to do the exercises. This leads to many interesting effects: (1) increased instructor workload and (2) low perceived return on investment from students. Instructor Workload. Arbaugh (2008) argues that third generation learning alters the role of the instructor. Because peers share knowledge and feedback, the instructor acts as a course facilitator when discussion dissipates and when corrective direction is needed. In the cases that peer feedback training was not given to participants, course facilitators noted that this learning environment caused exhaustive work. A course facilitator who was sent a technology satisfaction survey reported this effect: “Composing written answers takes much longer than having dialogue on the same subject (and is more open to misinterpretation). Also, in a classroom, everyone …discuss a few examples, but every single person would not have to respond, and then as an instructor have to respond to every participant (which I feel obligated to do in this kind of environment). It takes too long and ends up redundant.” This highlights a primary concern Bedwell and Salas (2008) noted: instructors must equip the students with skills. Otherwise, students will not freely exchange and respond to each other. Instead, as seen here, instructors will need to respond to each student. In a class that can have hundreds of students, this can be an exhausting task. Return on Investment. Kraiger (2008) noted that this type of learning environment is a natural progression for learners and many researchers have argued that this type of learning is 121 desired (e.g. Li & Steckelberg, 2004). However, without the peer feedback training, students were less likely to view the training as favorable. Beyond the lower utility reactions (see Table 4), learners may view a lower return on investment for the learning. For example, if the learner posts an exercise and never receives feedback on the posting, what was the purpose of the time it took to write out the exercise? Topping and colleagues (2000) noted that peer feedback was seen as time-consuming. A student who participated in the control condition who was sent a survey on the technology satisfaction reported something similar: “I do not want to take another course on Blackboard because it took more time for me to complete the course. Filling out postings for every exercise did not seem like a good use of my time.” However, in the peer feedback group, feedback was perceived as more useful and increased the relevance of the course content (Liu, Lin, Chiu, & Yuan, 2001). A student who participated in the manipulation who was sent a survey on the technology satisfaction summarized this: “The interactivity [with my partner] was amazing. I was able to interact … at my own time. And then I could actually apply what I learned because of the discussions I had.” Thus, in order to win employees with return on investment, the peer feedback must be of high quality. Black and William (1998) noted that return on investment was perceived higher in these environments. This study illustrates that effect and adds that return on investment from employees is reliant upon the quality of feedback interactions. Peer Feedback Should Be Encouraged in All Facets of the Training. This course was designed specifically to exploit the Blackboard environment. As such, it included a variety of peer-to-peer learning exercises that were not directly graded that could have influenced results. For example, learners blogged about their weekly “ah-ha” moments. While analyses of blogs was not in scope for this study, participants in the feedback training condition had 90% 122 participation in the optional blogging activity versus 40% in the control condition. This is an interesting finding because this type of meta-cognitive activity in itself can influence the overall learning experience. Wong, Wood, and Goodman (2012) noted that meta-cognitive activities include reflection on learning and reflecting on goals, especially when partnered with high specificity feedback. This effect actually deepens learning outcomes (Ericcsonn, Krampe, & Tesch-Romer, 1993; Wood et al., 2012), which may relate to the knowledge outcome result found in this study. Given that meta-cognitive activity increases learning outcomes, it makes sense that knowledge outcomes are affect, as in this study. Thus, peer feedback training can enhance other indirect places of peer-to-peer conversation – and should be encouraged. Training Effectiveness This study highlights another way to improve training effectiveness, at least in the context of third generation learning environments. Some important considerations however emerged: (1) the role of trainee characteristics impacted by training programs, (2) trainee characteristics affecting training outcomes, and (3) the role of the work environment. Trainee Characteristics Affected by Training Programs. Tannebaum and colleagues (1991) demonstrated the importance of motivation to learn on training effectiveness; similarly, Quiñones (1995) demonstrated the importance of self-efficacy on training effectiveness. Individuals who have increased motivation and self-efficacy tend to master more of the training information presented, which translates into increased knowledge and behavioral transfer. However, in this study, the combined effect of motivation and self-efficacy was not found. Instead, self-efficacy was a significant predictor of training outcomes, which links more to competence than motivation. Nonetheless, motivation was still an important variable for quality ratings. 123 The results of this study emphasize the importance of motivation and self-efficacy. Quiñones (1995) noted that trainers should be aware of the factors that can decrease motivation, and Gist (1989) highlighted the importance of self-efficacy. This study confirms these findings and also brings training research in a new direction. What is the impact of these characteristics in third generation learning? This study confirmed Topping (2000)’s proposal that third generation learning can positively impact trainee characteristics. However, only self-efficacy mattered in it relationship to training outcomes. Researchers should investigate if motivation has a unique effect in third generation environments. More specifically, the role of feedback in these environments may yield interesting effects. For example, this study focused on ensuring negative feedback was presented. This may have a detrimental effect on motivation when it comes from peers. Training Characteristics Affecting Training Outcomes. Feder (1991) stipulated that feedback traits may inhibit or enhance a learner’s ability to accept feedback. This study investigated two of those, feedback orientation and feedback environment. Both were found to be non-significant moderating variables. However, this reveals a contradiction to a recent research direction: Nease, Mudgett, and Quiñones (1999) proposed that organizations need to be more aware of individual differences that impact training effectiveness. Two critical components that were hypothesized to impact training effectiveness did not in this case. It is possible that third generation learning environments become less affected by individual differences because third generation learning environments by design are more tailored to individual preferences (Kraiger, 2008). However, researchers need to continue to examine individual traits affecting the effectiveness of this training environment. 124 A secondary research direction emerges from the non-significant moderating effect of feedback orientation and environment: can third generation learning environments alter feedback receptiveness? This study proposed that feedback orientation was a trait. However, some researchers have argued that there are factors that can enhance or inhibit the receptivity to feedback. In fact, Callister, Kramer, and Turban (1999) suggested that feedback-seeking behavior is dependent on the sources involved. For example, explaining why individuals may seek more feedback from instructors rather than peers, Callister et al. suggested that feedback seeking is primarily an impression management mechanism. Perhaps in third generation learning environments, feedback orientation becomes less static – or in the case of this study, participants may become more aware of their deficiencies going through the feedback training. Thus, a retroactive feedback orientation scale may have been more appropriate. More research is needed on the nature of feedback orientation, especially in the context of third generation learning. The Role of the Work Environment. Heimbeck, Frese, Sonnentag, and Keith (2003) and Joung, Hesketh, and Neal (2006) noted that the work environment is a key component to the success of training interventions. While the work environment was not explicitly measured, the work environment may have had some impact on the results. Examination of Figure 6 notes a decline in feedback elements in lesson four. Instructors hypothesized that work demands influenced a decline in elements. That is to say, because lesson four was nearing the end of the course, students may have felt pressure to finish up the course in light of upcoming work priorities and focused on making the last lesson count. This sentiment was seen in one student comment: “I found the exercises good but towards the end I was running out of time.” This comment illustrates that third generation training designers should be cognizant of learner demands and work demands. The effects of third generation learning environments with varying 125 work demands need to be tested. Furthermore, researchers should investigate further how work context may moderate or mediate the results of third generation learning environments. Related, the instructors of the courses in this study were well versed in the nature of third generation learning, given the nature of their roles within the organizations. Researchers should investigate the differentiating skills needed for instructors in third generation learning Peer Feedback Training This study emphasizes the need for peer feedback training within a third generation learning environment. As noted in the introduction, this is one of the first studies to utilize traditional feedback research based on Ilgen and others to increase feedback quality in third generation learning environments. In fact, this may be one of the most systematically documented research studies dealing with the process of giving feedback holistically, instead of elements of feedback trained on individually (e.g. Kluger & DeNisi, 1996; Kang, Oah, & Dickinson, 2003). This is an important contribution to the literature and for practitioners who want guidance on how to train on feedback giving (Dolezalek, 2004). It also leaves two important research questions: (1) what elements of feedback should be trained? and (2) when should the training occur? And one important implication for practitioners: peer feedback research should include rationale for social learning. Elements of Feedback Training. While the primary intent of this study was not to evaluate the best way to train employees on giving feedback, it does yield interesting results that should spark further research. In totality, the feedback training was largely effective as highlighted by a quote from a learner in the manipulation condition: “This really opened my eyes to the value that feedback can have and the importance in the delivery of feedback. I have never before looked at feedback as a responsibility of mine to provide good feedback to my peers/manager and make sure that I deliver it in 126 such a way that has an impact. Another key take-away that I had from this lesson was the importance placed on immediate feedback. I have often received feedback well after the fact, and it is often too late to make any noticeable changes.” This quote highlights the favorability of the feedback training, but were pieces of the training more effective than others? The best way to examine that is to examine the feedback elements. Investigating the individual ratings for each feedback element rated in the manipulation condition reveals an interesting pattern: almost 60% of participants never gave a reason for recommended actions to take at least once, and nearly 40% of participants never told what to do to improve at least once. On the flip side, nearly 100% of participants in the feedback training group went beyond praise for actions well done at least once, and nearly 90% of participants in the manipulation group referred to specific words or actions at least once. Not only does this deeper investigation reveal the underlying cause for the feedback element rating to be lower than the feedback quality, it reveals an outage in the feedback training. While the training did focus on those two elements, subsequent feedback training researchers should investigate how to improve on the delivery of these two elements. Timing of Peer Feedback Training. Secondly, this study made an explicit choice to train learners at the beginning of the training design. This was done to simplify the study and to enhance the main effect of the training on training outcomes. However, Sluijsmans and colleagues (2004) called for trainers to place the feedback training as an embedded component of the overall training. In fact, they cautioned against placing feedback training as a specific module, at the risk of the training feeling awkward or lost in later training context. A learner who posted in the penultimate exercise highlights this issue: “My next question is a little in jest, but I have been thinking back through this Blackboard experience, and I'm having trouble connecting the Feedback module we did… with everything else we've learned so far. Please don't get me wrong. I learned a lot 127 from the materials, but I [forgot] why we took so much time in [that exercise] rather than jumping right into the meat of this session.” This quote highlights the problem previous Sluijsmans proposed. Regardless, this study used the feedback training as a separate module and, as a result of this choice, had significant results of the peer feedback training. However, the results of quality of peer feedback on training outcomes were not as hypothesized. Perhaps a combined approach of a separate module plus an integrated approach may be the best. Re-examination of Figure 6 gives credence to this mixed approach, as feedback elements were highest at the beginning of the course and then dissipated. More research is needed to determine the exact timing of the intervention to be the most effective. And, for practitioners, this issue highlights that the peer feedback training may need to focus more on a blatant connection directly to the third generation learning environment, so participants see value in improving their feedback in third generation learning environments. Feedback Research Although a topic heavily researched (Latham & Locke, 1991), this study illustrated there is still work to be done in the area of understanding feedback. Two main themes of research direction came from this study: (1) clear definition of quality feedback, and (2) the role of the source needs more specificity. Quality Feedback Definition. As noted in the introduction, previous studies have shown positive impacts of feedback on learning outcomes and while others have not; some studies recommend frequent feedback; others do not… the effect is found across all aspects of feedback. This inconsistency in what defines quality is highlighted in the results of this study. This study utilized three variables to define quality feedback. They operated in different ways, which emphasizes the inconsistent nature of feedback studies (Kluger & DeNisi, 1996). 128 Thus, instead of finding a universal definition for feedback, it may be best for researchers to take into account training context (e.g. Martocchio, 1992) to define quality feedback. In this study, the training context was determined first, and then principles for feedback were derived based on the research. Perhaps, there are specific guidelines for feedback that can be associated with the type of training, the learners, the learning goals, etc. A learner in the manipulation condition highlighted this: “After reviewing…the principles [on feedback], I questioned the reasoning behind some of them. This seems to be contrary to what I’ve been taught in other feedback courses... Does this apply to only this course?” This quote, while highlighting that there are many misconception on quality feedback (e.g. sandwich feedback), illustrates perhaps researchers should move to a malleable definition of quality feedback. Thus, it may be helpful for researchers create a decision grid for quality feedback (e.g. if you train motor skills and have unfamiliar peers, use this specific definition of quality feedback). For example, if the results of this study are indicative of most third generation learning environments, quality feedback would be feedback that is specific, timely, and descriptive (not necessarily sign specific). Role of Source Requires Further Investigation. This study highlights and confirms many positive hypotheses about peers in learning such as improved learning satisfaction (Xiao & Lucking, 2008), but also highlights the role of that individual peer. Results for frequency were dependent on peer behavior, and in the peer feedback training condition, the quality of the peer feedback partner closely mirrored that of the peer partner. While this could be due to the nature of being in the manipulation condition (discussed as a limitation below), this may also be related to the relationship that is between the two partners. Ng and colleagues (2011) noted that who the peer is in relation to the other peer could impact the feedback. For example, peers having 129 dissimilar genders, ethnicities, and personal preferences may impact the peer feedback. While these variables may be less visible to the participants, age may impact outcomes in these environments as a younger generation may have more exposure to online learning versus older. Unfortunately, this study did not have enough volunteered demographic data to investigate this effect. Further research is needed to investigate this effect. Additionally, familiarity of the peer is an important factor. For example, MacLeod (1999) and Schaffer (1996) found that students were concerned about delivering negative feedback to friends who were classmates. This study had primarily new-to-role employees so familiarity was less of an issue. But in some cases, although not statistically measured, it appeared peers knew each other. For example, comments such as, “Great to hear from you again!” implied the peer partners had at least had contact with each other. While this occurred in less than 10% of peer partners in the manipulation condition, the quality rating of the peer feedback for this peerfamiliar sample was M = 3.64, which is .8 less than the overall group average. This finding confirms Falchikov (1995)’s finding that social familiarity influences the results of peer feedback and emphasizes the need for more investigation. Third generation learning environments may need regulations regarding class demographics. Study Limitations This study has limitations, which must be taken into account when interpreting the results and design of future studies. First, both feedback givers and feedback receivers in the manipulation received the peer feedback training. As a result, there may be an unaccounted treatment effect. For example, participants may have been more receptive to negative feedback 130 because they were told to give negative feedback. Future studies should attempt to isolate the effect of the training on just feedback givers. Second, the nature of demographic variables was relatively unknown. These variables may have implications for future application. For example, Blume and colleagues (2010) found that age and gender related to training outcomes. This may be especially important, given the organization did not track age. Given the nature of online learning and its dependence on technology adoption, age may be an important factor to consider when evaluating the effectiveness of online learning. For example, younger employees may be more accustom to online learning environments versus older employees. Third, this training was voluntary in nature, which has implications for motivation, selfefficacy, and training outcomes (Blume et al., 2010). Much training within organizations is mandatory. As such, future research should investigate the nature of required training with peerto-peer feedback to determine whether third generation learning would work as well. Finally, third generation learning is typically conducted in teams (Kraiger, 2008). This study took a dyadic approach to isolate the effects of the intervention. The very nature of a dual relationship allows for more intimate relationship building and/or a competitive spirit. These may have accounted for some of the results. Future research needs to evaluate whether larger groups or teams would interact in similar ways to generalize to common third generation learning environments.   131   PRACTICAL IMPLICATIONS & CONCLUSIONS   The intent of this study was to improve training effectiveness within environments that are third generation enabled. Organizations and practitioners should interpret the results with both optimism and caution as they plan their future learning strategy. First, with the words of caution, this study sheds some doubt that third generation learning occurs without strategic organizational planning. Kraiger (2008) hypothesized that third generation learning is a natural evolution, but this study suggests that evolution may need some instigation. While the end result of third generation learning may ultimately lead to a less organizational, employees in today’s workforce may be too overworked or too under skilled to give knowledge and feedback naturally. This study showed that in a natural environment without strategic planning employees did not contribute completely nor did they have the skill to give proper feedback. However, this study gives organizations optimism. Third generation learning can indeed happen when employees are equipped with the proper skills. A very short, targeted intervention that illustrates how to have meaningful, impactful interactions in these environments can have a significant influence on the results of the training program. This relatively cheap investment not only will benefit the interactions but also may have impacts on meta-cognitive activities within the course. In fact, the organization with which this study was piloted has now made the intervention a routine part of their own third generation learning environments. There is still much about third generation learning that needs more investigation. Ultimately, third generation learning will soon be a very hot topic within organizations. The role of the instructor, the role of individual characteristics, and the relationship to work environment are all major considerations that organizations will need to assess until researchers can 132 investigate these areas further. With this study, organizations now can be empowered to deliver third generation with positive training outcomes. 133 APPENDICES 134 APPENDIX A: CONSENT FORM This study investigates the effectiveness of specially designed training programs in improving training design in online classrooms. You will participate in your regularly scheduled training as normally would but have been asked to provide some additional data to help us understand how to better improve training experiences. This training will be used as part of a research study with Michigan State University to investigate training effectiveness. You will be asked to spend approximately 10 hours on this training research study. Your participation in this study is strictly voluntary and you may refuse to answer any questions, which you may find inappropriate without any penalty. In addition, you can discontinue the experiment if you feel it necessary to do so. Your individual results in this study will be CONFIDENTIAL. You will be assigned a unique eight-digit number which will appear on all questionnaire forms. The experimenter will only report the final data in an aggregate form, which does not allow any particular individual to be identified. You will not directly benefit from your participation in this study. However, your participation in this study may contribute to the understanding of online learning and improving the future of learning. There are no foreseeable risks associated with participating in this study. Your name and information will remain confidential. Your privacy will be protected to the maximum extent allowable by law. The data will be saved for the duration of the study and will only be accessible by the primary investigator and two additional researchers. You may refuse to participate in certain procedures or answer certain questions. Participation is completely voluntary. You may choose not to participate at all and may discontinue your participation at any time without penalty or loss of benefits. The privacy of your personal information is important and your confidentiality will be protected to the maximum extent allowable by law. This survey will be administered, collected, and analyzed by doctoral students at Michigan State University researchers, an approved research partner based in the United States. Any responses that you give are intended to improve the effectiveness of the class, the instructor(s), and other resources for learning. No answer that you give will be used to identify you personally. The collective data will be shared with the Learning & Development community responsible for the design and delivery of the course, and with other leaders with a business need to view the data. Your agreement with this statement and subsequent participation in the survey are voluntary. The data will be stored electronically for three years after the research closes at the primary investigator’s office at Michigan State University. No other organizations or Institutional Review Boards other than Michigan State University’s board and our organization will have access to the data. Conflict of Interest Disclosure. Please note that the project coordinator is currently employed at the current organization, and the data used is meant to complete dissertation requirements. The primary investigator shall ensure fair and proper treatment of the data as to not bias the results or favor any party in the research study. 135 If you have any questions or concerns about your participation in this project, you can reach the primary investigator, J. Kevin Ford by phone: 517-353-5006 or by regular mail, 315 Psychology Building, Michigan State University, E. Lansing, MI 48824; or the project coordinator, Adam Massman by phone: (512) 876-7691, email: massmana@msu.edu, or regular mail: 302 Psychology Building, East Lansing, MI 48824. If you have questions or concerns about your role and rights as a research participant, would like to obtain information or offer input, or would like to register a complaint about this study, you may contact, anonymously if you wish, the Michigan State University's Human Research Protection Program at 517-355-2180, Fax 517432-4503, or e-mail irb@msu.edu or regular mail at 207 Olds Hall, MSU, East Lansing, MI 48824. By continuing to the next step of completing the survey, you voluntarily give your consent to participate in this study. 136 APPENDIX B: PEER FEEDBACK TRAINING MODULE Peer feedback training module was given during a live Cisco WebExTM training. The training was delivered using a PowerPoint deck with various interactive pieces. Overall the course lasted 45 minutes. The peer feedback training module’s outline is listed below. Slide 1. Introduction: “fun with peer feedback” part of the Investing for Leadership series Slide 2. What we’ll cover today Slide 3. Lesson 1: Let’s talk about social learning a. What is social learning? b. Your opinion? What is social learning? Slide 4. We have become digital a. In the 18th century, you could pride yourself by saying you have read every book ever written… today, it would take you 15,000 years to read through every book in a Library – Stephen Hawking b. The rate of information shared is extremely fast c. “Technology is shifting power away from the editors, the publishers, the establishment, the media elite…. Now it’s the people who are taking control” – Rupert Murdoch (July, 2006) Slide 5. Welcome to Social Learning a. Social learning allows learning to go in new directions b. Peer to peer interactions are critical components c. Instructors are more like facilitators of your learning journey d. Apply it to your job more easily e. Bring in relevant examples from your job for other peers to benefit from f. Learn more about your own capability through providing feedback to others Slide 6. What are the components? Slide 7. Multiple Choice Quiz in Blackboard Slide 8. Lesson 2: Why does feedback matter? Slide 9. Your opinion? a. Now take the direct report(s)’ POV – why does feedback matter? Let’s look at some data… b. Employees who get regular feedback succeed! Employees who have a trusted advisor/mentor or strong relationship with manager show a 15–30 point advantage across virtually all aspects of the employee experience on the annual employee survey. Slide 10. How do you feel about feedback? a. Think of a time someone gave you feedback and you found it helpful or useful? b. What did they say or how did they say it? c. In other words, what made it memorable? d. Why is it hard to give feedback? e. why does feedback matter? Slide 11. Control Theory a. The notion that we have a goal and a current state… the only way we know the difference / distance between these is through feedback b. In other words, feedback is critical to change, development, and performance 137 Slide 12. What type of feedback do you typically get? a. Encouraging short feedback (“great job!”) b. Sandwiched feedback (“Great job on X. Improve Y. But great job on Z”) c. Timely constructive feedback (“Yesterday’s presentation where you talked about X could have been improved by…”) d. Short negative feedback (“Fix X.”) e. No consistent approach. The feedback I get varies. Slide 13. So… how can you give “good” feedback? a. Feedback is delivered immediately following a learning exercise. b. Feedback is a mix of positive and negative statements. c. Feedback is provided regularly. d. Feedback is as specific as possible. e. Feedback points to ways for the learner to achieve their learning goal. Slide 14. Principle 1. Feedback will not always be positive a. Feedback is generally either positive or negative b. Positive: praise, letting people know what they have done well c. Negative: constructive criticism, letting people know where they have room to improve Slide 15. Which gives you more information? a. What are some good or bad experiences you have had with negative feedback? Slide 16. Good examples of signed feedback a. The document you uploaded to the discussion board includes a paragraph on our company WWBCM. You didn’t represent the most current version of our WWBCM. b. The link you posted as an example of good leadership behaviors did not seem to relate well to the learning objectives of the course. Slide 17. Bad examples of signed feedback a. I thought that your essay was really good. The second paragraph seems to need some reworking. But the conclusion was perfect! Great job! b. Everything was really great. The entire proposal was just great. Slide 18. Principle 2. Feedback will be delivered immediately following the learning exercise. a. When is the best time to give feedback? b. Best Answer! Immediately after the behavior when it’s fresh in someone’s mind and allows them time to course-correct for next time c. Have you ever received feedback long after you did something? How did it feel? Slide 19. Principle 3. Feedback will help the learner achieve their learning goal. a. Person-Specific Feedback: Feedback should always focus on the behavior, not the person b. Positive: children whose abilities are praised tend to be less resilient to failure; those whose effort is praised are more persistent in the face of failure c. Negative: feedback targeted at the person feels like a character attack and is more likely to be dismissed and/or make the person defensive d. Other types: motivating, information, helps people get to their end goals Slide 20. Principle 4. Feedback will be as specific as possible a. Have you ever received feedback you couldn’t act on? b. Specificity refers to how specific or general the feedback is 138 c. There is a continuum of specificity of feedback d. Highly specific feedback gives exact errors that were discovered which would help an individual achieve his end goal quickly e. Low specific feedback gives very general statements which can have overarching statements Slide 21. Good examples of specific feedback a. In the third paragraph of your document, you start referring to your project plan. You should consider using the PACE model. b. In your example of the Lesson 2 Discussion board, you said you had a coaching moment with an employee and said, “…” I think given what we learned, you should have said, “…” because… Slide 22. Bad examples of specific feedback a. I think there needs to be some more thought to your business proposal. I didn’t quite understand it. b. Have you considered rewording the posting… because it seemed to be North America focused Slide 23. Good examples of specific feedback a. You already know how to deliver great visuals. Make sure you’ve executed the visuals to the level you typically do in your presentation. b. Your presentation you are giving might be more effective if you utilizes the strategies and processes we learned in the last lesson. Slide 24. Bad examples of specific feedback a. You are a great leader b. You are a quick learner c. I think its great because you are great! Slide 25. Principle 5. Feedback will happen regularly a. Regular feedback helps learners understand how they are doing throughout the course b. The more regular the feedback the more helpful you will be to your learning partner Slide 26. Good examples of timely, regular feedback a. Your posting yesterday reminded me of this other article that you might find beneficial to the questions your are struggling with b. I just found this article that would relate perfectly to the topic you are describing Slide 27. Bad examples of timely, regular feedback a. Your posting last week reminded me of this seminar I attended a year ago b. I was catching up on my email last night and realized I missed your posting… that was off! Slide 28. List 3 reasons why giving effective feedback is important to achieving your goals (Pause for this unsupervised reflection exercise) Slide 29. Lesson 3: Giving Effective Peer Feedback a. Social learning b. Giving feedback is a critical component of social learning c. During this course you will be giving feedback to your peer partner all along the way Slide 30. Learn to love feedback! 139 a. b. c. d. Learn to give effective feedback What to do Be an active feedback giver There are no best practices for how many feedback(s) you should give but remember everyone wants to be a better trainer! Its your job to help them get there! Slide 31. Use the elements of effective feedback a. Include negative feedback b. No one likes giving negative feedback, but it helps us achieve our end state (control theory) c. Don’t sandwich feedback (Positive-Negative-Feedback) d. Deliver negative feedback as needed Slide 32. What Not to Do in varied feedback a. Too general of feedback b. “Good job on that assignment” c. “I think you should rethink that post” d. Person specific feedback e. You are a real good leader f. You are a quick learner g. Only positive feedback h. Great… Good… Super… i. I have nothing to comment on j. Feedback a week after someone has demonstrated a behavior k. Last weeks posting needs some additional thought Slide 33. Now you try it: Deliver two example statements right now – one positive and one negative (Pause to review postings for accurate delivery) Slide 34. Learn to love feedback a. Make positive feedback COUNT b. Make negative feedback USEFUL! c. Specific things to try: i. Create an relationship for feedback where your feedback partner during this comfortable feel comfortable giving, receiving, and seeking feedback ii. Be a feedback model: ask for it, and when your partner gives it to you acknowledge it and USE it d. Give your partner the courtesy of honest and timely feedback that will help them grow and develop Slide 35. In sum… a. Feedback: a great tool! Just be mindful about what you are saying and how you are saying it Slide 36. Want to know more? 140 APPENDIX C: THE FEEDBACK ENVIRONMENT SCALE Instructions: Using the scale below indicate your level of agreement items. These items are intended to measure your organization’s feedback environment. Do not answer how you think you are expected to answer. Answer in an honest fashion. 1 = Strongly Agree 2 = Disagree 3 = Neither Agree nor Disagree 4 = Agree 5 = Strongly Agree 1. My manager is generally familiar with my performance on the job. 2. My manager is fair when evaluating my job performance. 3. My manager gives me useful feedback about my job performance. 4. The performance feedback I receive from my manager is helpful. 5. My manager is supportive when giving me feedback about my job performance. 6. When my manager gives me feedback, he or she is considerate of my feelings. 7. When I do a good job at work, my manager praises my performance. 8. My manager generally lets me know when I do a good job at work. 9. My manager tells me when my performance does not meet organizational standards. 10. When I don’t meet deadlines my manager lets me know. 11. My manager is usually available when I want performance information. 12. I interact with my manager on a daily basis. 13. My manager is often annoyed when I directly ask for performance feedback. (R) 14. My manager encourages me to ask for feedback whenever I am uncertain about my job performance. 15. My coworkers are generally familiar with my performance on the job. 16. My coworkers are fair when evaluating my job performance. 17. My coworkers give me useful feedback about my job performance. 18. The performance feedback I receive from my coworkers are helpful. 19. My coworkers are supportive when giving me feedback about my job performance. 20. When my coworkers give me feedback, they are considerate of my feelings. 21. When I do a good job at work, coworkers are praise my performance. 22. My coworkers generally let me know when I do a good job at work. 23. My coworkers tell me when my performance does not meet organizational standards. 24. When I don’t meet deadlines my coworkers let me know. 25. My coworkers are usually available when I want performance information. 26. I interact with my coworkers on a daily basis. 27. My coworkers are often annoyed when I directly ask for performance feedback. 28. My coworkers encourage me to ask for feedback whenever I am uncertain about my job performance. 141 APPENDIX D: THE FEEDBACK ORIENTATION SCALE Instructions: Using the scale below indicate your level of agreement items. These items are intended to measure your orientation to feedback. Do not answer how you think you are expected to answer. Answer in an honest fashion. 1 = Strongly Agree 2 = Disagree 3 = Neither Agree nor Disagree 4 = Agree 5 = Strongly Agree 1. Feedback contributes to my success at work. 2. To develop my skills at work, I rely on feedback. 3. Feedback is critical for improving performance. 4. Feedback from supervisors can help me advance in a company. 5. I find that feedback is critical for reaching my goals. 6. It is my responsibility to apply feedback to improve my performance 7. I hold myself accountable to respond to feedback appropriately. 8. I don’t feel a sense of closure until I respond to feedback. 9. If my supervisor gives me feedback, it is my responsibility to respond to it. 10. I feel obligated to make changes based on feedback. 11. I try to be aware of what other people think of me. 12. Using feedback, I am more aware of what people think of me. 13. Feedback helps me manage the impression I make on others. 14. Feedback lets me know how I am perceived by others. 15. I rely on feedback to help me make a good impression. 16. I feel self-assured when dealing with feedback. 17. Compared to others, I am more competent at handling feedback. 18. I believe that I have the ability to deal with feedback effectively. 19. I feel confident when responding to both positive a negative feedback. 20. I know that I can handle the feedback that I receive. 142 APPENDIX E: SELF-EFFICACY SCALE Instructions: Using the scale below indicate your level of agreement items. These items are intended to measure your confidence to apply the training skills you learned in the training. Do not answer how you think you are expected to answer. Answer in an honest fashion. 1 = Strongly Agree 2 = Disagree 3 = Neither Agree nor Disagree 4 = Agree 5 = Strongly Agree 1. I feel confident in my ability to use the adult learning skills in my training classes. 2. I think I can eventually reach a high level of performance applying adult learning teaching. 3. I am sure I can teach with adult learning principles effectively in a relatively short period of time 4. I don’t feel that I am as capable to perform the adult learning technique as other people. (R) 5. On average, other people are probably much more capable of performing this method of teaching as I am. 6. I am a fast learner, in comparison to other people. 7. I am not sure I can ever reach a high level of performance on applying adult learning teaching, no matter how much practice and training I get. (R) 8. It would take me a long time to learn how to teach using adult learning theory effectively. (R) 9. I am not confident that I can train using adult learning methods successfully. (R) 10. I doubt that my performance will be very adequate on teaching with adult learning skills. (R) 143 APPENDIX F: MOTIVATION SCALE Instructions: Using the scale below indicate your level of agreement items. These items are intended to measure your motivation to try out your newly learned training skills on the job. Do not answer how you think you are expected to answer. Answer in an honest fashion. 1 = Strongly Agree 2 = Disagree 3 = Neither Agree nor Disagree 4 = Agree 5 = Strongly Agree 1. 2. 3. 4. 5. 6. I am motivated to use the skills emphasized in this training program. I will try to perform as well as I can applying these skills to future trainings. I want to improve my performance on applying adult learning in my teaching. I am going to put forth a lot of effort executing training centered on adult learning principles. I am going to ignore the principles I learned here and continue to train how I have been. (R) I did not expect to pay much attention to the material presented during this training program. (R) 7. I am very unmotivated to do well applying adult learning theory to my training. (R) 8. If I get frustrated or discouraged utilizing adult learning theory, I will utilize the strategies learned in the training. 9. I have no desire to increase my performance on training with adult learning theory. (R) 10. I really could care less about my performance on training with adult learning theory. (R) 144 APPENDIX G: KNOWLEDGE ASSESSMENT Instructions: The following test measures your understanding of the material, which you have covered over the past five modules. Your responses to this test will determine your satisfactory understanding of adult learning and training within the Blackboard environment. Read each question and select the appropriate response. There is only one correct answer. You will not be allowed to retake this assessment. _______________________________________________________________________ 1. What model is the best for teaching in online courses? a. ADDIE b. Systems Model c. GB30 d. Depends on the objectives of your course 2. One of your courses involves a very compliance focused document that learners must acknowledge they have read. What is the best way to train this type of compliance material? a. Traditional delivery with instructor lead presentation b. Peer to peer learning interaction c. A mixed blend of instructor lead with social learning d. Should not be within a training environment 3. What is the most a reader should have to read per lesson in an online course? a. A paragraph b. A printed page c. Two printed pages d. Three printed pages 4. You are teaching a class on a concept like PACE which has much organizational familiarity. How would you make sure that your learners are fully engaged? a. Pre-assess their knowledge on the PACE model and tailor information to those learners b. Begin the course with clear learning objectives and teach the masses c. Begin the course with prescribed learning objectives and build in flex-course policies d. Remove most course content and enable as much social learning as possible 5. In self-paced learning, what should you establish clearly with your learners? a. There is new technology to digest content b. Learners should estimate just as much time as they would for a classroom environment learning c. Instructors will be less available than in classroom learning environments d. Learners will only get out of the course as much as they put into the course 145 6. In Blackboard, what role best describes the role of the trainer? a. A subject matter expert who creates the content for the course b. An online version of instructor who presents material in the front of the classroom c. A facilitator of knowledge exchange between peer to peer, peer to instructor, and student to instructor d. There is no consistent need for a instructor once course material is made for Blackboard 7. You are teaching a class that has a discussion board in one your courses. You realize that there are minimal postings, but learners have already moved to the next module. What most likely is the reason for lack of participation in the discussion board? a. Learners are not equipped with the skills to use discussion boards b. The discussion board may have a technical error c. Course objectives do not tie directly with social learning activities d. The group of learners in the course do not fit the profile for an ideal Blackboard learning environment 8. What is a “good rule of thumb” (principle) for media variety in a online course? a. For every one page of text, have two videos b. For every one page of text, have three social learning activities c. For every two pages of text, have one interactive PowerPoint deck d. For every three pages of text, deliver one live classroom session 9. What differentiates Blackboard courses from current LMS courses? a. Blackboard allows us to better enroll learners into and off of the LMS b. Blackboard enables our social learning, technology enabled learning strategy c. Blackboard includes interactive content d. Blackboard integrates well with Lectora 10. One of your course lessons involves synthesizing best practices your learners have seen in their organization around team effectiveness. What social learning enabled tool would you use in your course module? a. A discussion board b. A blog c. A wiki d. A group exercise 11. When evaluating a course’s effectiveness, how can you ensure your outcomes are measurable? a. Creating measurable course objectives b. Using Metrics that Matter c. Assessing MOA capability d. 360 SDA Feedback 12. Which part of teaching a course is most commonly overlooked, but the most critical component to ensuring the success of your course? 146 a. b. c. d. Assessing the needs of your learners Evaluating the outcomes of your course Creating visually appealing material Deliver the course with high energy and enthusiasm 13. You are teaching a class that refers to the OGSM. How would you have your learners consume the OGSM? a. Using the Content Management System b. Upload the latest version to the course c. Place a link to the Intranet’s purpose.pg.com d. Have them obtain a copy through find.pg.com 14. You notice that there are a few students who are lagging behind in their interaction with other students online. How would you handle the situation? a. Ask them to drop the course and enroll in a live delivery course b. Place them in a group together to hold private conversations c. Reach out to them to understand their constraints d. Change your teaching strategy to be more inclusive of various learning styles 15. What is the biggest challenge in enabling social learning within online classroom environments? a. Generational differences in the workforce b. Motivational differences to participate c. Technological competency differences to understand how to interact d. Manager support to work on trainings 16. When deciding what content to include within a course, what is the best way to decide what should be and what should be left out? a. Whether it helps the learner achieve a course objective b. Whether or not the course subject matter expert/course builder determines it is important information c. Whether the information is dated or current d. Whether it fits a particular style of learning 17. For a on demand course, what is the recommended maximum length of time a student should be allowed to not long into a course? a. 1 day b. 3 days c. 5 days 7 days 18. When holding live distance training sessions, what is the maximum time the instructor speaks before allowing the learners to participate or contribute? a. 1 hour b. 20 minutes c. 90 minutes 147 d. 45 minutes 19. At what point in the training process creation should the actual content be created? a. Before the course objectives have been outlined b. After learning objectives have been established c. After an organization decides what documents should be included d. After the target audience has been selected 20. After learners begin a course, which of the following elements can change to fit the learner’s educational journey? a. Learning objectives b. Course content c. Delivery vehicle for content d. The instructor 148 APPENDIX H: ADULT LEARNING TASK Instructions: To complete the course and become certified as internal trainer using the Blackboard environment, you must now create a detailed course outline for one of your course modules. You must achieve satisfactory levels of the components outlined below in order to pass this course. Your course instructors will evaluate the course modules and give you feedback on your performance. You will have one week to complete this Adult Learning Task. _______________________________________________________________________ Some notes: Consider the following notes during your course creation. • If you are not planning a course currently, create a course for the new hire orientation standardization project • You will not be evaluated on the actual content itself for the course. Instead it will be the design of the course. For example, you will not be evaluated if you have a video of Bob McDonald versus another leader talking about your subject. Instead, how you present the video and have your students engage with the video will be subject to evaluation. • You should take this task seriously as it is your opportunity to gather direct feedback on how to make your course more dynamic. Elements of evaluation: You will be evaluated against the five elements you learned in the final lesson on designing effective learning in Blackboard environments which build off the information and knowledge you learned in Lessons 1-4. 1. Design of Course with Learner Centric Design a. Course modules specifically outline what the learner will be able to achieve/learn by end of course b. Course modules include learning objectives written with all three elements of quality learning objectives c. Course modules include robust, accurate content d. Instructor includes flex-course policies and includes lesson plans to revise based on previous course content/experiences 2. Enhancing Learner Experiences through Dynamic Content a. Learner has to read less than one page per module b. Learner has a variety of media outlets to engage in content c. Learner is held accountable for consuming content d. Learner has opportunity to contribute to content for module/lesson 3. Enabling Easy Course Navigation a. Learner has less than 4 clicks to consume content for course b. Learner does not have to download more than one course content item (document, presentation, video, etc.) c. Learner never leaves the Blackboard environment d. Navigation is conducive to all types of learners, regardless of user platform (SEWP 5, O&G, or MAC) 4. Supporting Collaborative Learning Environments 149 a. Learner engages in a peer to peer exercise in each module b. Peer to peer exercises support learning objectives c. Sufficiently motivate peers to both give and receive content d. Course utilizes at least two methods of peer to peer learning 5. Ensuring Proper Evaluation and Learning Progress a. Identified evaluation criteria for effective assessment b. Course has notifications and early warning systems built in to enable engaged learner and instructor awareness c. Learners see clear connection between evaluation and learning objectives using MTM metrics d. Strategy in place for organizational ROI Evaluation criteria: For each element, the course will be evaluated on a scale from 1-5 on how well the course implements that principle, using the criteria below: 1. Very Poor. The course illustrates very little evidence that the instructor embraced this principle when designing course. None of the sub criteria are met. 2. Poor. The course illustrates minimal evidence that instructor utilized the principle to design the course. Only one of the sub criteria were met. 3. Average. The course illustrates some evidence that the instructor utilized the principle but may have not executed in the desired way. Two of the sub criteria were met, but not executed as intended. 4. Good. The course illustrates sufficient evidence that the instructor utilized the principle in the design of the module. Three of the sub criteria were met. 5. Excellent. The course illustrates effective evidence that instructor utilized the principle in the design of the module. All sub criteria were met. Overall, the Adult Learning Task has 25 possible points (5 points x 5 criteria). A course must achieve an 18 out of 25 to achieve satisfactory level of completion. 150 APPENDIX I: FEEDBACK UTILITY Instructions: Using the scale below indicate your level of agreement with each statement about the peer feedback you received in this course. These items are intended to measure your impressions of the feedback you received from your peers during the training. Do not answer how you think you are expected to answer. Answer in an honest fashion. 1 = Strongly Agree 2 = Disagree 3 = Neither Agree nor Disagree 4 = Agree 5 = Strongly Agree 1. Feedback I received from peers during this training will contribute to my success at training. 2. To develop my skills during this training, I relied on feedback from my peers. 3. Feedback from my peers was critical for improving performance in the course. 4. Feedback from my peers helped me advance in adult learning teaching technique. 5. I found that feedback from peers was critical for reaching my goals in this course. 151 APPENDIX J: TRAINING UTILITY Instructions: Using the scale below indicate your level of agreement with each statement. These statements are intended to measure your satisfaction with the training. Do not answer how you think you are expected to answer. Answer in an honest fashion. 1 = Strongly Agree 2 = Disagree 3 = Neither Agree nor Disagree 4 = Agree 5 = Strongly Agree 1. The instructor’s energy and enthusiasm kept me actively engaged. 2. I felt that the instructor was knowledgeable about the subject. 3. The scope of the material was appropriate to fit my needs. 4. The participant materials will be useful on the job. 5. The examples presented helped me understand the content 6. I felt the objectives of this learning offering were met. 7. I learned new knowledge and skills from this training 8. I will be able to apply the knowledge and skills learned to my job. 9. This training will improve my job performance. 10. This training was a worthwhile investment in my career development. 152 APPENDIX K: DEBRIEF LETTER Dear Participant, Thank you very much for participating in the “Introduction to Instructional Design” course, part of a study in partnership with Michigan State University. Below you will find more information about the purpose of this study as well as a list of counseling and informational resources. The purpose of this study was to learn what can help improve the effectiveness of online social learning programs. Previous research has shown that the effectiveness of online social learning depends highly on the interactions between peers and that those interactions typically are of low quality unless there is some type of training to help learners understand what to do. Previous research, though, has been inconsistent on how to develop these skills. This was an attempt to see if one method of training participants on how to deliver effective peer to peer feedback was beneficial in online social learning. There were two conditions for the training course you attended. One received a training on how to give quality peer to peer feedback while the other did not. You were assigned to a condition based on the timing of your enrollment. So, if you volunteered to participate early (the first one hundred), you were in the peer training intervention condition. If you volunteered later in the study (the second one hundred), you were assigned to the control group. This allowed for us to have an equal amount of participants in each condition while not interrupting any normal business procedures and training reporting systems. We expected that those who received the peer feedback training had higher motivation and confidence to transfer the skills that were learned in the course as well as had higher training outcomes such as higher test scores. We were also interested in how your own feedback environments and orientations interacted with the effects. For example, if your organization gives you quality feedback normally, you may already have the skills you need to give good feedback. For further information on this topic, we suggest you read: • • • • Falchikov, N. (1995). Peer feedback marking: developing peer assessment. Innovations in Education and Training International, 32, 175-187. Kraiger, K. (2008). Transforming our models of learning and development: Web-based instruction as enabler of third-generation instruction. Industrial and Organizational Psychology, 1(4), 454-467. Sluijsmans, D., Brand-Gruwel, S., Van Merriënbor, J., & Bastiaens, T. J. (2003). The training of peer assessment skills to promote the development of reflection skills in teacher education. Studies in Educational Evaluation, 29, 23–42. Topping, K. (1998). Peer feedback between students in colleges and universities. Review of educational research, 68(3), 249-276. We would like to thank you again for your participation. Participants who are interested in learning more about the results of this study may send the researchers a request for a summary of the findings via email at massmana@msu.edu. They may also send any comments, questions or concerns regarding the study to Adam J. Massman 153 Thank you. 154 APPENDIX L: FIGURES FIGURE 1: A CONCEPTUAL MODEL 155 FIGURE 2: INTERACTION OF FEEDBACK ENVIRONMENT AND FEEDBACK TRAINING ON FEEDBACK QUALITY High Quality Peer Feedback High Feedback Environment Low Feedback Environment Poor Quality Peer Feedback No Peer Feedback Training (Control) Peer Feedback Training (Manipulation) 156 FIGURE 3: INTERACTION OF FEEDBACK QUALITY AND FEEDBACK ORIENTATION ON TRAINEE CHARACTERISTICS High Trainee Characteristics (Motivation & Self-Efficacy) High Feedback Orientation Low Feedback Orientation Low Trainee Characteristics (Motivation & Self-Efficacy) High Quality Peer Feedback Poor Quality Peer Feedback 157 FIGURE 4. PROCEDURAL DIAGRAM OF STUDY 158 FIGURE 5. PEER FEEDBACK QUALITY RATING OVER TIME 159 FIGURE 6. PEER FEEDBACK ELEMENTS OVER TIME 160 FIGURE 7. RESULTING CONCEPTUAL MODEL 161 APPENDIX M: TABLES TABLE 1. TRAINING LESSONS, FEEDBACK EXERCISES, AND RELATION TO COURES OBJECTIVES Lesson 3: Introduction to Adult Learning Theory Lesson 2: Introduction to Instructional Design Lesson 1: Course Overview Content Feedback Exercise Learners are asked to share the best learning experience they have had and defend their reasoning; Peers are asked to evaluate their partners learning experiences – did their rationale for what defined “good training” make sense? Learners propose a training program (either new or revision of an existing training) that they will be asked to create for their employees at some point in their future for their function and must select their model for training using the elements of needs assessments; Peers are asked to evaluate the model Learners are asked to describe a time when they attended a training and relate it to adult learning concepts – did the course use adult learning; peers were asked to evaluate their responses Lesson 4: Introduction to Teaching Online Learners are asked to think about their own course they are planning on teaching and evaluate which aspects are “quality learning” for online teaching; peers are asked to evaluate these aspects Lesson 5: Introduction to Teaching in Blackboard Learners discuss potential ideas of how they will deliver content and present their vision for putting their content in Blackboard; peers are asked to evaluate these aspects 162 Relation to Course Objectives Learners become familiar with how to conduct dynamic learning online through seeing an example of peerto-peer learning and understanding the participatory nature of social learning, which is the type of training they will be required to make at the end of the training Learners are able to use needs assessments appropriately and apply the models learned for their own training course(s) being made; Peers help validate the appropriate choice for a quality final product(s) Learners are able to apply their learning about adult learning to examples of training to help them infuse adult learning principles into training program; peers help learners distinguish the complimentary nature of quality instructional design and adult learning principles Learners are able to find parallel face to face exercises to online classroom; peers help refine the ideas to create an even more defined training program with quality online learning exercises Learners are able to match ideas presented in quality instructional design to offerings in delivery platform; peers help ensure they leverage the full capability of technologies available TABLE 2. EXPERT PEER FEEDBACK GRADING RUBRIC Peer Feedback Quality Definition Quality peer feedback evaluation is the degree to which a feedback statement provides a detailed, objective statement, which enables the learner to reach a desired end state of learning. The feedback should clarify what good performance should have been, facilitating reflection about their own performance while delivering high quality about their performance. The feedback should be motivating in nature but not focus on delivering positive feedback in the absence of reality. Quality Considerations The feedback statements should be evaluated (rated on a scale from 1 to five) using the following dimensions: 1. Sign. Did they use the sign of the feedback with justification? Was it only positive feedback? 2. Specificity. Did they give the learner specific information about their performance or did they give general feedback that is not actionable? 3. Helpfulness. Did they focus on the information or the activity to help them reach their end goal of learning the information, or did they focus on giving feedback about the person? 4. Timeliness. Did they give feedback on the proximate activity or was it about a topic formerly covered? Ratings Use the following guidelines as ways to rate the peer feedback statements (1) Poor Quality Peer Feedback: The feedback was absolutely useless to the feedback receiver. The feedback focused on the person, not the information at hand. It was overly positive and too vague for the learner to make any behavioral and knowledge course corrections. Examples: § Good job on your supporting example! § You are an amazing learner! § I found your discussion post to be interesting! (2) Weak Quality Peer Feedback: The feedback was relatively non helpful. The feedback may have been specific but it was only positive and had nothing helpful to drive further refinement of understanding, or the feedback was not specific enough to help the user understand where they went wrong. Examples: § Great job on your blog about how team effectiveness has increased through the use of PACE. § I think you may want to re-evaluate what you did with this proposed model of package development. (3) Medium Quality Peer Feedback: The feedback was adequately helpful. The use of sign was appropriate with some supporting evidence with short specific statement. The feedback may have included mixed signed feedback or focused some on person-specific feedback. 163 TABLE 2 (CONT’D). Examples: § Good job on team effectiveness example, but I would have thought that your project manager discussion was a bit weak. But great overall! § You are a really great trainer already! I could tell that through your example of supporting your director you really understand the concepts of adult learning. (4) Good Quality Peer Feedback: The feedback was helpful. The use of sign was appropriate with specific supporting evidence. The feedback included consistent signed feedback to drive a clear understanding of the direction the receiver should go, but may have been slightly dated to be less effective for the user. Examples: § The document you uploaded to the discussion board includes a paragraph on our company WWBCM. You didn’t represent the most current version of our WWBCM. § I was catching up on my email last night and realized I missed your posting… that was off! You definitely should review what the instructor posted in regards to the five elements of effective BAM roles. (5) Superior/High Quality Peer Feedback: The feedback mirrored the exact principles of delivering quality feedback. The feedback sign was given appropriately with strong supporting evidence. The feedback was consistent, high specific, and ultimately helped the learner know how to do better in future performance scenarios. Examples: § You already know how to deliver great visuals. Make sure you’ve executed the visuals to the level you typically do in your presentation. Look at slide number 9, notice the contrast between that slide and the presentation you gave yesterday. § In your example of the Lesson 2 Discussion board, you said you had a coaching moment with an employee and said, “…” I think given what we learned, you should have said, “…” because… 164 TABLE 3. OVERALL MEANS, STANDARD DEVIATIONS, AND GROUP MEANS Overall Control Intervention Variable M SD M SD M SD n 203 107 96 Fb. Environ. 3.65 .51 3.72 .54 3.58 .46 Quality Rating 2.29 .87 2.94 .67 1.58 .37 Fb. Element 1.86 .73 2.34 .54 1.31 .49 Fb. Frequency 4.34 1.72 4.99 1.49 3.61 1.72 Fb. Orientation 2.73 .52 2.67 .58 2.68 .45 Motivation 3.33 .38 3.39 .42 3.25 .33 Self-Efficacy 3.18 .49 3.36 .48 2.98 .42 Fb. Utility 3.88 .71 4.10 .43 3.62 .85 Training Utility 4.41 .42 4.48 .46 4.34 .34 Knowledge 79.43 12.78 82.52 9.82 75.99 14.74 Behavior* 22.80 2.02 22.84 2.03 22.74 2.03 *Behavior measure had n = 78 for control, n = 89 for manipulation, 165 TABLE 4. CORRELATIONS AND RELIABILITIES FOR VARIABLES Variable 1 2 3 4 5 6 7 1 Ethnicity -2 Tenure -.06 -3 Gender -.42 -.06 -4 Condition -.28 -.16 -.04 -5 Fb. Environ. .05 .08 -.04 .14 (.92) 6 Quality Rating .02 -.14 -.06 .78** .07 (.88) 7 Fb. Element .01 -.12 .01 .71** .08 .75** (.67) 8 Fb. Frequency -.19 .-.13 .03 .40** .03 .30** .29** 9 Fb. Orientation .04 -.05 .04 .09 .16* .10 .17* 10 Motivation -.12 -.07 .05 .19** .06 .24** .13 11 Self-Efficacy .09 -.03 -.11 .39** .08 .34** .19** 12 Fb. Utility .19 -.12 -.02 .35** .11 .29** .27** 13 Training Utility .13 .01 .07 .17* .05 .16* .10 14 Knowledge .09 -.15 .03 .25** .05 .25* .27* 15 Behavior -.05 .01 .03 .03 .01 .06 .01 Note. Condition was coded 0 for control and 1 for manipulation. N = 203 * p < .05, ** p <.01 166 TABLE 4 (CONT’D). Variable 8 9 10 11 12 13 14 15 1 Ethnicity 2 Tenure 3 Gender 4 Condition 5 Fb. Environ. 6 Quality Rating 7 Fb. Element 8 Fb. Frequency -9 Fb. Orientation .04 (.88) 10 Motivation .10 .18* (.79) 11 Self-Efficacy .23** .21** .29** (.78) 12 Fb. Utility .11 .20** .04 .28** (.81) 13 Training Utility -.02 -.06 .10 .26** .19* (.76) 14 Knowledge .17* .10 .02 .02 .06 .12 (.66) 15 Behavior .05 .04 .02 .03 .04 .08 .03 (.69) Note. Condition was coded 0 for control and 1 for manipulation. N = 203 * p < .05, ** p <.01 167 TABLE 5. CORRELATIONS FOR OUTCOMES AND FEEDBACK ELEMENTS Variable 1 2 1 Element 1 -2 Element 2 -.28** -3 Element 3 -.12 .44** 4 Element 4 -.12 .51** 5 Element 5 -.14 .36** 6 Fb. Utility -.31** .29** 7 Training Utility .04 .15* 8 Knowledge -.03 .26** 9 Behavior -.02 -.08 N = 203 * p < .05, ** p <.01 168 3 4 5 -.58** -.49** .69** -.32** .29** .26** .09 .06 .05 .18* .24** .17 .01 .04 .06 REFERENCES 169 REFERENCES Ackerman, D., & Gross, B. (2010). Instructor feedback. How much do students really want? Journal of Marketing Education, 32, 172-181. Adams, J. A. (1987). Historical review and appraisal of research on the learning, retention, and transfer of human motor skills. Psychological Bulletin, 101, 41–74. Aguinis, H., Mazurkiewicz, M. D., & Heggestad, E. D. (2009). Using web-based frame of reference training to decrease biases in personality based job analysis: An experimental field study. Personnel Psychology, 62, 405-438. Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage. Airasian, P. W. (1997). Classroom assessment. In Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. Ajzen, I., & Fishbein, M. (1977). Attitude behavior relations: A theoretical analysis and review of empirical research. Psychological Bulletin, 84, 888-918. Allen, M., & Naughton, J. (2011). Social Learning: A call to action for learning professionals. The American Society for Training & Development. Retrieved from: http://www.astd.org/LC/0811_allen.htm. Alavosius, M. P., & Sulzer-Azaroff, B. (1990). Acquisition and maintenance of health-care routines as a function of feedback density. Journal of Applied Behavior Analysis, 23, 151-162. Alvero, A. M., Bucklin, B. R., & Austin, J. (2001). An objective review of the effectiveness and essential characteristics of performance feedback in organizational settings (1985-1998). Journal of Organizational Behavior Management, 21(1), 3-29. Anastasi, A. (1998). Psychological Testing (6th ed.). New York: Macmillian. Anderson, J. R., Conrad, C. G., & Corbett, A. T. (1989). Skill acquisition and the LISP Tutor. Cognitive Science, 13, 467–505. Anderson, J. R., Corbett, A. T., Koedinger, K., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4, 167-207. Annett, J. (1969). Feedback and human behavior. In Goodman, J. S., & Wood, R. E. (2004). Feedback specificity, learning opportunities, and learning. Journal of Applied Psychology, 89(5), 809-821. Antonioni, D. (1996). Designing an effective 360-degree appraisal feedback process. Organizational Dynamics, 25(2), 24–38. 170 Arbaugh, J. B. (2008). Instructors as facilitators of learner-learning interaction in third generation learning environments. Industrial and Organizational Psychology, 1, 487-490. Archer, J. E., Kent, G. W., & Mote, F. A. (1956). Effect of long-term practice and time on target information feedback on a complex tracking task. Journal of Experimental Psychology, 51, 103–112. Argyris, C. (1991). Teaching smart people how to learn. Harvard Business Review, May/June, 99-109. Ashford, S. J., & Black, S. J. (1996). Proactivity during organizational entry: The role of desire for control. Journal of Applied Psychology, 81, 199–214. Ashford, S. J., & Cummings, L. L. (1983). Feedback as an individual resource: Personal strategies of creating information. Organizational Behavior and Human Performance, 32, 370-398. Ashford, S. J., & Tsui, A. S. 1991. Self-regulation for managerial effectiveness: The role of active feedback seeking. Academy of Management Journal, 34: 251–280. Atwater, L.E., Waldman, D., Atwater, D., & Carter P. (2000). An upward feedback field experiment. Supervisors’ cynicism, follow-up and commitment to subordinates. Personnel Psychology, 53, 275-297. Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computerbased instruction. Journal of Educational Computing Research, 13(2), 11-127. Baddeley. A. D., & Logie, R. H. (1999). Working memory: The multiple-component model. In A. Miyake and P. Shah (Eds.) Models of working memory: Mechanisms of active maintenance and executive control. United Kingdom: Cambridge University Press. Balzer, W. K., Doherty, M. E., & O’Connor, R., Jr. (1989). Effects of cognitive feed- back on performance. Psychological Bulletin, 106(3), 410–433. Balcazar, F., Hopkins, B. L., & Suarez, Y. (1985-86). A critical, objective review of performance feedback. Journal of Organizational Behavior Management, 7(3/4), 65-89. Baldwin, T. T. (1992). Effect of alternative modeling strategies on outcomes of interpersonal personal skills training. Journal of Applied Psychology, 77, 147-154. Baldwin, T.T., Magjuka, R.J., & Loher, B.T. (1991). The perils of participation: effects of trainee choice on motivation and learning. Personnel Psychology, 44, 51–66. Bandura, A. (1977). Self-efficacy mechanism in human agency. American Psychologist, 37, 122147. Bandura, A. (1982). Self-efficacy mechanism in human agency. American Psychologist, 37(2), 122-147. 171 Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall. Bandura, A. (1991). Self-regulation of motivation through anticipatory and self-regulatory mechanisms. In R. A. Dienstbier (Ed.), Perspectives on motivation: Nebraska Symposium on Motivation. Lincoln: University of Nebraska Press. Bandura, A. (1997). Self-efficacy: The exercise of control. New York: Freeman. Bandura, A., & Cervone, D. (1983). Self-evaluative and self-efficacy mechanisms governing the motivational effects of goal systems. Journal of Personality and Social Psychology, 45, 1017-1028. Bandura, A., & Cervone, D. (1986). Differential engagement of self-reactive influences in cognitive motivation. Organizational Behavior and Human Decision Processes, 38, 92113. Bangert-Drowns, R. L., Kulik, C. L. C., Kulik, J. A., & Morgan, M. T. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61(2), 213-238. Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83, 762–765. Barrick, M. R., Stewart, G. L., & Piotrowski, M. (2002). Personality and job performance: Test of mediating effects of motivation among sales representatives. Journal of Applied Psychology, 87(1), 1-9. Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182. Baumeister, R. F. (1998). The self. In London, M., & Smither, J. W. (2002). Feedback orientation, feedback culture, and the longitudinal performance management process. Human Resource Management Review, 12, 81-100. Baumeister, R. F. (1999). The nature and structure of the self: an overview. In London, M., & Smither, J. W. (2002). Feedback orientation, feedback culture, and the longitudinal performance management process. Human Resource Management Review, 12, 81-100. Bedwell, W. L., & Salas, E. (2008). If you build it, will they interact? The importance of the instructor. Industrial and Organizational Psychology, 1, 491-493. Bell, B. S., & Kozlowski, S. W. J. (2002). Adaptive guidance: Enhancing self-regulation, knowledge, and performance in technology-based training. Personnel Psychology, 5, 267-306. Benedict, M. E., & Levine, E. L. (1988). Delay and distortion: Tacit influence on performance appraisal effectiveness. Journal of Applied Psychology, 73, 507-514. 172 Bennett, N., & Kell, J. (1989). A good start? Four year olds in infant schools. In Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. Bernardin, H.J, Cooke, D.K., & Villanova, P.D. (2000). Conscientiousness and agreeableness as predictors of rating leniency. Journal of Applied Psychology, 85(2), 232-236. Birkeland, T. S. (1986). Variations of feedback related to teaching paragraph structure to technicians. In Xiao, Y. (2010). The effects of training in peer assessment on university students’ writing performance and peer assessment quality in an online environment (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3429548). Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In Goodman, J. S., & Wood, R. E. (2004). Feedback specificity, learning opportunities, and learning. Journal of Applied Psychology, 89(5), 809-821. Black, P. & William, D. (1998). Assessment and classroom learning, Assessment in Education, 5(1), 7–74. Blackboard (2011). Blackboard Learn: Designing exemplary courses. Washington DC: Blackboard Inc. Bloom, A. J., & Hautaluoma, J. E. (1987). Effects of message valence, communicator credibility, and source anonymity on reactions to peer feedback. The Journal of Social Psychology, 127, 329-338. Blume, B., Ford, J.K., Baldwin, T., & Huang, J. (2010). Transfer of training: A meta-analytic review. Journal of Management, 20, 1-41. Bolton, A. E. (2006). Immediate versus delayed feedback in simulation based training: Matching feedback delivery timing to the cognitive demands of the training exercise (Doctoral Dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3233644). Boud, D. (2000). Sustainable assessment: rethinking assessment for the learning society, Studies in Continuing Education, 22(2), 151–167. Brackbill, Y., Bravos, A., & Starr, R. H. (1962). Delay-improved retention of a difficult task. Journal of Comparative and Physiological Psychology, 55, 947-952. Brophy, J. (1981). Teacher praise: A functional analysis. Review of Educational Research, 51, 5– 32. Brosvic, G. M., Epistein, M. L., Dihoff, R., E., & Cook, M. J. (2006). Acquisition and retention of Esperanto: The case for error correction and immediate feedback. The Psychological Record, 56, 205-218. 173 Brown, K. G. (2001). Using computers to deliver training: Which employees learn and why? Personnel Psychology, 54, 271–296. Butler, R. (1988). Enhancing and undermining intrinsic motivation: the effects of task-involving and ego-involving evaluation on interest and involvement, British Journal of Educational Psychology, 58, 1–14. Butler, A. C., Karpicke, J. D., & Roediger, H. L., III (2007). The effect of type and timing of feedback on learning from multiple-choice tests. Journal of Experimental Psychology: Applied, 13, 273-281. Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theo- retical synthesis. Review of Educational Research, 65(3), 245–274. Bryant, S. E. (2005). The impact of peer mentoring on organizational knowledge creation and sharing, Group and Organization Management, 30, 319-338. Callister, R. R., Kramer, M. W., & Turban, D. B. (1999). Feedback seeking following career transitions. Academy of Management Journal, 42, 429-438. Cannon-Bowers, J.A., Salas, E., Tannenbaum, S.I. & Mathieu, J.E. (1995). Toward theoreticallybased principles of training effectiveness: A model and initial empirical investigation. Military Psychology, 7, 141-164. Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education, 31, 219-233. Carson, J. G., & Nelson, G. L. (1996). Chinese students’ perceptions of ESL peer response group interaction. Journal of Second Language Writing, 5, 1-9. Carver, C. S., & Scheier, M. F. (1981). Attention and self-regulation: A control theory approach to human behavior. In Vancouver, J. B., & Tischner, E. C. (2004). The effect of feedback sign on task performance depends on self-concept discrepancies. Journal of Applied Psychology, 89(6), 1092-1098. Carver, C. S., & Scheier, M. F. (1998). On the self-regulation of behavior. New York: Cambridge University Press. Cascio, W. F. (1998). Applied psychology in human resource management (5th ed.). In Goodman, J. S., & Wood, R. E. (2004). Feedback specificity, learning opportunities, and learning. Journal of Applied Psychology, 89(5), 809-821. Cederblom, D. (1982). The performance appraisal interview: a review, implications, and suggestions. Academy of Management Review, 7, 219–227. Chaundron, C. (1983). Evaluating writing: Effects of feedback on revision. In Topping, K. (1998). Peer assessment between students in colleges and universities. Review of educational research, 68(3), 249-276. 174 Cheng, W., & Warren, M. (1999). Peer and teacher assessment of the oral and written tasks of a group project. Assessment & Evaluation in Higher Education, 24, 301–314. Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182. Cho, K., Schunn, C. D., & Wilson, R. W. (2006). Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. Journal of Educational Psychology, 98(4), 891- 901. Chhokar, J. S., & Wallin, J. A. (1984). A field study of the effect of feedback frequency on performance. Journal of Applied Psychology, 69, 524-530. Christina, R. W., & Bjork, R. A. (1991). Optimizing long-term retention and transfer. In Goodman, J. S., & Wood, R. E. (2004). Feedback specificity, learning opportunities, and learning. Journal of Applied Psychology, 89(5), 809-821. Cohen, D. J. (1990). What motivates trainees. Training & Development, October, 91-93. Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum. Colquitt, J., LePine, J., and Noe, R.. (2000) Toward an Integrative theory of training motivation: a meta-analytic path analysis of 20 years of research, Journal of Applied Psychology, 85(5), 678-707. Conway, J.M., & Huffcutt, A.I. (1997) Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10 ,331–360. Corbett, A. T., Anderson, J. R., Paterson, E. G. (1990). Student modeling and tutoring flexibility in the Lisp Intelligent Tutoring System. In Bolton, A. E. (2006). Immediate versus delayed feedback in simulation based training: Matching feedback delivery timing to the cognitive demands of the training exercises (Doctoral Dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3233644). Corbett, A. T., Koedinger, K. R. & Anderson, J. R. (1997). Intelligent tutoring systems. In M.G. Helander, T.K. Landauer & P.V. Prabhu (Eds.), Handbook of Human-Computer Interaction (pp. 849-74). Amsterdam: Elsevier. Cramer, E. M., & Bock, R. D. (1966). Multivariate analysis. Review of Educational Research, 36, 604-617. Cromwell, S. & Kolb, J. (2004). An examination of work-environment support factors affecting transfer of supervisory skills training to the workplace. Human Resource Development Quarterly, 15(4), 449–71. 175 Cron, W. L., Slocum, J. W., VandeWalle, D., & Fu, Q. (2005). The role of goal orientation on negative emotions and goal setting when initial performance falls short of one’s performance goal. Human Performance, 18, 55–80. Decker, P. J., & Nathan, B. (1985). Behavior modeling training: Principles and applications. In Kraiger, K. (2008). Transforming our models of learning and development: Web-based instruction as enabler of third-generation instruction. Industrial and organizational psychology, 1, 454-467. Daniels, J. A. (1997). The influence of performance feedback and causal attributions upon ratings of counseling self-efficacy. Unpublished doctoral dissertation, University of Nebraska. DeNisi, A., & Kluger, A.N. (2000). Feedback Effectiveness: Can 360-Degree Appraisals be Improved?, Academy of Management Executive, 14, 129-139. Dick, W., Carey, L., & Carey, J. O., (2001). The Systematic Design of Instruction (5th ed.). New York: Addison-Wesley, Longman. DiMicco, J., Millen, D. R., Geyer, W., Dugan, C., Brownholtz, B., & Muller, M. (2008). Motivations for social networking at work. In Proceedings of the ACM 2008 Conference on Computer Supported Cooperative Work. New York: ACM. Dolezalek, H. (2004). Training magazine’s 23rd annual comprehensive analysis of employersponsored training in the United States, Training, 41(10), 20. Dweck, C. (1999). Self-theories: their role in motivation, personality and development. In Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback. Studies in Higher Education, 31(2), 199218. Earley, P. C., Northcraft, G. B., Lee, C., & Lituchy, T. R. (1990). Impact of process and outcome feedback on the relation of goal setting to task performance. Academy of Management Journal, 33(1), 87–105. Eddy, W. B., Glad, D. D., Wilkins, D. M., Painter, M., Zavala, A., & Sedlacek, W. E. (1967). Organizational effects on training: A study conducted on a public administration program. Training and Development Journal, 21, 15-23. Edmondson, A. (2002). The local and variegated nature of learning in organizations: A group level perspective. Organization Science, 13(2), 128-146. Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406. Erikson, G. (1987). Peer evaluation as a teaching-learning strategy in baccalaureate education for community health nursing. Journal of Nursing Education, 26(5), 204-206. 176 Facteau, J., Dobbins, G., Russell, J., Ladd, R., & Kudisch, J. (1995). The influence of general perceptions of the training environment on pre-training motivation and percent training transfers. Journal of Management, 21(1), 1-25. Falchikov, N. (1986). Product comparisons and process benefits of collaborative peer group and self assessments. Assessment and Evaluation in Higher Education, 11, 146-166. Falchikov, N. (1995). Peer feedback marking: developing peer assessment. Innovations in Education and Training International, 32, 175-187. Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A metaanalysis comparing peer and teacher marks. Review of Educational Research, 70, 287– 322. Fallows, S. & Chandramohan, B. (2001). Multiple approaches to assessment: reflections on use of tutor, peer and self assessment, Teaching in Higher Education, 6(2), 229-246. Fedor, D. B. (1991). Recipient responses on feedback: Seeking and giving. In Ryan, A. M., Brutus, S., Greguras, G. J., & Hakel, M. D. (2000). Receptivity to assessment based feedback for management development. Journal of Management Development, 19(4), 252-276. Fedor, D. B. & Bettenhausen, K. C. (1989). The impact of purpose, participant preconceptions and self-serving bias on the acceptance of peer evaluations. Group and Organization Studies, 14, 182-197. Field, A. (2000). Discovering statistics using SPSS for Windows: London: Sage Publications. Fisher, C. D. (1979). Transmission of positive and negative feedback to subordinates: A laboratory investigation. Journal of Applied Psychology, 64, 533-540. Fisher, S. L., Wasserman, M. E., & Orvis, K. A. (2010). Trainee reactions to learner control: An important link in the e-learning equation. International Journal of Training and Development, 14(3), 198-208. Flynn, J. L. (1992). Cooperative learning and Gagne’s events of instruction: A syncretic view. Educational Technology, October, 53-60. Ford, J. K., Quiñones, M. A., Sego, D. J., & Sorra, J. S. (1992). Factors affecting the opportunity to perform trained tasks on the job. Personnel Psychology, 45, 511-527. Ford, J. K., Smith, E. M., Weissbein, D., Gully, S., & Salas, E. (1998). Relationships of goal orientation, metagcognitive activity, and practice strategies with learning outcomes and transfer. Journal of Applied Psychology, 83, 218-232. Frayne, C. A., & Latham, G. P. (1987). Application of social learning theory to employee selfmanagement of attendance. Journal of Applied Psychology, 72(3), 387-392. 177 Gagne, R. M. (1962). Military training and principles of learning. American Psychologist, 18, 83-91. Gagne, R. M. (1965). The Conditions of Learning. New York: CBS College Publishing. Gagne, K. (2002). One day at a time: Using performance management to translate strategy into results. Workspan, 45(2). Gagne, R. & Driscoll, M. (1988). Essentials of learning for instruction. Englewood Cliffs, NJ: Prentice-Hall. Gagne, R., Briggs, L. & Wager, W. (1992). Principles of instructional design. Fort Worth, TX: Harcourt. Garrison, D. R. (1993). A cognitive constructivist view of distance education: An analysis of teaching–learning assumptions. Distance Education, 14, 199-211. Garson, G. D. (2011). Factor analysis. Retrieved from North Carolina University Web Site: http://faculty.chass.ncsu.edu/garson/. Ghorpade, J. (2000). Managing five paradoxes of 360-Degree Feedback. Academy of Management Executive, 14(1), 140, 11. Ghorpade, J. & Lackritz, J. R. (2001). Peer evaluation in the classroom: A check for sex and race/ethnicity effects. Journal of Education for Business, May/June, 274-281. Gibson (2000). Feedback delays: How can decision makers learn not to buy a new car every time the garage is empty? In Bolton, A. E. (2006). Immediate versus delayed feedback in simulation based training: Matching feedback delivery timing to the cognitive demands of the training exercises (Doctoral Dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3233644). Gist, M. E. (1989). The influence of training method on self-efficacy and idea generation among managers. Personnel Psychology, 42(4), 787-805. Gist, M. E., & Mitchell, T. R. (1992). Self-efficacy: A theoretical analysis of its determinants and malleability. Academy of Management Review, 17(2), 183-212. Gist, M. E., Schwoere, C., & Rosen, B. (1989). Effects of alternative training methods of selfefficacy and performance in computer software training. Journal of Applied Psychology, 74, 884-891. Gist, M. E., Stevens, C., & Bavetta, A. G. (1991). Effects of self-efficacy and post-training intervention on the acquisition and maintenance of complex interpersonal skills. Personnel Psychology, 44(4), 837-861. Goldstein, I. L. (1974). Training: Program development and evaluation. Monterrey, California: Brooks/Cole Publishing. 178 Goldstein, I. L. (1986). Training in organizations: Needs assessment, development, and evaluation. Pacific Grove, CA: Brooks/Cole. Goldstein, I. L., & Ford, J. K. (2002). Training in organizations. Belmont, CA: Wadsworth. Gomez-Mejia, L. R., Balkin, D. B., & Cardy, R. L. (2001). Managing human resources. In Goodman, J. S., & Wood, R. E. (2004). Feedback specificity, learning opportunities, and learning. Journal of Applied Psychology, 89(5), 809-821. Gomersall, E.R., & Myers, M.S. (1966). Breakthrough in on-the-job training. Harvard Business Review, 44, 62-72. Goodman, J. S. (1998). The interactive effects of task and external feedback on practice performance and learning. Organizational Behavior and Human Decision Processes, 76, 232–252. Goodman, J. S., & Wood, R. E. (2004). Feedback specificity, learning opportunities, and learning. Journal of Applied Psychology, 89(5), 809-821. Goodman, J. S., Wood, R. E., & Chen, Z. (2011). Feedback specificity, information processing, and transfer of training. Organizational Behavior and Human Decision Processes, 115, 253–267. Goodman, J. S., Wood, R. E., & Hendrick, M. (2004). Feedback specificity, exploration, and learning. Journal of Applied Psychology, 89, 248-262. Green, S., Salkind, N., & Akey, T. (2000). Using SPSS for Windows. USA: Prentice Hall. Gregory, J. B. (2010). Employee coaching: The importance of supervisor/subordinate relationship and related constructs. Unpublished doctoral dissertation: University of Akron, Akron, OH. Greguras, G. J., & Robie, C. (1998). A new look at within-source inter-rater reliability of 360degree feedback ratings. Journal of Applied Psychology, 83(6), 960-968. Grice, J. W., & Iwasaki, M. (2007). A truly multivariate approach to MANOVA. Applied Multivariate Research, 12, 199-226. Guilford, W. H. (2001). Teaching peer review and the process of scientific writing. Innovations and Ideas, 25(3), 167-175. Guthrie, E. R. (1935). The Psychology of Learning. New York: Harper. Haaga, D. A. F. (1993). Peer review of term papers in graduate psychology courses. Teaching of Psychology, 20(1), 28–32. Hanrahan, S. J., & Isaacs, G. (2001). Assessing self-and peer-assessment: The student’s views. Higher Education Research and Development, 20(1), 53-70. 179 Harlen, W. & Crick, R. D. (2003). Testing and motivation for learning, Assessment in Education, 10(2), 169–207. Harris, M., & Schaubroeck, J. (1988). A meta-analysis of self-supervisor, self peer and peer supervisor ratings. Personnel Psychology, 41, 43-62. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. Hattrup, K. E. (1992). Affirmative action in organizational hiring: Self-regulatory processes underlying beneficiary reactions. Unpublished doctoral dissertation, Michigan State University, East Lansing. Hawkins, J. (1993). Technology and the organization of schooling. Communications of the ACM, 36(5), 30-34. Hazucha, J.F., Hezlett, S.A., & Schneider, R.J. (1993). The impact of multi-source feedback on management skills development. Human Resource Management, 32, 325-351. Heimbeck, D., Frese, M., Sonnentag, S., & Keith, N. (2003). Integrating errors into the training process: The function of error management instructions and the role of goal orientation. Personnel Psychology, 56, 333-361. Henning, J. M., Weidner T. G., & Marty M. C. (2008). Peer assisted learning in clinical education: literature review. Athletic Training Education Journal, 3, 84-90. Higgins, E. T. (1997). Beyond pleasure and pain. American Psychologist, 52, 1280–1300. Higgins, E.T. (1998). Promotion and prevention: Regulatory focus as a motivational principle. In Van-Dijk, D., & Kluger, A. N. (2004). Feedback sign effect on motivation: Is it moderated by regulatory focus? Applied Psychology: An international review, 53(1), 113135. Higgins, R., Hartley, P., & Skelton, A. (2002). The conscientious consumer: Reconsidering the role of assessment feedback in student learning. Studies in Higher Education, 27, 53-64. Hilling, K. & Tattersall, A. (1997). Boosting trainee’s expectation of success through knowledge of performance norms. Behavior & Information Technology, 16(6), 363-364. Hounsell, D. (2003). Student feedback, learning, and development. In M. Slowey & D. Watson (Eds.), Higher education and the lifecourse (pp. 67-78). Berkshire, UK: SRHE & Open University Press. Howell, D. C. (2007). Statistical methods for psychology. Belmont, CA: Thompson Wadsworth. Hyland, P. (2000). Learning from feedback in assessment. In In P. Hyland & A. Booth (Eds.), The practice of university history teaching (pp. 233-247). Manchester: Manchester University Press. 180 Hylton, M. E. (2006). Online versus classroom based instruction: A comparative study of outcomes ina diversity course. The Journal of Baccalaureate Social Work, 11(2), 102114. Idson, L. C., & Higgins, E. T. (2000). How current feedback and chronic effectiveness influence motivation: Everything to gain versus everything to lose. European Journal of Social Psychology, 30, 583-592. Ilgen, D. R. (1971). Satisfaction with performance as a function of the initial level of expected performance and deviation from expectations. Organizational Behavior and Human Performance, 6, 345-361. Ilgen, D.R, Fisher, C.D., & Taylor, M.S. (1979) Consequences of individual feedback on behavior in organization. Journal of Applied Psychology, 64, 349-371. Ilgen, D. R., & Knowlton, W. A. (1980). Performance attributional effects on feedback from supervisors. Organizational Behavior and Human Performance, 25, 441-456. Ivey, A. E. (1971). Microcounseling: Innovations in interviewing training. Oxford, England: Charles C. Thomas, Inc. Jaasma, M. A., & Koper, R. J. (1999). The relationship of student- faculty out-of-class communication to instructor immediacy and trust and to student motivation. Communication Education, 48, 41-47. Jacobs, G., & Zhang, S. (1989). Peer feedback in second language writing instruction: Boon or Bane? In Topping, K. (1998) Peer assessment between students in colleges and universities. Review of educational research, 68(3), 249-276. James, L. R., & Brett, J. M. (1984). Mediators, moderators, and tests for mediation. Journal of Applied Psychology, 69, 307–321. Johnson, D. W., Johnson, R., T., & Smith, K. A. (1998). Cooperative learning returns to college. Change, 30(4), 26-35. Jonassen, D. H. (1997). Thinking technology: toward a constructivist design model. Educational Technology, 34(3), 34-37. Joung, W., Hesketh, B., Neal, A. (2006). Using “War Stories” to train for adaptive performance: Is it better to learn from errors or success. Applied Psychology: An International Review, 55, 282-302. Kang, K., Oah, S., & Dickinson, A. M. (2003). The relative effects of different frequencies of feedback on work performance: A simulation. Journal of Organizational Behavior, 23(4), 21-53. Kaplan, R. M., & Saccuzzo, D. P. (2001). Psychological testing: Principles, applications, and issues. Belmont, CA: Wadsworth. 181 Kerr, P. M., Park, K. H., & Domazlicky, B. R. (1995). Peer grading of essays in a principles of microeconomics course. Journal of Education for Business, July/August, 356-361. Khan, B. H. (1997). Web-based instruction. In Kraiger, K. (2008). Transforming our models of learning and development: Web-based instruction as enabler of third-generation instruction. Industrial and organizational psychology, 1, 454-467. Kim, J. O., & Mueller, C. W. (1978). Introduction to factor analysis: What it is and how to do it. Newbury Park: Sage. Kirkpatrick, D. L. (1960). Techniques for evaluating training programs. Journal of American Society of Training and Development, 14(1). Kluger, A.N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284. Kluger, A.N. & DeNisi, A. (1998). Feedback interventions: Toward the understanding of a double-edge sword. Current Directions in Psychological Science. 7, 67-72. Kluger, A. N., & DeNisi, A. (2005). The interactive effect of feedback sign and task type on motivation and performance (Technical Report 1158). Arlington, VA: U.S. Research Institute for the Behavioral and Social Sciences. Knowles, M. (1980). The modern practice of adult education: From pedagogy to andragogy. Englewood Cliffs: Prentice Hall/Cambridge. Knowles, M., Holton, E., & Swanson, R. A. (2005). The Adult Learner, Sixth Edition. New York: Butterworth-Heinemann. Komaki, J., Heinzmann, A. T., & Lawson, L. (1980). Effect of training and feedback: Component analysis of a behavioral safety program. Journal of Applied Psychology, 65(3), 261-270. Kopelman, R. (1986). Objective feedback. In Kluger, A. N., & DeNisi, A. (1996). The effects of feedback intervention on performance: A historical review, a meta-analysis, and preliminary feedback intervention theory. Psychological Bulletin, 2, 254-284. Kraiger, K. (2008). Transforming our models of learning and development: Web-based instruction as enabler of third-generation instruction. Industrial and Organizational Psychology, 1(4), 454-467. Kraiger, K., & Ford, J. K. (2006). The expanding role of workforce training: Themes and trends influencing training research and practice. In Koppes, L. (2007). Historical perspectives in industrial and organizational psychology. Lawrence Eirbaum Associates: Mahwah, NJ. Kraiger, K., & Jerden, E., (2007). A new look at learner control: Meta-analytic results and direction for future research. In Kraiger, K. (2008). Transforming our models of learning 182 and development: Web-based instruction as enabler of third-generation instruction. Industrial and organizational psychology, 1, 454-467. Kreitner, R., & Kinicki, A. (2001). Organizational behavior (5th ed.). In Goodman, J. S., & Wood, R. E. (2004). Feedback specificity, learning opportunities, and learning. Journal of Applied Psychology, 89(5), 809-821. Krueger, N., & Dickson, P. R. (1994). How believing in ourselves increases risk taking: Perceived self-efficacy and opportunity. Decision Sciences, 25(3), 385-401. Kulhavy, R. W. & Anderson, R. C. (1972). Delay-retention effect with multiple-choice tests. Journal of Educational Psychology, 63, 505-512. Kulik, J. A., & Kulik, C. L. C. (1988). Timing of feedback and verbal learning. Review of Educational Research, 58(1), 79-97. Locke, E. A. & Latham, G. P. (1991). Self-Regulation through Goal Setting. Organizational Behavior And Human Decision Processes, 50, 212 – 247. Ladyshewsky, R., & Gotjamos, E. (1997). Communication skill development in health and professional education: The use of standardized patients in combination with a peer assessment strategy. Journal of Allied Health, 26(4), 177-186. Latham, G.P., & Locke, E.A. (1991). Self-regulation through goal setting. Organizational Behavior and Human Decision Processes, 50, 212-247. Lave, J., & Wenger, E. (1991). Situated learning: legitimated peripheral participation. In Kraiger, K. (2008). Transforming our models of learning and development: Web-based instruction as enabler of third-generation instruction. Industrial and organizational psychology, 1, 454-467. Larson, J. R. (1986). Supervisors’ performance feedback to subordinates: The impact of subordinate performance valence and outcome dependence. Organizational Behavior and Human Decision Processes, 37, 391-408. Larson, L. M. (1998). The social cognitive model of counselor training. The Counseling Psychologist, 26, 219-273. Leidner, D. E., & Jarvenpaa, S. L. (1995). The use of information technology to enhance management school education: A theoretical view. MIS Quatertly, 19(3), 265-291. Leivo, A. K. (2001). A field study of the effects of gradually terminated public feedback on housekeeping performance. Journal of Applied Social Psychology, 31, 1184-1203. Lemke, J. L. (1997). Cognition, context, and learning: A social semiotic perspective. In D. Kirshner & J. A. Whitson (Eds.), Situated cognition: Social, semiotic and psychological perspectives (pp. 37-55). Mahwah NJ: Erlbaum. 183 Levy, P. E., Albright, M. D., Cawley, B. D., & Williams, J. R. (1995). Situational and individual determinants of feedback seeking: A closer look at the process. Organizational Behavior and Human Decision Processes, 62, 23-37. Li, L., & Steckelberg, A., (2004). Using peer feedback to enhance student meaningful learning. In Xiao, Y. (2010). The effects of training in peer assessment on university students’ writing performance and peer assessment quality in an online environment (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3429548). Lim, D. & Morris, M. (2006). Influence of trainee characteristics, instructional satisfaction, an organizational climate on perceived learning and training transfer. Human Resource Development Quarterly, 17(1), 85–115. Lin, S. S. J., Liu, E. Z. F., & Yuan, S. M. (2001a). Web-based peer assessment: feedback for students with various thinking-styles. Journal of Computer Assisted Learning, 17, 420– 432. Lin, S. S. J., Liu, E. Z., & Yuan, S. M. (2001b). Web peer review: the learner as both adapter and reviewer. IEEE Transactions on Education, 44, 246–251. Linderbaum, B. A., & Levy, P. E. (2010). The development and validation of the feedback orientation scale. Journal of Management, 36(6), 1372-1405. Lipnevich, A. A., & Smith, J. K. (2009). Effects of differential feedback on students’ examination performance. Journal of Experimental Psychology, 15(4), 319-333. Liu, E. Z. F., Lin, S. S. J., Chiu, C. H. & Yuan, S. M. (2001). Web-based peer review: an effective web-learning strategy with the learner as both adapter and reviewer. IEEE Transactions on Education, 44, 3, 246–251. Locke, E. A. and Latham, G. P. (1984). Goal-setting: A motivational technique that works. Englewood Cliffs, N.J.: Prentice-Hall. Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57, 705–717. London, M., & Smither, J. W. (1995). Can multi-source feedback change self-awareness and behavior? Theory-based applications and directions for research. Personnel Psychology, 48, 803–840. London, M., & Smither, J.W. (1999). Empowered self-development and continuous learning. Human Resource Management Journal, 38, 3-16. London, M., & Smither, J. W. (2002). Feedback orientation, feedback culture, and the longitudinal performance management process. Human Resource Management Review, 12, 81-100. 184 London, M., Smither, J.W., & Adsit, D.J. (1997). Accountability: The Achilles' heel of multisource feedback. Group & Organization Management, 22, 162-185. Lord, R. G., & Hanges, P. J. (1987). A control systems model of organizational motivation: Theoretical development and applied implications. Behavioral Science, 32, 161-178. Lu, R., & Boi, L. (2007). A comparison of anonymous versus identifiable e-peer review on college student writing performance and the extent of critical feedback. Journal of Interactive Online Learning, 6(2), 100-115. Lysakowski, R. S., & Walberg, H. J. (1982). Instructional effects of cues, participation, and corrective feedback: A quantitative synthesis. American Educational Research Journal, 19, 559–578. MacLeod, L. (1999). Computer-aided peer review of writing. Business Communication Quarterly, 62(3), 87-95. Mansell, W. (2005). Control theory and psychopathology: An integrateive approach. Psychology and Psychotherapy: Theory Research and Practice, 78, 141-178. Markowitz, N. & Renner, E. (1966). The delay-retention effect and feedback. Journal of Experimental Psychology, 72, 452-455. Martocchio, J. J. (1992). Microcomputer usages as an opportunity: the influence of context in employee training. Personnel Psychology, 45, 529-552. Martocchio, J. J., & Dulebohn, J. (1994). Performance feedback effects in training: The role of perceived controllability. Personnel Psychology, 47(2), 357-373. Martocchio, J. J., & Webster, J. (1992). Effects of feedback and cognitive playfulness on performance in microcomputer software training. Personnel Psychology, 19(1), 127-146. Marty, M., Henning, J., & Willse, J. (2011). Students provide accurate, but not always detailed feedback to a peer performing an orthopedic assessment skill. The International Journal of Allied Health Sciences and Practice, 9(1), 2-7. Mathieu, J. E. & Martineau, J. (1997). Individual and situational influences on training motivation. In J.K. Ford (Ed.), Improving Training Effectiveness in Work Organizations. pp. 193-222. Mahwah, NJ: LEA. Mathieu, J. E., Martineau, J. W. & Tannenbaum, S. I. (1993). Individual and situational influences on the development of self efficacy: Implications for training effectiveness. Personnel Psychology, 46, 125-147. Matsui, T., Okada, A., Kakuyama, T., (1982). Influence of achievement need on goal setting, pay and feedback effectiveness. Journal of Applied Psychology, 67, 645-648. Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press. 185 Medvedeff, M., Gregory, J. B., & Levy, P. E. (2008). How attributes of the feedback message affect subsequent feedback seeking: The interactive effects of feedback sign and type. Psychologica Belgica, 48(2-3), 109-125. Mehrens, W.A., Popham, W.J., & Ryan, J.M. (1998). How to prepare students for performance assessments. Educational Measurements: Issues and Practice, 17(1), 18-22. Meyers, L.S., Gamst, G., & Guarino, A. (2006). Applied multivariate research: Design and interpretation. Thousand Oaks, CA: Sage Publishers. Miller, C. D., Morrill, W. H., & Uhleman, M. R. (1970). Micro-counseling: An experimental study of pre-practicum training in communicating test results. Counselor Education and Supervision, 9(3), 171-177. Moore, M. G. (1989). Three types of interaction. American Journal of Distance Education, 3(2), 1-6. Moreno, R. (2004). Decreasing cognitive load for novice students: Effects of explanatory versus corrective feedback in discovery-based multimedia. Instructional Science, 32, 99–113. Morrison, G. R., Ross, S. M., & Kemp, J. E. (2004). Designing effective instruction (4th ed.). Hoboken, NJ: John Wiley & Sons, Inc. Mowl, G., & Pain, R. (1995). Using self and peer assessment to improve students’ essay writing – A case study from geography. Innovations in Education and Training International, 32, 324-335. Nease, A. A., Mudgett, B. O., & Quiñones, M. A. (1999). Relationships among feedback sign, self-efficacy, and acceptance of performance feedback. Journal of Applied Psychology, 84(5), 806-814. Ng, K., Koh, C., Ang, S., Kennedy, J. C., & Chan, K. (2011) Rating leniency and halo in multisource feedback ratings: Testing cultural assumptions of power distance and individual-collectivism. Journal of Applied Psychology, 96(5), 1033-1044. Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback. Studies in Higher Education, 31(2), 199218. Nijman, D., Nijhof, W., Wognum, A. & Veldkamp, B. (2006). Exploring differential effects of supervisor support on transfer of training. Journal of European Industrial Training, 30(7), 529–49. Nilson, L. B. (2003). Improving student peer feedback. College Teaching, 51(1), 34-39. Noe, R. A. (1986). Trainee’s attributes and attitudes: neglected influences on training effectiveness. Academy of Management Review, 11, 736-749. 186 Noe, R. A., & Colquitt, J. A. (2002). Planning for training impact: Principles of training effectiveness. In K. Kraiger (Ed.), Creating, implementing, and managing effective training and development: State of the art lessons for practice. San Francisco, CA: Jossey-Bass. Noe, R. A., & Schmitt, N. (1986). The influence of trainee attitudes on training effectiveness: Test of a model. Personnel Psychology, 44, 51-65. Nolan, T. V., Jarema, K. A., & Austin, J. (1999). An objective review of the Journal of Organizational Behavior Management: 1987-1997. Journal of Organizational Behavior Management, 19, 83-114. Northcraft, G. B., & Ashford, S. J. (1990). The preservation of self in everyday life: The effects of performance expectations and feedback context on feedback inquiry. Organizational Behavior and Human Decision Processes, 47, 42-64. Nowack, K. (1992). Self-assessment and rater-assessment as a dimension of management development. Human Resources Development Quarterly, 3, 141-155. Nowack, K. (2005). Longitudinal evaluation of a 360-degree feedback program: Implications for best practices. Paper presented at the 20th Annual Conference of the Society for Industrial and Organizational Psychology, Los Angeles, March 2005. O'Loughlin, M. (1992). Rethinking science education: Beyond Piagetian constructivism toward a sociocultural model of teaching and learning. Journal of Research in Science Teaching, 29, 791-820. Orsmond, P., Merry, S., & Reiling, K. (1996). The importance of marking criteria in the use of peer assessment, Assessment and Evaluation in Higher Education, 21, 239–50. Orsmond, P., Merry, S., & Reiling, K. (2000). The use of student derived marking criteria in peer and self-assessment. Assessment and Evaluation in Higher Education, 25, 309-323. Osburn, H. K., & Mumford, M. D. (2006). Creativity and planning: Training interventions to develop creative problem-solving skills. Creativity Research Journal, 18(2), 173-190. Palincsar, A. S. (2003). Ann L Brown: Advancing a theoretical model of learning and instruction. In B. J. Zimmerman and D. H. Schunk (Eds.), Educational psychology: A century of contributions, Mahwah, NJ: Erlbaum. Papinczak, T., Young, L., Groves, M., & Haynes. M. (2007). An analysis of peer, self, and tutor assessment in problem-based learning tutorials. Medical Teaching, 29(5), 122-132. Paulus, T. M. (1999). The effect of peer and teacher feedback on student writing. Journal of Second Language Writing, 83(2), 265-289. Piaget, J. (1969). Science of education and the psychology of the child. New York: Norton. 187 Plutsky, S., & Wilson, B. A. (2004). Comparison of three methods for teaching and evaluating writing: A quasi-experimental study. The Delta Pi Epsilon Journal, 46(1), 50-61. Podsakoff, P. M., & Farth, J. H. (1989). Effects of feedback sign and credibility on goal setting and task performance. Organizational Behavior and Human Decision Processes, 44, 6567. Quiñones, M. A. (1995). Pretraining context effects: Training assignment as feedback. Journal of Applied Psychology, 80(2), 226-238. Ramesh, P. (2011, Apr. 18). Its not YOU… It’s your company culture that’s not ready for social media. [Web log]. http://www.crttbuzzbin.com/2011/04/18/dont-blame-yourself-its-yourcompany-culture-thats-not-ready-for-social-media/ Ramsden, P. (2003). Learning to teach in higher education. In Ackerman, D., & Gross, B. (2010). Instructor feedback. How much do students really want? Journal of Marketing Education, 32, 172-181. Renn, R. W., & Fedor, D. B. (2001). Development and field test of a feedback seeking, selfefficacy and goal setting model of work performance. Journal of Management, 27, 563584. Redmond, M. R., Mumford, M. D., & Teach, R. J. (1993). Putting creativity to work: Leader influences on subordinate creativity. Organizational Behavior and Human Decision Processes, 55, 120–151. Reid, D. H., & Parsons, M. B. (1996). A comparison of staff acceptability of immediate versus delayed verbal feedback in staff training. Journal of Organizational Behavioral Management, 16(2), 35-47. Richardson, V. (1997). Constructivist teacher education: Building a world of new understandings London: Falmer. Rich, J. T. (2002). The solution for employee performance mismanagement. Workspan, 45(2), 16. Richer, D. L. (1992). The effects of two feedback systems on first year college students’ writing proficiencies. Dissertation Abstracts International, 53. Robinson, J. M. (1999). Computer assisted peer review. Computer Assisted Assessment, 95-102. Rogers, W. A., Rousseau, G. K., & Fisk, A. D. (1999). Applications of attention research. In Bolton, A. E. (2006). Immediate versus delayed feedback in simulation based training: Matching feedback delivery timing to the cognitive demands of the training exercises (Doctoral Dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3233644). 188 Rosen, C. C. (2006). Politics, stress, and exchange perceptions: A dual process model relating organizational politics to employee outcomes. Unpublished doctoral dissertation, The University of Akron, Akron, OH. Rowold, J. (2007). Individual influences on knowledge acquisition in a call center training context in Germany. International Journal of Training and Development, 11(1) 21-34. Russ-Eft, D. (2002). A typology of training design and work environment factors affecting workplace learning and transfer. Human Resource Development Review, 1(1), 45–65. Rutherford, W. J., Corbin, C. B., & Chase, L. A. (1992). Factors influencing intrinsic motivation towards physical activity. Health Values, 16(5), 19-24. Ryan, A. M., Brutus, S., Greguras, G. J., & Hakel, M. D. (2000). Receptivity to assessment based feedback for management development. Journal of Management Development, 19(4), 252-276. Sadler, R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119–144. Saito, H., & Fujita, T. (2004). Characteristics and user acceptance of peer rating in EFL writing classrooms. Language Teaching Research, 8(1), 31-54. Saks, A. M., & Haccount, R. R. (2008). Is the “third generation model” new and is it the holy grail of adaptive learning? Industrial and Organizational Psychology, 1(4), 480-483. Salmoni, A. W., Schmidt, R. A., & Walter, C. B. (1984). Knowledge of results and motor learning. A review of critical reappraisal. Psychological Bulletin, 95, 355-386. Salvaggio, A. N., Schneider, B., Nishii, L. H., Mayer, D. M., Ramesh, A., & Lyon, J. S. (2007). Manager personality, manager service quality orientation, and climate: est of a model. Journal of Applied Psychology, 92(6), 1741-1750. Salvati, N., Gosselin, A., & Morin, D. (2003). Performance feedback overload: An exploratory field study of its antecedents and consequences. Montreal, Quebec: HEC Montreal. Schaffer, J. (1996). Peer response that works. Journal of Teaching Writing, 15(1), 81-89. Scheiter, K., & Gerjets, P. (2007). Learner control in hypermedia environments. Educational Psychology Review, 19, 285-307. Schelfhout W., Dochy F., & Janssens S. (2004). The use of self, peer and teacher assessment as a feedback system in a learning environment aimed at fostering skills of cooperation in an entrepreneurial context. Assessment and Evaluation in Higher Education, 29(2), 177-201. Schifter, D., & Simon, M. (1992). Assessing teachers development of constructivist view of mathematics learning. Teaching and Teacher Education, 8(2), 187-197. 189 Schmidt, R. A. (1991). Frequent augmented feedback can degrade learning: Evidence and interpretations. In Goodman, J. S., & Wood, R. E. (2004). Feedback specificity, learning opportunities, and learning. Journal of Applied Psychology, 89(5), 809-821. Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations of practice: Common principles in three paradigms suggest new concepts for training. Psychological Science, 3, 207-217. Schneider, D. J. (1973). Implicit personality theory: A review. Psychological Bulletin, 79(5), 294-309. Schooler, L. J. & Anderson, J. R. (1990). The disruptive potential of immediate feedback. In Bolton, A. E. (2006). Immediate versus delayed feedback in simulation based training: Matching feedback delivery timing to the cognitive demands of the training exercise (Doctoral Dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3233644). Schunk, D. H. (1989). Self-regulation and self-efficacy during self-regulated learning. Educational Psychologist, 25(1), 71-86. Sims, R. & Jones, D. (2003). Where practice informs theory: Reshaping instructional design for academic communities of practice in online teaching and learning. Information Technology, Education and Society, 4(1), 3-20. Sitzmann, T., Kraiger, K, Stewart, D., & Wisher, R. (2006). The comparative effectiveness of web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59, 623664. Slavin, R.E. (1990). Cooperative learning: Theory, research, and practice. New Jersey: Prentice Hall. Sluijsmans, D., Brand-Gruwel, S. & Van Merriënbor, J. (2002). Peer assessment training in teacher education: effects on performance and perceptions. Assessment and Evaluation in Higher Education, 27(5), 443-454. Sluijsmans, D., Brand-Gruwel, S., Van Merriënbor, J., & Bastiaens, T. J. (2003). The training of peer assessment skills to promote the development of reflection skills in teacher education. Studies in Educational Evaluation, 29, 23–42. Sluijsmans, D., Brand-Gruwel, S., Van Merriënbor, J., & Martens, R. L. (2004). Teaching teachers in peer-assessment skills: Effects on performance and perceptions. Innovations in Educational and Teaching International, 41(1), 59-78. Sluijsmans, D., Moerkerke, G., Van Merriënbor, J. & Dochy, F. (2001) Peer assessment in problem-based learning, Studies in Educational Evaluation, 27, 153-173. Smith-Jentsch, K.A., Zeisig, R.L., Acton, B., McPherson, J.A. (1998). Team dimensional training: A strategy for guided team self-correction. In J.A. Cannon-Bowers & E. Salas 190 (Eds.), Making decision under stress: Implications for individual and team training (pp 271-297). APA: Washington, DC. Springer, D. (1953). Ratings of candidates for promotion by co-workers and supervisors. Journal of Applied Psychology, 64, 124–131. Stajkovic, A. D., & Luthans, F. (1998). Self-efficacy and work-related performance: a metaanalysis. Psychological Bulletin, 124, 240–261. Steele, C. M. (1975). Name-calling and compliance. Journal of Personality and Social Psychology, 31:361-369. Steelman, L. A., Levy, P. E., & Snell, A. F. (2004). The feedback environment scale: Construct definition, measurement, and validation. Educational and Psychological Measurement, 64, 165-184. Steelman, L.A., & Rutkowski, K.A. (2004). Moderators of employee reactions to negative feedback. Journal of Managerial Psychology, 19, 6-18. Stefani, L. A. (1994). Peer, self, and tutor assessment – Relative reliabilities. Studies in Higher Education, 19, 69-75. Stevens, A. L., & Collins, A. (1980). Multiple conceptual models of a complex system. In Kraiger, K. (2008). Transforming our models of learning and development: Web-based instruction as enabler of third-generation instruction. Industrial and organizational psychology, 1, 454-467. Stone, D., Gueutal, H. G., & McIntosh, B. (1984). The effects of feedback sequence and expertise of the rater on perceived feedback accuracy. Personnel Psychology, 37, 487506. Stone, E., & Stone, D. (1984). The effects of multiple sources of performance feedback and feedback favorability on self-perceived task competence and perceived feedback accuracy. Journal of Management, 10(3), 371-378. Sweller, J., Van Merriënboer, J., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251–296. Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Allyn and Bacon. Tannenbaum, S. I. (1997). Enhancing continuous learning: Diagnostic findings from multiple companies. Human Resource Management, 36, 437-452. Tannenbaum, S. I., Mathieu, J. E., & Cannon, J. A. (1991). Meeting trainees’ expectations: The influence of training fulfillment on the development of commitment, self-efficacy, and motivation. Journal of Applied Psychology, 76(6), 759-769. 191 Tenenbaum, G., & Goldring, E. (1989). A meta-analysis of the effect of enhanced instruction: Cues, participation, reinforcement and feedback and correctives on motor skill learning. Journal of Research and Development in Education, 22, 53–64. Tesser, A., & Rosen, S. (1975). The reluctance to transmit bad news. In Steelman, L.A., & Rutkowski, K.A. (2004). Moderators of employee reactions to negative feedback. Journal of Managerial Psychology, 19, 6-18. Thompson, T. (1998). Metamemory accuracy: Effects of feedback and the stability of individual differences. American Journal of Psychology, 111(1), 33–42. Thorndike, E. (1927). The law of effect. American Journal of Psychology, 39, 212-222. Toleman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55(4), 189-208. Toll, B., O’Malley, S., & Katulak, N. (2007). Comparing gain-and loss-framed messages for smoking cessation with sustained release bupropion: A randomized control trail. Psychology of Addictive Behaviors, 21(4), 534-544. Tornow, W. & London, M. (1998). Maximizing the value of 360-degree feedback: a process for successful individual and organizational development. In The Impact of 360-Degree Feedback on Leadership Development (Doctoral Dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3249898). Topping, K. (1998). Peer assessment between students in colleges and universities. Review of educational research, 68(3), 249-276. Topping, K., & Ehly, S. (1998). Peer assisted learning. In Xiao, Y. (2010). The effects of training in peer assessment on university students’ writing performance and peer assessment quality in an online environment (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3429548). Topping, K.J., Smith, E.F., Swanson, I. & Elliot, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & Evaluation in Higher Education, 25(2), 146–169. Tracey, J. B., Tannenbaum, S. I., & Kavanagh, M. J. (1995). Applying trained skills on the job: The importance of the work environment. Journal of Applied Psychology, 80, 239-252. Tracey, J.B., Hinkin, T.R., Tannenbaum, S., & Mathieu, J.E. (2001). The influence of individual characteristics and the work environment on varying levels of training outcomes. Human Resource Development Quarterly, 12, 5–23. Trowbridge, M. A., & Cason, H. (1932). An experimental study of Thorndike’s theory of learning. Journal of General Psychology, 7, 245-260. 192 Tyler, R. W. (1949). Basic principles of curriculum and instruction. In Kraiger, K. (2008). Transforming our models of learning and development: Web-based instruction as enabler of third-generation instruction. Industrial and organizational psychology, 1, 454-467. Tsai, C. C., Lin, S. S. J., & Yuan, S. M. (2002). Developing science activities through a networked peer assessment system. Computers and Education, 38, 241–252. Tseng, S. C., & Tsai, C. C. (2007). On-line peer assessment and the role of the peer feedback: a study of high school computer course. Computers & Education, 49, 1161-1174. Vancouver, J. B., & Tischner, E. C. (2004). The effect of feedback sign on task performance depends on self-concept discrepancies. Journal of Applied Psychology, 89(6), 1092-1098. Van den Bossche, P., Segers, M., & Janssen, N. (2010). Transfer of training: the role of feedback in supportive social networks. International Journal Training and Development, 14(2), 81-94. Van-Dijk, D., & Kluger, A. N. (2004). Feedback sign effect on motivation: Is it moderated by regulatory focus? Applied Psychology: An international review, 53(1), 113-135. Van Hooft, E. A., van der Flier, H., & Minne, M. R. (2006). Construct validity of multi-source performance ratings: An examination of the relationship of self-, supervisor-and peerratings with cognitive and personality measures. International Journal of Selection and Assessment, 14(1), 67-81. Venables, A., & Summit, R. (2003). Enhancing scientific essay writing using peer assessment. Innovations in Education and Teaching International, 40(3), 281-290. Vickerman, P. (2008). Student perspectives on formative peer assessment: An attempt to deepen learning? Assessment and Evaluation in Higher Education, 1, 1-10. Villanova, P. D., Bernardin, H. J., Dahmus, S. A., & Sims, R. L. (1993). Rater leniency and performance appraisal discomfort. Educational and Psychological Measurement, 53(3), 789-799. Vrasidas, C., & McIsaac, M. S. (1999). Interaction in an online course. In Kraiger, K. (2008). Transforming our models of learning and development: Web-based instruction as enabler of third-generation instruction. Industrial and organizational psychology, 1, 454-467. Vygotsky, L. S. (1971). The psychology of art. Cambridge, MA: MIT Presss. Walberg, H. J. (1982). What makes schooling effective? Contemporary Education Review, 1, 1– 34. Weiner, I. B., Freedheim, D. K., Schinka, J. A., & Velicer, W. F. (2003). Handbook of psychology: Research methods in psychology. New York: Wiley. Westerman, J. W., & Rosse, J. G. (1997). Reducing the threat of rater nonparticipation in 360- 193 degree feedback systems. Group and Organization Management, 22(2), 288-309. Winstein, C. J., & Schmidt, R. A. (1990). Reduced frequency of knowledge of results enhances motor skills learning. Journal of Experimental Psychology, 16(4), 677-691. Whipple, W. R. (1987). Collaborative learning: Recognizing it when we see it. AAHE Bulletin, October 1987, 3-5. Whitaker, B., Dahling, J., & Levy, P. (2007). The development of the feedback environment and role clarity of job performance. Journal of Management, 33(4), 570-591. Whitehead, J.R., & Corbin, C.B. (1991). Youth fitness testing: The effect of percentile based evaluative feedback on intrinsic motivation. Research Quarterly for Exercise and Sport, 62, 225–231. Wilkinson, S. S. (1981). The relationship of teacher praise and student achievement: A metaanalysis of selected research. Dissertation Abstracts International, 41(9–A), 3998. Williams, L. (1993). Upon opening the black box and finding it empty: Social constructivism and the philosophy of technology. Science, Technology, and Human Values, 18, 362-378. Wohlers, A. J., Hall, M. J., & London, M. (1993). Subordinates rating managers: Organizational and demographic correlates of self- subordinate agreement. Journal of Occupational and Organizational Psychology, 66, 263–275. Wong, M. W. K., Wood, Robert, E., & Goodman, J. S. (2012). Metacognitive training, feedback specificity, metacognitive activity, and transfer of training. Manuscript prepared for publication. Xiao, Y. (2010). The effects of training in peer assessment on university students’ writing performance and peer assessment quality in an online environment (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3429548). Xiao, Y., & Lucking, R. (2008). The impact of two types of peer assessment on students’ performance and satisfaction within a Wiki environment. Internet and Higher Education, 11(3-4), 186-193. Yoder, R. J. (2009). Learning Cognitive Feedback Specificity during Training and the Effect on Learning Cognitive Tasks (Doctoral Dissertation). Retrieved from ProQuest Dissertations and Theses (UMI No. 3390442). Yorke, M (2003). Formative assessment in higher education: moves towards theory and the enhancement of pedagogic practice, Higher Education, 45(4), 477–501. Yu, J., & Murphy, K. R. (1993). Modesty bias in self-ratings or performance: A test of the cultural relativity hypothesis. Personnel Psychology, 46, 357-363. Yukl, G., & Lepsinger, R. (1995). How to get the most out of 360-degree feedback. Training, 32, 194 45–50. Zhao, Y. (1998). The effects of anonymity on computer-mediated peer review. International Journal of Educational Telecommunication, 4(4), 311-345. Zhu, W. (1994). Effects of training for peer revision in college freshman composition classes. Dissertation Abstracts International, 55, 209. Zhu, W. (1995). Effects of training for peer response on students’ comments and interaction. Written Communication, 12(4), 492-528. Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. 195