SCAFFOLDING SELF-REGULATED LEARNING ONLINE: A STUDY IN HIGH SCHOOL MATHEMATICS CLASSROOMS By Kristen Marie Kereluik A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Educational Psychology and Educational Technology - Doctor of Philosophy 2013 ABSTRACT SCAFFOLDING SELF-REGULATED LEARNING ONLINE: A STUDY IN HIGH SCHOOL MATHEMATICAL CLASSROOMS By Kristen Marie Kereluik This research explores the implementation and utilization of self-regulated learning (SRL) scaffolds (i.e. videos, journals, surveys) in online K-12 courses. This project is grounded in research on online education as well as theory and research around self-regulated learning in both online and offline contexts. This research is conducted through Michigan Virtual School's (MVS) learning management system (LMS) over two academic terms in six high school mathematics courses. Participating students (N=69) completed the pre-survey consisting of the PISA Student Characteristics Questionnaire (Artelt, Baumert, Julius-McElvany, & Peschar, 2003), a self-regulated learning assessment (Cleary and Zimmerman, 2004) and an online learning readiness assessment (Roblyer & Davis, 2008). Following the pre-survey participants were be randomly assigned to one of three conditions (experimental, control I, control II) and received either self-regulatory, general interventions (in the form of videos, journals, and short surveys), or no interventions--respective of condition. Research questions were as follows: 1) can computer mediated contexts be designed to fully facilitate and support adolescents’ use of selfregulated learning processes in learning; 2) does the presence of SRL scaffolding in online K-12 courses lead to gains in adolescent learners' a) self-efficacy and motivation or b) individual and environmental control, do a. and b. vary across dimensions of preparedness; 3) does the presence of SRL scaffolding lead to greater a) domain knowledge? b) student retention in online K-12 courses; 4) how does SRL scaffolding influence adolescent learners’ ability to successfully regulate their learning to produce improved student achievement outcomes in computer mediated contexts? Exposure to interventions had no effect on self-efficacy, motivation, individual, or environmental control outcomes nor did initial preparedness for online learning. Further neither exposure to interventions or initial preparedness had an effect on final course grade or retention. These results along with design considerations related to integrating SRL scaffolds in online K12 courses are discussed and possible reasons for the lack of effectiveness are outlined as well as future design iterations that may lead to effective SRL scaffolds for online K-12 students. ii ACKNOWLEDGEMENTS First and foremost, I would like to express my thanks and gratitude to my advisor Punya Mishra for agreeing to take me on as a student and for his constant support, guidance, and countless opportunities during my time as a Ph.D. student at Michigan State University. I largely attribute my growth as a writer, thinker, and researcher to his advice and feedback. I would also like to thank Cary J. Roseth who approached my work with enthusiasm and provided thoughtful feedback on many projects, including significant feedback on this dissertation. I would also like to thank Carrie Heeter who provided insightful commentary, even from afar, and helped guide this dissertation by opening new areas of research and opportunity. I would like to give a special thank you to Rand Spiro, my first unofficial advisor who recognized my ambition and convinced me to stay at Michigan State University. I greatly appreciated his commitment to my progress through the program and seeing me from start to finish. I also want to thank the staff and teachers of the Michigan Virtual University who not only enthusiastically agreed to allow me to conduct my research in their courses but provided support throughout the process. A special thanks to Joe Freidhoff for his support and guidance, as well as his willingness to help when any issues arouse and his gentle pushing of me to finish. Finally I would like to thank you family and friends for their love and support. iv iii TABLE OF CONTENTS LIST OF TABLES.......................................................................................................................viii LIST OF FIGURES........................................................................................................................ix CHAPTER 1 Introduction...............................................................................................................................1 Statement of the Problem...............................................................................................1 Purpose Statement..........................................................................................................3 Significance of the Study...............................................................................................5 Relevant Terms.........................................................................................................................7 CHAPTER 2 Literature Review.....................................................................................................................8 Online Education...........................................................................................................8 The rise of online education.................................................................................8 Affordances and challenges of online learning....................................................9 Michigan Virtual School......................................................................................9 Self-Regulated Learning..............................................................................................11 History and emergence of self-regulated learning.............................................11 Guiding theory of SRL.......................................................................................12 Self-regulated learning processes and strategies................................................15 Forethought...........................................................................................15 Performance control..............................................................................15 Self-reflection.........................................................................................16 The role of self-efficacy.....................................................................................16 Academic learning goals....................................................................................17 Models of self-regulated learning training.........................................................17 Zimmerman and colleagues..................................................................17 Azevedo and colleagues........................................................................19 Azevedo and Cromley (2004) ...............................................................19 Azevedo et al. (2005) ............................................................................20 Azevedo et al. (2008) ............................................................................20 Azevedo, Witherspoon, Chauncey, Burkett, & Fike (2009)...................20 Self-regulated learning in distance education....................................................23 Self-regulated learning in CMCs.......................................................................24 Computer-mediated scaffolds............................................................................25 CHAPTER 3 Methods.................................................................................................................................27 Research Questions...................................................................................................27 Research question 1..........................................................................................27 Research question 2..........................................................................................27 Research question 3..........................................................................................29 ivv Research question 4..........................................................................................30 Participants...............................................................................................................30 Research Design.......................................................................................................32 Procedures................................................................................................................35 Independent Variables..............................................................................................35 Experimental condition....................................................................................36 SRL Videos............................................................................................36 Planning journal...................................................................................36 Reflection journal..................................................................................37 End of unit surveys................................................................................37 Control I condition...........................................................................................38 Videos....................................................................................................38 Journals.................................................................................................38 End of unit surveys................................................................................38 Dependent Variables................................................................................................39 Pre/post survey................................................................................................39 Online learning self-assessment............................................................40 PISA Student Characteristics Questionnaire.........................................40 Self-regulated learning questionnaire....................................................41 Final course grade...........................................................................................41 Researcher observations..................................................................................41 CHAPTER 5 Results....................................................................................................................................42 Self-Regulated Learning in Computer Mediated Contexts......................................42 Research question 1........................................................................................42 Planning and self-reflection ..................................................................42 Performance control ..............................................................................43 Changes in Self-Regulated Learning.......................................................................44 Initial group differences..................................................................................44 Research question 2........................................................................................45 Self-efficacy and motivation by experimental group..............................46 Individual and environmental control by experimental group..............47 Self-efficacy and motivation by initial preparedness.............................48 Individual and environmental control by initial preparedness..............49 Domain Knowledge and Retention........................................................................50 Research question 3.......................................................................................50 Domain knowledge and retention by group...........................................50 Domain knowledge by intervention completion for the experimental condition...............................................................52 Research question 4.......................................................................................53 Abby.......................................................................................................54 Carrie.....................................................................................................54 James.....................................................................................................54 vi v CHAPTER 6 Discussion.............................................................................................................................56 Design Implications.................................................................................................56 Interventions based on previous research....................................................56 Design and interventions should remain pedagogically sound...................58 Easily accessible interventions..........................................................58 Long study duration...........................................................................59 Designed to maximize group differences....................................................59 Implications of Analyses and Results ....................................................................60 CMCs can support self-regulated learning by effective support requires significant human intervention.............................................................60 Embedded, hard, indirect, and static interventions are ineffective at supporting self-regulated learning .......................................................62 Embedded scaffolds.........................................................................63 Hard scaffolds.................................................................................63 Static scaffolds................................................................................ 64 There is a relationship between completing interventions and success in online course..................................................................................65 Successfully retaining and completing online courses depends on more than initial preparedness for online learning......................................66 CHAPTER 7 Limitations.............................................................................................................................69 Attrition....................................................................................................................69 Participant feedback.................................................................................................71 Low overall response rate........................................................................................71 No domain knowledge pretest ................................................................................72 Conclusions and Implications................................................................................................73 Future Research.....................................................................................................................75 APPENDICES APPENDIX A: Assessments ...............................................................................................78 APPENDIX B: Consent / Assent..........................................................................................85 APPENDIX C: Course Mapping..........................................................................................89 BIBLIOGRAPHY..........................................................................................................................90 vii vi LIST OF TABLES Table 1: Dimensions of quality and type of online education..........................................................8 Table 2: Assessment specificity guide used during diagnostic assessment component.................18 Table 3: Course enrollment and participation totals.....................................................................32 Table 4: Initial participation and membership groups in each course..........................................34 Table 5: Mean final course grade by course and semester...........................................................34 Table 6: Descriptive statistics for self-efficacy and motivation by experimental group................47 Table 7: Descriptive statistics for individual and environmental control by experimental group.................................................................................................48 Table 8: Descriptive statistics for self-efficacy and motivation by initial preparedness...............49 Table 9: Descriptive statistics for individual and environmental control by initial preparedness................................................................................................50 Table 10: Normality of final course grade by experimental group...............................................50 Table 11: Descriptive statistics for final grade and date of last access by experimental group.................................................................................................52 Table 12: Descriptive statistics for final course grade and date of last access by initial preparedness................................................................................................52 Table 13: Descriptive statistics for final course grade by intervention completion in the experimental condition......................................................................................53 Table 14: Course week number by semester..................................................................................89 viii vii LIST OF FIGURES Figure 1: Image: Self-regulatory process elements.......................................................................14 Figure 2: Chart: Condition progression chart................................................................................39 Figure 3: Chart: Participant flow chart..........................................................................................45 ix viii CHAPTER 1 Introduction We believe that by the year 2019 half of all classes for grades K-12 will be taught online… The rise of online learning carries with it an unprecedented opportunity to transform the schooling system into a student-centric one that can affordably customize for different student needs by allowing all students to learn at their appropriate pace and path, thereby allowing each student to realize his or her fullest potential… -Clayton Christensen Online K-12 education has seen significant gains in enrollment in the last decade, from 45,000 K-12 students in 2000 to more than 3 million in 2009 (Horn & Staker, 2011; Watson, Murin, Vashaw, Gemin, & Rapp, 2010). Florida Virtual School exceeded 300,000 course enrollments for the 2011-2012 school year and the Michigan Virtual School in particular registered over 20,000 course registrations in the 2012-13 academic year, both virtual schools are among the largest in the nation and are considered significant contributors to online education by the Evergreen Education Group (Watson, Murin, Vashaw, Gemin, & Rapp, 2012). This new era of online K-12 education is markedly different than it’s predecessor, distance education, in that a vast majority of online K-12 programs serve local students a the school or district level and provided blended, or supplemental instruction, instead of fully online programs (Horn & Staker, 2011; Watson et al., 2010). Statement of the Problem While online K-12 educational experiences are becoming available for more students nationwide, and in handful of states such as Michigan, required for graduation, research suggests 1 that online learning presents as many challenges as it does opportunities. In a review of published findings of research on hypermedia learning environments, Dillon and Gabbard (1998) found that hypermedia alone does not offer gains in comprehension (over print materials), but that hypermedia, unsurprisingly instead offers greater learner control. While this holds promise for individualized instruction, Dillon and Gabbard (1998) found that lower ability students demonstrated the greatest difficulty exploiting the greater learning control to their advantage. Additionally, Azevedo and colleagues are adamant that “too few learners are skilled at regulating and optimizing learning” in hypertext (online) environments (Azevedo & Cromley, 2004), and have built a research agenda around supporting and facilitating such learning (Azevedo, 2005; Azevedo & Cromley, 2004; Azevedo, Cromley, Thomas, Seibert, & Tron, 2003; Azevedo, Cromley, Winters, Moos, & Greene, 2005; Azevedo, Moos, Greene, Winters, & Cromley, 2008). Additionally, prior research by the author suggests that adolescents, while knowledgeable at learning intrinsically interesting content online, struggle to set appropriate learning goals and select appropriate strategies for academic learning in the same context (Kereluik & Mishra, 2011). While research on learning with hypermedia reports the struggles of students generally, data suggests that minority students may be at the greatest disadvantage in online virtual schools (Roblyer & Davis, 2008). Roblyer and Davis report that in at least one large virtual school minority students were less likely to enroll in online courses, and more likely to drop out once enrolled. Roblyer and her colleagues (Roblyer & Davis, 2008; Roblyer & Marshall, 2003) have lead initiatives to create instruments to accurately predict students performance in online learning environments, and identify students who would likely benefit from additional supports. Roblyer and colleagues have made great strides in identifying relevant and impactful student and 2 environmental characteristics as well as complex modeling of the interactions between each, however once identified there is no standard for what additional supports to provide. Identifying students who need assistance to be successful online is the first step in helping students succeed, however once identified there needs to be an empirical foundation for support, and a system in place to facilitate such support. One such support that is both empirically founded and shows promise in hypermedia learning environments is self-regulated learning (SRL) training, tutoring and support. Selfregulation has a strong and established history with learning and academic achievement (Brown, 1987; Paris & Winograd, 1990) and recent research has continued to demonstrate positive effects of SRL on motivation and academic achievement (Schunk, 1996; Wood, Bandura & Bailey, 1990). Additionally, as work by Azevedo and colleagues as demonstrated (Azevedo, 2005; Azevedo & Cromley, 2004; Azevedo, Cromley, Thomas, Seibert, & Tron, 2003; Azevedo, Cromley, Winters, Moos, & Greene, 2005; Azevedo, Moos, Greene, Winters, & Cromley, 2008) students trained in SRL and given SRL supports develop more robust conceptual models than students without SRL training and support. The greatest gains in conceptual understanding were shown among students receiving external regulation support from personal human tutors (Azevedo et al., 2008). While there has been marked success in the use of human tutors as supports in fostering self-regulation and metacognition, the use of human tutors is not feasible in most schools, nor is it scalable to larger institutions and online learning environments. The question then becomes, can computer mediated contexts be designed to fully facilitate and support adolescents use of self-regulated learning processes in learning? Purpose Statement This study attempts to address the question above and explore the theoretical proof-of3 concept behind an electronic system intended to scaffold student self-regulated learning. The SRL scaffolds will be fully integrated into an existing LMS system (Blackboard) in which learners can input and access specific self-regulatory processes targeted towards supporting learners cognitive, motivational, behavioral and contextual factors during all three phases of SRL. The SRL scaffolds were founded on theories of SRL by Cleary & Zimmerman (2004) and Zimmerman (1989). The prompts and scripted feedback were modeled off of the human tutoring prompts developed by Azevedo, Moos, Greene, Winters, & Cromley (2008). The SRL scaffolds broadly prompted students to plan time and effort, monitor progress towards goals, and reflect on their learning. Specifically, the SRL scaffolds prompted learners to set proximal learning goals, journal their plans and progress towards proximal and general course goals, utilize SRL strategies, and reflect on efforts and evaluative their performance. The proposed dissertation study follows a track of research on adolescent learning in contexts mediated by technology (Kereluik & Mishra, 2011; Kereluik & Mishra, 2012). Previous research studies conducted by the authors interviewed adolescents on how they use the Internet in their personal and academic lives. The most impactful finding was that adolescents not only think different about intrinsically motivated and academically motivated tasks online, but that they also possess different expectations and utilize different strategies for each type of online task. Given this finding the author developed a framework for adolescent acquisition of disciplinary knowledge in contexts mediated by technology (Kereluik & Mishra, 2012). This framework specifically targeted the findings from previous research and focused explicitly on scaffolding adolescents’ metacognition and self-regulated learning (Brown, 1987; Flavell, 1979; Zimmerman, 1989, 2002) in contexts mediated by technology. This study builds upon this 4 existing research track and investigate self-regulated learning through the development, and implementation of a research-driven external regulation system intended to scaffold student learning online. This study was guided by three pragmatic rules; first, all interventions were based on previous research. There is a robust history of SRL scaffolding that could not, and should not be ignored. Second, all interventions must remain pedagogically sound and must be feasible in typical classroom settings (virtual or otherwise). Clearly there are a number of variables that could have been manipulated within this study however this particular study was focused on demonstrating the concept of SRL scaffolding in contexts mediated by technology. Third, this study was designed to maximize the differences between the control and experimental groups. Again, this study is intended to serve as a foundational proof of concept and as such all experimental design considerations were made with this consideration in mind. Significance of the Study Through the development of a SRL scaffolding system, and strict adherence to the three pragmatic rules listed above, this study expected the following contributions to educational research and practice. First, this research extends the empirical base established by Azevedo and colleagues (2004, 2005, 2008, 2009) by developing content neutral regulatory scaffolds and ones that addresses all three phases of the self-regulated learning cyclical feedback look (Zimmerman, 1989, 2002). Second, this study guides the production of novel, theory-driven, self-regulatory scaffolds (either independently or as part of an existing LMS system). Additionally, regardless of the statistical significance of the findings they will be immediately available to significant stakeholders in K-12 online education. Third, and finally, this study serves as a model for how future educational, specifically educational technology, research should be conducted. 5 Educational psychology, while founded on the principled stance of true experiments of psychology, must be by definition, relevant to educational practice. While psychology is concerned with identifying and isolating individual processes, educational practice cannot isolate individual processes from the complex interaction that is learning. Educational research should be pedagogically sound in that it provides specific recommendations for implementation founded on a solid base of research. 6 Relevant Terms Online learning: instruction via a web-based educational delivery system that includes software to provide a structured learning environment (Watson et al., 2010). Self-regulated learning: learning in which learners are metacognitivly, motivationally, and behaviorally active participants, utilizing specified strategies in their learning process to achieve academic goals based on perceptions of self-efficacy (Zimmerman, 1989). Computer mediated contexts (CMC): contexts where the interaction between an individual and some content (information, communication with others, etc.) is facilitated through either individual or networked computers. Metacognition: awareness and monitoring of one's own cognition (Brown, 1987; Flavell, 1979). Hypermedia: and extension of hypertext in which graphics, audio, video, plain text and hyperlinks interact to create non-linear mediums of information (Wikipedia, 2012). LMS: Learning management system (LMS) is a software application for the administration, documentation, tracking, reporting and delivery of online education courses or training programs (Wikipedia, 2012). 7 CHAPTER 2 Literature Review Online Education The rise of online education. As of 2011, 40 states had a state virtual school or state-led online initiative with Florida, North Carolina, Michigan, Montana, Idaho and Georgia leading the way in providing quality online K-12 education options. According to Smith, Clark, and Blomeyer (2005), “state-level virtual schools are developed, administered, or funded in part by state government, and intended to provide online learning statewide” (p.7). State virtual schools are not alone in offering K-12 online education, virtual charter schools also play a major role in K-12 online education. Programs vary not only in funding models, and providers but in terms of course offerings and quality as well. There are numerous factors that affect the type and quality of online education; of those factors four comprise the most significant contribution in differentiating available online education programs (see Table 1). Table 1 Dimensions of quality and type of online education Comprehensiveness Reach Delivery Type of instruction Whether the online program provides full-time online enrollment or provides supplemental courses to students enrolled in another full-time school. Whether the online program operates within a school district, across multiple school districts, across a state, or in a few cases, nationally or internationally. Whether the online programs are synchronous or asynchronous. Whether the online program is fully online, fully face-to-face, or a combination of both. States leading the way in online K-12 education, Florida, Utah, Idaho, Ohio, Wisconsin, and Michigan, are at the forefront in part because they have begun to pass new legislation dictating the enrollment, offering, accessibility, and quality of online course providers. 8 Michigan's recent revisions of the School Aid Act opens supplemental online learning to every student in the state, initially guaranteeing each student the right to take up to two online courses per academic unit (i.e. semester, trimester) paid for by their local district. This legislation allows students and parents to choose online education options from a vast array of providers and is likely to contribute to substantial growth in enrollments in online courses across the state. Affordances and challenges of online learning. The remarkable growth in online education has not come without widespread skepticism about the current state and anticipated future of online education. Smith, Clark, & Blomeyer (2005) offer an overview of research on electronic distance education and online learning, reporting on meta-analyses funded by the North Central Regional Education Laboratory (NCREL) to determine the effects of technology on academic performance. Out of this NCREL initiative, Waxman, Connell, and Gray (2002) found small positive effect for computer-mediated instruction across 20 studies. Waxman, Lin, and Michko (2003) expanded on Waxman’s original work, including 42 studies. Waxman, et al. (2003) found positive effects for cognitive and affective outcomes for computer-mediated instruction but negative effects for behavioral outcomes such as persistence in learning tasks. Decades of research, like the review referenced above by NCREL suggest, teaching and learning online can work, but also that it presents several challenges. Specifically for minority (Roblyer & Davis, 2008) and students new to online learning, who typically take more than a year to adjust to the demands of online education (Watson et al., 2010). Michigan Virtual School. Michigan is among states leading the way in K-12 online education and has one of the largest state virtual schools, Michigan Virtual School (MVS), in the country with over 20,000 enrollments in the 2012-2013 school year. MVS is a nonprofit institution funding by legislative appropriations, course tuition and private grants (Watson et al., 9 2010). Michigan was the first state to pass an online learning requirement before graduation, with a handful of additional states following suit in recent years. The Michigan Department of Education requirement instructs students to: 1) take an online course, 2) participate in an online learning experience, or 3) participate in online experiences incorporated into each of the required credit courses of the Michigan Merit Curriculum. Additionally, as stated previously Governor Synder changes to the School Aid Act further open online learning options by requiring schools to allow students to initially take up to two online courses per learning term and more courses upon demonstration of success. Aware of the potential challenges learners may face in online courses, Michigan Virtual University (2010) outlined specific recommendations for Full-time Online Learners in an effort to provide guidance for large-scale implementation of full-time online learning options, and a learner readiness rubric. The MVU identified the following characteristics of successful online learners: Self-motivated: learners can and do direct their own learning environment towards fulfilling course requirements, and achieving success in the course. Independent learner: learners are able to take advantage of the affordances of the individualization and individual pacing of online learning environments. Computer literate: learners possess working knowledge of email, the Internet and basic keyboarding skills. Time management: learners must be able to organize and plan their learning according to their own needs. Effective writing skills: learners must be able to write clearly, through a variety of media, and express their ideas effectively. 10 Personal commitment: learners must posses a strong desire to participate, complete course tasks, and achieve academic success. While there are specific technological recommendations and characteristics specific to online learning, many of the characteristics of successful online learners presented by MVU are consistent with theories of self-regulated learning (SRL) including self-motivated learners, time management, and personal commitment to learning. Given this overlap it seems reasonable that an external regulation system designed to facilitate and scaffold SRL in online courses would be both necessary and highly effective, particularly for students who struggle to self-regulate. SRL will be explored in greater detail below including guiding theories, prior studies on SRL training, and research on SRL in CMCs. Self-Regulated Learning History and emergence of self-regulated learning. Self-regulated learning emerged in the 1980’s and gained prominence in the 1990’s around the same time that research on hypermedia learning environments were also gaining prominence as new frontiers in educational research (Dinsmore, Alexander, & Loughlin, 2008). Early theories of SRL positioned it as an integrated theory of learning, one that addressed cognitive, motivation, and contextual factors, instead of isolating the cognitive/metacognitive (Brown, 1987; Flavell, 1979) or social factors (Bandura, 1982, 1986; Corno & Mandinach, 1983). According to Boekaerts and Cascallar's (2006) comprehensive overview of SRL and answer to the question what is self-regulated learning, is “self-regulation refers to multi-component, iterative, self-steering processes that target one’s own cognitions, feelings, and actions, as well as features of the environment for modulation in the service of one's own goals” (p.199). This broad definition holds several important implications and assumptions regarding 11 general theories of SRL. First, the broad definition implies that SRL is not a linear process with a definitive start and finish; rather SRL is a cyclical process in which learners move through multiple times during learning tasks. Second, similar to the first assumption, this definition implies that SRL is not something one possesses or lacks, but rather SRL is a process involving multiple strategies and sub-processes. Third, this definition implies and assumes that learning, specifically SRL, is dependent on more than cognition, and that motivation, context and learning behaviors contribute significantly to the learning process. Fourth, this general definition implies that SRL is content neutral to the extent that SRL processes can exist within specific domains, as well as across domains. Broad theories of SRL, and accompanying assumptions, will be the guiding theoretical perspective of this study, though the specific theoretical framework informing this study will be Zimmerman’s social cognitive view of self-regulated learning (Zimmerman, 1989, 2002). Guiding theory of SRL. Zimmerman’s particular theory of self-regulated learning was chosen as the foundation for this study for three reasons: first, Zimmerman’s theory is widely accepted and highly regarded as one of the leading SRL theories; second, Zimmerman’s theory is very similar to the broad definition provided by Boekaerts & Cascallar’s (2006) and carries the same assumptions of SRL; third, Zimmerman’s theory unlike other theories, particularly those closely associated with SRL in hypermedia learning environments (Azevedo, 2005; Azevedo & Cromley, 2004;Azevedo et al., 2005, 2008), places equal emphasis on all phases of the SRL process including the final, self-reflection phase of SRL. Zimmerman (1989) defines self-regulated learners as those that are metacognitivly, motivationally, and behaviorally active participants, utilizing specified strategies in their learning process to achieve academic goals based on perceptions of self-efficacy. This 12 conceptualization of self-regulated learners is largely founded on the central role of three elements in the learning process; students' self-regulated learning strategies, students’ selfefficacy perceptions, and students' commitment to their academic goals. Zimmerman's theory of SRL is founded on the social learning theories and the work of Albert Bandura (Bandura, 1986), and as such subsequently holds several additional social-cognitive assumptions. The first assumption is triadic reciprocality, or the influence among individual, environmental, and behavioral processes, as such, self-regulated learning is not determined exclusively by individual processes. It should be cautioned however that reciprocality does not mean equality in terms of symmetry in strength, rather it refers to the influence of the individual, environmental and behavioral (Bandura, 1986). This first assumption is closely related to the third broad assumption of SRL and interaction among several factors. The second social-cognitive assumption is that the strength and causation among individual, environmental, and behavioral influences can be altered through personal efforts to self-regulate, outcomes of behavioral performance, and changes in the environmental contexts (Bandura, 1986). This assumption is integral to theories of SRL in that not only do individual elements interact, but that learners can manipulate, control, and even orchestrate this interaction. Given this, training learners to understand this interaction and fostering the skills necessary to effectively orchestrate the interaction is central to learning. 13 FORETHOUGHT PERFORMANCE CONTROL SELFREFLECTION Figure 1. Image: Self-regulatory process elements. Given these assumptions, Zimmerman posits that self-regulated learners are proactive and incorporate self-regulation processes, strategies, and self-beliefs and regulate their behavior in three cyclical phases (see Figure 1): forethought, performance control, and self-reflection (Cleary & Zimmerman, 2004). While the specific processes and strategies will be explained in greater detail below, forethought processes include goal setting and strategic planning as well as motivational aspects such as self-efficacy and outcome expectations. As such, forethought processes influence learners ability and willingness to engage in the learning activity and performance control phase, specifically self-control and self-observation. The performance control phase, and accompanying processes are directed towards maximizing learning by guiding the learning though systematic monitoring of performance (Cleary & Zimmerman, 2004; Zimmerman, 1989). The performance control phase also sets the stage for the third phase, selfreflection, in that learners gather information to be used in evaluating the effectiveness of their strategic plan, and in doing so improve the effectiveness of future learning activities. The final phase also involves evaluating one's performance and making any necessary adjustments as one begins the cycle again. 14 Self-regulated learning processes and strategies. Self-regulated learning strategies are the first essential element of SRL and are actions or processes directed at acquiring information or skills based on learner perceptions of agency or purpose (Zimmerman, 1989). In accordance with the triadic reciprocality detailed previously, self-regulatory strategies are broken into three classes; strategies designed to control behavior, strategies designed to control the environment, and strategies designed to control covert processes. Zimmerman and Pons (1986) identified self-regulated learning strategies and processes specific to environmental control such as selecting and arranging physical settings to make learning easier, and individual processes such as self-consequating, learner arrangement of imagination of rewards or punishments for success or failure. Further, self-regulated learning strategies can also be broken down into processes relevant to each of the cyclical phases, forethought, performance control, and self-reflection. Forethought. The primary strategies and processes of the forethought phase are goal setting and strategic planning. Zimmerman defines goal setting in accordance with Locke & Latham (1990) as, deciding on specific outcomes of learning, and strategic planning as selecting or creating a strategy to optimize one's learning (Zimmerman, 2002). When setting academic learning goals, Zimmerman (2002) and Bandura (1986) advocate for setting clear, attainable, proximal learning goals instead of in addition to general goals. Azevedo and colleagues (Azevedo & Cromley, 2004; Azevedo, Cromley, & Seibert, 2004) work on SRL in hypermedia learning environments identified planning, goals, and prior knowledge activation as essential forethought strategies. Performance control. The primary self-regulated learning strategies and processes of the performance control phase are self-control and self-observation (Cleary & Zimmerman, 2004). 15 Self-control processes guide the learning activity through the sub-processes of self-instruction, imagery, attention focusing, and task strategies, whereas self-observation is the systematic monitoring of one's learning performance (Zimmerman, 1989). Additionally, Zimmerman and Martinez-Pons (1986) identified organizing and transforming materials, information seeking, record keeping, self-monitoring, rehearsing, memorizing, and help-seeking as effective performance control strategies. Azevedo and colleagues (Azevedo & Cromley, 2004; Azevedo, Cromley, & Seibert, 2004) also identified a number of effective performance control strategies including, but not limited to, self-questioning, content evaluations, goal-directed searches, taking notes, summarization, hypothesizing, and deliberate environmental control. Self-reflection. The primary self-reflection processes and strategies are self-judgments and self-reactions (Cleary & Zimmerman, 2004). Self-judgments consist of self-evaluation, systematic comparisons of one's performance against specific criteria, prior behavior, or others; and causal attributions, perceived causes of outcomes of the learning activity. Self-regulated learners in the self-reflection phase evaluate their performance relative to self-standards, attribute poor performance to faulty strategies, and make strategic adjustments before beginning the learning again. Zimmerman and Martinez-Pons (1986) identified the following self-reflection strategies, self-evaluating and reviewing records. The role of self-efficacy. Self-efficacy is the second essential element of SRL (Bandura, 1986; Rosenthal & Bandura, 1978; Shunk, 1986; Zimmerman, 1989). Self-efficacy refers specifically to the perceptions of an individuals capability to direct necessary actions towards the performance and attainment of one's goals (Bandura, 1986; Zimmerman, 1989). Self-efficacy is deemed essential as it has been found to be related to two aspects of the cyclical feedback loop, learners' use of strategies and self-monitoring. Further, self-efficacy has been found to have a 16 strong influence on positive learning behaviors, and behavioral performance. Academic learning goals. Academic learning goals are the third central element of Zimmerman's (1989) theory of self-regulated learning. According to Zimmerman (1989) and Bandura (1986), academic learning goals must be proximal, as outcomes that are too general or distant to cue specific strategies present unnecessary learning difficulties. Instead, learners should identify and develop proximal goals as guides for developing a course of action in attainment of the desired learning goal. Additionally, proximal goals encourage greater selfefficacy (the second essential element of Zimmerman's SRL theory), as well as higher skill and intrinsic interest. These three central elements underlying Zimmerman's theory of SRL, strategies, selfefficacy, and academic goals are critical to developing self-regulated learners, unfortunately many learners have not yet developed the necessary understanding of these elements and struggle to learn in challenging environments, such as hypermedia learning environments and contexts mediated by technology. Fortunately, there has been considerable effort devoted to identifying effective SRL trainings in traditional learning environments, as well as SRL scaffolding in CMCs. Both research agendas will be discussed in further detail below. Models of self-regulated learning training. This study is theoretically guided by Zimmerman's (1989, 2000) social-cognitive theory or self-regulated learning, and empirically guided by Cleary and Zimmerman (2004) Self-Regulation Empowerment Program (SREP) and work by Azevedo and colleagues on SRL training and scaffolding in computer mediated contexts (CMCs). Zimmerman and colleagues. The SREP is a school-based training program incorporating aspects of Zimmerman's (2002) cyclical model of self-regulated learning. The problem solving 17 process utilized by the SREP is based on problem-solving models that involve defining problem areas, measuring behaviors, developing and implementing interventions, and assessing efficacy. The SREP builds on these models of problem solving by extending notions autonomy and shifting responsibility from external parties to individual learners. The SREP training process consists of two primary components: (a) the diagnostic assessment, (b) and developing the selfregulated learner. The SREP diagnostic assessment was slightly modified for the specific context of this study and utilized as a guide for the Self-Regulated Learning Questionnaire. The SREP diagnostic assessment begins at a general level, and moves towards specific strategy selection and implementation, see Table 2 for Cleary and Zimmerman's (2004) specific assessment guide. Table 2 Assessment specificity guide used during diagnostic assessment component Level of Specificity Class Assessment Question Assessment Procedures In which class(es) does the student struggle? Grading Criteria On which grading criteria in that class does the student perform poorly? Strategy Which study and self-regulation strategies does the student use to perform well in that class? Microanalytic How does the student select, use, and regulate specific strategies to perform specific tasks within that particular class? Review report cards, teacher interviews Review tests/quizzes/lab reports, teacher interview Retrospective self-reports, structured interview, study material review Think alouds, microanalytic assessment procedures Once the diagnostic assessment is complete, the focus transitions addressing and modifying any deficits discovered during the diagnostic assessment. Addressing the deficits and developing self-regulated learners is accomplished through three steps: (1) developing student empowerment, (2) expanding learners' range of available strategies and processes, and (3) enabling students to utilize the self-regulation cyclical feedback loop. Students undergoing the SREP are taught to 18 develop a strategic plan for attaining learning goals, select and implement effective learning strategies, monitor their learning performance, evaluate their effectiveness and learning outcomes, and make any necessary strategic adjustments. While not strictly experimental in nature, Zimmerman and Cleary (2004) report initial, anecdotal success with the SREP in actual academic settings, showing early positive effects on student achievement and motivation. Initial promise aside, a limitation of the SREP is that the diagnostic assessment and intervention are designed to be implemented by school professionals in close physical proximity to learners, and as such is out of reach of learners in online, or blended online learning contexts. Given this limitation, work by Azevedo and colleagues (Azevedo & Cromley 2004; Azevedo et al. 2004, 2005) will serve as an additional empirical guide to research on self-regulated learning in contexts mediated by technology. Azevedo and colleagues. The work by Azevedo and colleagues has focused on SRL training, and various SRL scaffolds and support structures for learners in hypermedia learning environments. Through the work of Azevedo and colleagues a more clear picture of the affordances and constraints of various SRL scaffolds has emerged as has a better understanding how to support learners in CMCs. While Azevedo and colleagues have conducted numerous studies on SRL in hypermedia learning environments, only a select few will be detailed below, selected for their conclusions and relevance to this particular study. Azevedo and Cromley (2004) examined the effect of one 30-minute SRL training session on empirically based SRL strategies intended to foster and support conceptual understanding of the circulatory system in a hypermedia learning environment. Researchers found that this training session facilitated a significantly greater conceptual shift in the experimental condition than the control. 19 Azevedo et al. (2005) examined the effectiveness of three scaffolding conditions, adaptive scaffolding (AS), fixed scaffolding (FS), no scaffolding (NS), on learning about the circulatory system in hypermedia learning environments. Researchers found that learners in the AS and NS conditions gained significantly more declarative knowledge than learners in the FS condition. Learners in the AS condition also demonstrated the largest shift in mental models of the circulatory system. Researchers concluded the gains demonstrated by learners in the AS condition were a result of learners effectively planning their learning, activating prior knowledge, monitoring their learning and progress towards their learning goal, utilizing effective strategies, and seeking help as needed. Researchers also concluded that fewer gains were demonstrated by the FS condition as a result of learners in the FS condition utilizing ineffective strategies impeding their overall learning. Azevedo et al. (2008) examined the effects of SRL and externally-facilitated self-regulated learning (ERL) when learning about the circulatory system in hypermedia learning environments. ERL was achieved through the use of human tutors assigned a tutoring script (see Appendix), which generally prompted students to activate prior knowledge, plan their time and effort, monitor their progress towards their learning goal, and utilize SRL strategies. Researchers found that learners in the ERL condition gained significantly more declarative knowledge about the circulatory system. Through the use of think-alouds, researchers determined this was due to learners in the ERL condition regulating their learning through the activation of prior knowledge, monitoring their learning, deploying effective strategies, and seeking help when needed. Azevedo, Witherspoon, Chauncey, Burkett, & Fike (2009) explored the development and small-scale pilot implementation of MetaTutor, a hypermedia learning environment designed to train and foster SRL on the circulatory system. This was a non-experimental study however 20 preliminary data suggested that learners possessed little declarative knowledge on SRL and as such could theoretically benefit from such a system. The MetaTutor interface consisted of general learning goals set by the teacher, and learner generated proximal learning goals. The MetaTutor interface also presented potential SRL strategies and processes, from which learners selected those they intended to utilize. The interface also prompted planning, monitoring, and strategy utilizing when deemed appropriate by the system. These four studies, both individually and collectively, present several important considerations regarding the design and development of the SRL scaffolding system. First, as demonstrated by Azevedo and Cromley (2004), an initial orientation and introduction to SRL, including why SRL is important or relevant, SRL phases, and SRL processes, can produce shortterm benefits in terms of conceptual understanding and possibly set the stage for long-term use. Second, the tutoring script utilized by Azevedo et al. (2008) was demonstrated as effective and as such will be adapted to serve as the basis for the SRL scaffolds and prompts. Additionally, both the 2008 and 2009 Azevedo et al. studies placed great emphasis on the processes of planning and monitoring, as well as utilizing effective SRL strategies. These three SRL processes, demonstrated effective in the 2008 study, and comprising the foundation of MetaTutor, also played a significant role in the design of this study. Third, the this system utilized both general as well as learner-generated proximal learning goals advocated by Zimmerman (1989) and Bandura (1986) and utilized by Azevedo et al. (2009). While the work by Azevedo and colleagues has provided an empirical model for SRL in CMCs, as well as provided specific design recommendations, collectively these studies have limitations and implications for general learning in CMCs, and specifically online learning. First, the experiments by Azevedo and colleagues were relatively short in duration, with all 21 interventions lasting approximately one hour. Returning to the original aims of this study, one of which was to remain pedagogically relevant, the short duration is a limitation in that learning, particularly online learning, is not isolated to pre-defined hour-long sessions. This study addresses this limitation and retains pedagogical relevance through providing longer-duration SRL scaffolding with both general course, and assignment specific scaffolds. Second, the work of Azevedo and colleagues admittedly almost exclusively focused on the use of cognitive prompts. While in rare cases researchers prompted learners interest evaluation, a vast majority of prompts; planning, monitoring, and strategy, were designed to scaffold learners cognition. This is a limitation in that the broad definition of SRL provided by Boekaerts and Cascallar (2006), and Zimmerman's (1989) specific definition state that SRL is comprised of not only cognition and metacognition factors, but learners' motivation, feelings of self-efficacy, behavior, and environmental features. To have a complete SRL system the system must be design to support and scaffold all aspects of SRL, to at least some extent, otherwise it is a cognitive/metacognitive scaffold and the full academic benefits associated with SRL cannot be assumed. Third, the SRL training and scaffolding by Azevedo and colleagues also exclusively focused on the forethought and performance control phase of SRL (Cleary & Zimmerman, 2004). As evidenced by the tutoring script utilized by Azevedo et al. (2008) SRL prompts focued on processes associated with forethought (setting learning goals, planning time / effort) and performance control (utilizing strategies, monitoring learning, evaluate success towards goal) and did not prompt learners to evaluate their overall performance, self-efficacy, motivation, interest, or possible negative environmental features. These self-reflective processes are important in helping learners recognize their learning patterns and fostering long-term changes in 22 learning behavior and SRL. Fourth, experiments by Azevedo and colleagues focued on very specific content knowledge, knowledge of the circulatory system, even with respect to the design of the MetaTutor which is intended to scaffold only this specific content. As such, MetaTutor cannot be easily adapted to other content areas. This limitation again speaks to the aim of the study to remain pedagogically relevant. SRL is a content neutral process, and as such a content neutral scaffolding system is not only appropriate but easily adapted to any content. Self-regulated learning in distance education. There is little research looking specifically at self-regulated in synchronous and asynchronous online courses and the implications of realtime versus student-paced course progressions. However, online education was born out of distance education and aside from the specific digital technology holds many of the same characteristics of distance education, including but not limited to physical and in many cases, temporal separation of teacher and student. A recently published meta-analysis by Means, Toyama, Murphy, and Bakia (2013) on the effectiveness of online and blended learning contexts also discusses previous meta-analyses on distance learning which largely found no difference in student learning outcomes between students who took a face-to-face and those who took a distance education course. Means et al. do note however that student self-pacing in distance learning courses a small positive effect on student learning outcomes (Bernard et al., 2004). That finding may not be directly related to SRL however it holds an important implication, self-pacing and the ability to progress individually through a course relies heavily on SRL strategies. Students separated from their teachers, online or off, must be able to plan and execute their progression through the course and must monitor their own progress towards the final end goal. Without well developed SRL strategies students would not be able to take advantage of the 23 benefits afforded by self-pacing in distance education courses. Self-regulated learning in CMCs. Specific empirical limitations aside, as evidenced by Azevedo and colleagues computer based-scaffolds hold immense promise for online education and other models that utilize CMCs. CMCs have been widely utilized since the mid 1990's in the field of education to support the learning of complex or challenging topics through the promise of individualized instruction (Azevedo & Jacobson, 2008; Devolder, van Braak, & Tondeur, 2012; Lajoie & Azevedo, 2006). CMCs, by nature of design, are open-ended, nonlinear, non-sequential learning environments that afford multiple representational formats (text, graphical elements, audio, video) and greater learner control. These characteristics, while offering several affordances, also present new challenges to learners. CMCs require learners to have developed the skills and knowledge necessary to overcome these unique challenges, to be successful learners. Skills and knowledge shown to influence learning in CMCs include motivational characteristics, cognitive, and metacognitive skills (Hannafin, Land, & Oliver, 1999). Research also suggests that learners in CMCs need to posses greater engagement than in face-to-face learning contexts (Winters, Greene, & Costich, 2008). Much of the research on learning in CMCs has focused on fostering and supporting selfregulated learning, and the most prominent track of research has been by Azevedo and colleagues (discussed at length above). Azevedo's work is underscored by the observation that learners, especially young learners or learners new to CMCs, are not able to take advantage of the affordances of CMCs because they often lack self-regulation skills necessary to overcome the challenges (Azevedo, 2005). Learning can be facilitated, and challenges mitigated, by welldesigned CMCs, that is CMCs with design features that account for the necessity of selfregulated learning processes (Devolder et al., 2012). One particular design feature is the use of 24 scaffolds, or technology-mediated supports for learners engaged in specific learning tasks (Dabbagh & Kitsantas, 2005; Sharma & Hannafin, 2007). Computer-mediated scaffolds. Devolder et al. (2012) provide a comprehensive overview of computer-mediated scaffolds, providing an updated definition of scaffolds as tools, strategies, or guides that supports learners' acquisition of knowledge and understanding. Echoing research on the challenges of learning in CMCs Devolder et al. assert that high levels of knowledge acquisition and understanding are out of reach of many learners' without such systematic supports. Based on work by Hannafin et al. (1999), Devolder et al. identify four types of scaffolds. Conceptual scaffolds: those that guide learners on what to consider when a problem or task is pre-defined. Metacognitive scaffolds: those that offer different ways to think about problems, or different strategies to be considered. Procedural scaffolds: those that guide learners in using available features and affordances of CBLEs. Strategic scaffolds: those that guide learners on how to approach learning activities or tasks. Additionally, Devolder et al. identify four non-exclusive dimensions along which scaffolds can exist. Embedded --- non-embedded: Embedded scaffolds are integrated into the learning environment; non-embedded scaffolds depend on the personal initiative of the learner. Hard/fixed --- soft/adaptive: Hard scaffolds are static or fixed whereas soft are customizable and negotiable. Dynamic --- static: Dynamic scaffolds are interactive and provide learner feedback, static 25 scaffolds offer fixed guidelines and procedures. Direct --- indirect: Direct scaffolds instantly deliver instruction whereas indirect scaffolds subtlety guide the learning process. Devolder et al.'s critical analysis had two findings directly relevant to the present study. First, most scaffolds in CMCs were conceptual or cognitive, and related directly to problem-solving, in the form of hints or domain specific strategies. The analysis revealed, in line with previous research on scientific knowledge acquisition and integration (Kauffman, Xun, Kui, & Ching-Huei, 2008) that these prompts were generally effective at scaffolding learners' cognition. Additionally, most of the cognitive prompts were activated during the performance control phase as strategy activators. Second, following cognitive scaffolds, strategic scaffolds were the second most popular type of scaffold. While the name given to each type of strategic prompt was different between different studies, strategic prompts generally took one of two forms, prompts designed to impart either domain-specific or domain-general information. Devolder et al. however caution against domain-general prompts as previous literature has demonstrated that domain-general prompts are largely ineffective when used alone. Additionally, Devolder et al. caution that all scaffolds and types of prompts differ in effectiveness based on the method of delivery. Third, very few studies examined the effectiveness of scaffolds designed to support learners motivation, behavior, and context - three critical areas of Zimmerman's (1989) theory of self-regulated learning. This research study directly addressed this gap by scaffolding learners cognitive, behavioral, affective, and environmental processes along different dimensions of the delivery continuum. 26 CHAPTER 3 Methods Research Questions The research questions follow the three pragmatic rules, to be based on previous research, to remain pedagogically sound, and maximize the effect of the intervention. The three research questions are presented in detail below. Research question 1. Can computer mediated contexts be designed to fully facilitate and support adolescents’ use of self-regulated learning processes in learning? The first research question emerges from, and addresses the limitations of work by Azevedo and colleagues on SRL in CMCs. While Azevedo demonstrated that hypermedia learning environments can support learners' cognition and conceptual understanding during the forethought and performance control phases, other elements of SRL theory and the final phase of SRL, self-reflection, were not included in their study. This study, specifically addressed through the first research question, attempted to determine if computer mediated contexts can support all elements and phases of SRL. This research question will be answered primarily by researcher observations and participant journal data collected within the LMS system, both of which, along with all the following measures are explained in greater detail below. Research question 2. Does the presence of SRL scaffolding in online K-12 courses lead to gains in adolescent learners': a. self-efficacy and motivation? b. individual and environmental control? c. Do a. and b. vary across dimensions of preparedness? The second research question, while related to the first is distinctly different in that it 27 attempted to determine in what areas exactly CMCs could facilitate and support SRL. Research question 2.a. emerged out of the assertion of self-efficacy as the second essential element of SRL (Bandura, 1986; Rosenthal & Bandura, 1978; Shunk, 1986; Zimmerman, 1989). Self-efficacy is essential to SRL, and warrants individual consideration as it is closely related to all stages of the cyclical feedback loop, and behavioral and environmental control. This question was answered by comparing participants' pre/post-survey scores on the Motivation Strategies and Self-Related Beliefs subtests of the PISA Student Characteristics Questionnaire (Artelt, Baumert, Julius-McElvany, & Peschar, 2003). Hypotheses for research question 2.a. are as follows: Experimental hypothesis: H1: Online K-12 students exposed to self-regulated learning scaffolds will demonstrate greater gains in self-efficacy and motivation. Null hypothesis: H0: Online K-12 students exposed to self-regulated learning scaffolds will not demonstrate greater gains in self-efficacy and motivation. Research question 2.b. emerged from the assertion that strategies are the first essential element of SRL (Zimmerman, 1989). Strategies in Zimmerman's theory are broken into three classes; strategies designed to target behavioral control, environmental control, and individual covert processes. This research question investigated how effectively the system could facilitate and scaffold all areas of SRL strategy. This research question was answered by comparing participants' pre/post-survey scores on the Learning Strategies subtest of the PISA Student Characteristics Questionnaire (Artelt, Baumert, Julius-McElvany, & Peschar, 2003). Hypotheses for research question 2.b. are as follows: Experimental hypothesis: H1: Online K-12 students exposed to self-regulated learning scaffolds will demonstrate greater gains in individual and environmental control. 28 Null hypothesis: H0: Online K-12 students exposed to self-regulated learning scaffolds will not demonstrate greater gains in individual and environmental control. Research question 2.c. was intended to investigate whether there are any differences in facilitating and scaffolding SRL across learner preparedness for online learning as evidenced through the Online Learning Preparedness Assessment developed by Roybler and colleagues. This research question was answered by comparing learners who were deemed "prepared" for online learning against those deemed "unprepared" across research questions 2.a. through 2.b. Hypotheses for research question 2.c. are as follows: Experimental hypothesis: H1: Students deemed “prepared for online learning” will demonstrate greater gains in self-efficacy and motivation. Null hypothesis: H0: Students deemed “prepared for online learning” will not demonstrate greater gains in self-efficacy and motivation. Research question 3. Does the presence of SRL scaffolding lead to greater: a. domain knowledge? b. student retention in online K-12 courses? Research question 2.a. emerged out of the essential nature of academic learning goals and knowledge acquisition in SRL theory (Zimmerman, 1989). This question was targeted specifically at the pragmatism of the SRL scaffolding system in supporting online learning. This question was answered through the final course grade. Research question 3.b. was designed to elicit information about the effectiveness of SRL scaffolds in retaining more students, for longer periods of time in online K-12 courses. This information although apart from learning variables and measures is important in any educational context. This question was answered through date of last access tracked through the course LMS. 29 Hypotheses for research question 3 are as follows: Experimental hypothesis: H1: Online K-12 students exposed to self-regulated learning scaffolds will persist longer and achieve greater course completion percentages than students not exposed to such scaffolds. Null hypothesis: H0: Online K-12 students exposed to self-regulated learning scaffolds will not persist longer and achieve greater course completion percentages than students not exposed to such scaffolds. Research question 4. How does SRL scaffolding influence adolescent learners’ ability to successfully regulate their learning to produce improved student achievement outcomes in computer mediated contexts? The fourth research question was designed to understand how SRL scaffolding in CMCs supported participants learning. This questions relied primarily on the ability to reject the null hypotheses of research questions 2 and 3 and was answered using the overall results of those analyses as well as participant data from the Self-Regulated Learning Questionnaire. Participants The population of interest for this study was adolescents ages 14-18 enrolled in at least one fully online course. The sample was drawn from students enrolled in Algebra 1A/1B, 2A/2B, and Geometry A/B with the Michigan Virtual School (MVS), operated by the Michigan Virtual University (MVU). These specific courses were chosen for their consistently high enrollment relative to other courses (for a full summary of MVS enrollments see MVU, 2012), low course completion percentages, and consistency of instructor across courses. The large enrollments of these courses offered greater access and variability between participants. Additionally, the 30 completion rates of these courses also offered greater variability and suggest that a number of students could benefit from additional learning supports. Enrollment totals for each course are provided in the table 3 below as well as participation totals and attrition. Total enrollment reflects the total number of students enrolled at the start of the semester. This number was used as the enrollment statistic to accurately reflect course enrollment and attrition, even though there is high fluidity of enrollments in the first several weeks. Total initial participation reflects the number of students who assented to participated and completed the pre-survey. Rate of attrition is also provided, high rates of attrition were observed for all courses in both semesters. Possible reasons for this are explained in more detail in the limitations section. 31 Table 3 Course enrollment and participation totals Course Count Total Enrollment Mean Course Comple -tion Course Attrition Count Intervention Participation .1 Count Total Initial Participation 6 Rate of Attrition 3 Count Total Final Participation 2 Algebra 1A S1 Algebra 1B S1 Algebra 2A S1 Algebra 2B S1 Geometry A S1 Geometry B S1 Algebra 1A S2 Algebra 1B S2 Algebra 2A S2 Algebra 2B S2 Geometry A S2 Geometry B S2 21 73.81 17 59.80 .18 10 0 3 .7 31 60.63 .2 9 4 2 .78 11 59.32 .18 2 1 0 1 28 71.20 .18 17 7 5 .7 7 50.96 0 2 0 0 1 14 77.78 .14 6 2 1 .83 26 79.37 .19 1 0 0 1 5 38.61 0 0 0 0 0 22 87.45 .09 6 0 2 .66 23 62.60 .04 7 3 0 1 29 76.38 .2 3 2 3 0 .67 Research Design Each of the courses (algebra 1A/1B, algebra 2A/2B, geometry A/B) had all four of the possible experimental conditions (no consent, control I, control II, and experimental) within the section (see Table 4). This was done to control for instructor effects and was achieved through membership groups and adaptive content release within the Blackboard LMS. The no consent condition (A0) consisted of students who did not receive parental consent to participate in the 32 study, these students were not exposed to any interventions and no data was collected from them, all students initially defaulted to this condition until assenting to participate in the study. Assenting students were then randomly assigned to the control I, control II, and experimental conditions. Random assignment was done through randomly listing batches of assenting students and using a random number generator to select a starting point after which participants were given a 1 (experimental), 2 (control I), or 3 (control II). Participants were weighted into the experimental and control I conditions. As final participation was unknown during early batch sorting priority was given to the two conditions to ensure comparability across groups. The control II condition was a true control, students in this condition completed the presurvey near the start of the course and a post-survey near the end. Students in this condition were not exposed to any additional interventions. The control I condition and the experimental condition were exposed to completely identical interventions including pre/post-survey; videos; journals; and end of unit surveys. The difference between the two conditions was in the content of the interventions, the experimental condition interventions focused on SRL whereas the control I condition interventions were general in nature, and unrelated to SRL. 33 Table 4 Initial participation and membership groups in each course Algebra Algebra Algebra Algebra Geometry Geometry 1A 1B 2A 2B A B S1 S2 S1 S2 S1 S2 S1 S2 S1 S2 S1 S2 A0: No consent; no intervention Control II - - - - - - - - - - - - 1 1 3 0 3 0 0 1 3 2 0 1 Control I 2 2 3 0 3 0 1 2 7 2 1 1 Experimental 3 3 4 1 3 0 1 3 7 2 1 1 Table 5 Mean final course grade by course and semester Algebra Algebra Algebra Algebra Geometry Geometry 1A 1B 2A 2B A B S1 S2 S1 S2 S1 S2 S1 S2 S1 S2 S1 S2 A0: No consent; no intervention (course mean) Control II 74 78 60 79 61 39 59 88 71 63 51 76 96 73 62 - 42 - - 98 63 48 - 95 Control I 81 15 44 - 33 - 66 68 54 22 70 96 Experimental 59 47 70 10 65 - 69 56 67 55 68 75 34 Procedures Participants were contacted and recruited through the MVU LMS associated with each of the six courses. Parental consent forms and adolescent assent forms were placed at the start of the pre-survey. Parents and adolescent participants were asked to enter their initials as acknowledgement of consent consistent with Institutional Review Board approval. Consent was required to continue to the rest of the pre-survey. Pre/post-survey data was collected through and stored securely on Opinio. Journals and end of unit surveys were embedded in the Blackboard LMS, data from these interventions was stored securely in the LMS. Independent Variables All interventions were embedded within corresponding units of each of the six courses. Interventions were not deliberately placed immediately at the start of the semester to allow time for participants to enroll in the course and to complete and return the consent/assent forms. Participants were reminded to complete all research related activities four times throughout the semester (weeks 7, 13, 16, and 18). Reminders were deliberately placed more closely near the end of the course for two specific purposes. First, historically a large number of students complete a majority of the work for the course in the latter half of the semester. Second, postsurvey data was crucial to the study and therefore participants were reminded frequently as they were finishing the course to complete the post-survey. See appendix D for screen shots of each reminder. 35 Experimental condition. Interventions for the experimental condition were within one or more of the three phases of SRL identified by Zimmerman (1989), forethought, performance control, and self-reflection. There are four distinct interventions: SRL introductory and refresher videos, planning journal, reflection journal and end of unit survey, each intervention is explained in further detail below in conjunction with Devolder et al.'s (2012) scaffold types and Zimmerman's phases of SRL. SRL Videos. Self-regulated learning videos were designed as metacognitive and procedural scaffolds of forethought and performance control. SRL videos were approximately 3:30 in duration and focused on introducing concepts of SRL, reviewing concepts of SRL (in video 2 and 3) and providing study tips and suggestions. Videos can be found at http://www.youtube.com/channel/UCrNKYF-5If5C7DAZ7MloOqw/videos, experimental condition corresponds to Group 1 in the video titles. According to Devolder et al.'s (2012) criteria SRL videos were: embedded, fixed, static, and direct. Limitations arising from this categorization are discussed in chapter 6. Planning journal. Planning journals were designed as metacognitive scaffolds of forethought. Participants in the experimental condition were asked to complete a short planning journal at the start of units 2 though 5, following the SRL or general video. All experimental journal prompts were taken directly from Cleary and Zimmerman (2004) and are comprised of the following questions; do you have a goal you are trying to achieve this unit? What strategies are you planning on using to achieve your goals for this unit? How do you decide which strategy to use when preparing for the unit? How sure are you that you can get the grade you want on the unit test? In effort to maintain consistency across participants and conditions no feedback was provided to participants regarding their journal entries. This is discussed in further detail in the 36 limitations section. According to Devolder et al.'s (2012) criteria planning journals were: embedded, hard, static, and indirect. Limitations arising from this categorization are discussed in chapter 6. Reflection journal. Reflection journals were designed as conceptual and metacognitive scaffolds of performance control and self-reflection. As participants were separated by both space and time it was difficult to directly observe performance control, because of these limitations, performance control was measured indirectly though the reflection journal and end of unit survey, placed at the end of the unit following the unit test. Self-reflection was scaffolded similarly to forethought through end of unit journals. These journals were placed at the end of units 2 though 5 (or corresponding units) following the unit test, all experimental self-reflection journal prompts were directly from Cleary and Zimmerman (2004). Self-reflection journals were comprised of the following questions; did you meet your goals for this unit? How satisfied are you with your performance on your the unit test? What is the main reason why you got the grade you did on your unit test? What do you need to do to improve your performance on your next unit test? According to Devolder et al.'s (2012) criteria reflection surveys were: embedded, hard, static, and indirect. Limitations arising from this categorization are discussed in chapter 6. End of unit surveys. End of unit surveys were designed as conceptual and metacognitive scaffolds of performance control and self-reflection. Questions on the end of unit survey were drawn from Clearly and Zimmerman (2004) and the PISA Student Characteristics Questionnaire (Artelt, Baumert, Julius-McElvany, & Peschar, 2003). The end of unit survey for the experimental condition contained the following questions; did you have to try to motivate yourself to study for the unit quiz? Did you keep track of where and or how long you studied for your unit quiz? Did you utilize any specific strategies during the unit or on the unit test? I'm 37 confident I can do an excellent job on assignments and test. I'm confident I can master the skills being taught. According to Devolder et al.'s (2012) criteria end of unit surveys were: embedded, hard, static, and indirect. Limitations arising from this categorization are discussed in the limitation section. Control I condition. The control I interventions mirrored the experimental interventions in terms of type of quantity and intervention type (videos, planning journal, reflection journal, and end of unit survey) however the control I interventions differed in content and were general in nature, unrelated to SRL. Videos. The control I video were approximately 30 seconds and were a general introduction to the course overall. This introduction also served as the introduction of the experimental videos, however the experimental videos continued on with specific SRL content whereas the control I ended after the general introduction. Videos can be found at http://www.youtube.com/channel/UCrNKYF-5If5C7DAZ7MloOqw/videos, control I corresponds to Group 2 in the video titles. Journals. The control I beginning of unit journal prompts were composed of the following questions: how is everything going in the course so far? Are you looking forward to the upcoming unit? The control end of unit journal asked: how can you use what you learned in this unit in future mathematics courses? How can you use what you learned in this unit in real world situations? End of unit survey. The control I end of unit survey was comprised of the following questions; did you like the content of the unit? How would you rate this unit compared to the previous units? Overall how would you rate this course so far? Will you use what you learned this unit in future mathematics courses? Will you use what you learned in this course in future 38 mathematics courses? All interventions according to Devolder et al. (2012) categorizations correspond to the experimental interventions described above. Experimental  Unit  1     Control  IUnit  1     Control  II  Unit  1     Pre-­‐test   Pre-­‐test   Pre-­‐test   Unit  2   Unit  2   Unit  2   SRL  Video,  Journal,  Survey   General    Video,  Journal,   Survey   [No  interventions]   Unit  3   Unit  3   Unit  3   SRL  Video,  Journal,  Survey   General    Video,  Journal,   Survey   [No  interventions]   Unit  4   Unit  4   Unit  4   SRL  Video,  Journal,  Survey   General    Video,  Journal,   Survey   [No  interventions]   Unit  5   Unit  5   Unit  5   Post-­‐test   Post-­‐test   Post-­‐test   Figure 2. Chart: Condition progression chart. Dependent Variables Pre/post-survey. The pre/post-surveys were intended to measure participants growth in SRL throughout the semester and specifically answer research questions 1, 2, and 4. Participants in all three conditions completed the pre/post-survey. The pre/post-survey are comprised of questions adapted from the Online Learning Self-Assessment (Roblyer & Davis, 2008), the PISA 39 Student Characteristics Questionnaire (Artelt, Baumert, Julius-McElvany, & Peschar, 2003), and the Self-Regulated Learning Questionnaire (Cleary & Zimmerman, 2004). The complete pre/post-survey is can be found in the appendix. Online learning self-assessment. The Online Learning Self-Assessment was developed by Roblyer for the Online Masters in Education program at the University of Tennessee, Chattanooga. All questions and subsequent qualities are based on previous research indicating an association with successful online students. PISA Student Characteristics Questionnaire. The Program for International Student Assessment (PISA) Student Characteristics Questionnaire was developed as part of the OECD's 2000 Student Approaches to Learning international survey. The survey, and subsequent report was designed to offer a detailed analysis of particular learning characteristics across several countries. The results from the 2000 survey confirmed a strong link between student approaches to learning and student learning outcomes. PISA survey is based on 11 learner characteristics, these characteristic categories are grouped under three broad elements of motivation, self-related beliefs and learning strategies. The quality of the Student Characteristics Questionnaire were tested in PISA field trials (Peschar, Veenstra and Molenaar, 1999) wherein the validity and reliability of the scales proves to be satisfactory, with reliability coefficients on average, all above 0.7. This questionnaire was selected for three specific reasons, first, while not explicitly focusing on SRL by name, many of the constructs of the PISA questionnaire map directly onto SRL elements and processes. Second the PISA questionnaire was specifically designed for utilization with adolescent populations. Third, the survey went through rigorous development and field-testing resulting in satisfactory validity and reliability estimates. The PISA questionnaire data was utilized to answer research question 1, 2, and 4. 40 Self-regulated learning questionnaire. The SRL learning questionnaire was modeled off Cleary & Zimmerman's (2004) Self Regulation Empowerment Program (SREP) diagnostic assessment. The assessment was self-report and asked participants questions related to their learning history and general use of self-regulated learning strategies. The assessment followed the format established by Cleary & Zimmerman of moving from general to specific microanalytic questions. The self-regulated learning questionnaire was primarily short-answer format and provided qualitative data specifically in response to research question 1. Final course grade. The final course grade is a metric utilized specifically by the Michigan Virtual School. The total refers to how many of the total available points an individual student earned. The MVS uses this as their official metric as they do not award grades (that is done by the students local district based on the final course grade provided by MVS). The final course grade was utilized as a metric to gauge learners overall learning. While not a perfect measure of knowledge and learning, the scaffolds were designed to scaffold learners academic learning, and assist students struggling in CMCs. As such, they were expected to produce academic benefits, one such benefit being successful final course grade evidenced though a high course completion total. Researcher observations. As this was a proof of concept and in many was an exploratory study researcher observations regarding design considerations, intervention implementation, and participant interactions provided unique insight and contributed to answering many of the research questions. 41 CHAPTER 5 Results Self-Regulated Learning in Computer Mediated Contexts Research question 1. Research question 1 asked, can computer mediated contexts be designed to fully facilitate and support adolescents’ use of self-regulated learning processes in learning? While previous research suggests this answer to be "yes", this question sought to understanding if computer mediated contexts can fully support self-regulated learning. Further, this question sought to identify what challenges and opportunities CMCs present to SRL scaffolding. Research question 1 was answered through an exploratory case study in which the online course and embedded SRL scaffolds constituted a case. Data was drawn from researcher observations regarding the design and configuration of the interventions as well as observations of student interactions with the interventions. In the interest of clarity each of the three phases of self-regulated learning (planning, performance control, and self-reflection) will be discussed first with respect to the specific CMC utilized in the study, and second with CMC's generally. Planning and self-reflection. Facilitating and supporting planning and self-reflection was possible through the CMC utilized in this study, the Blackboard LMS. Students (albeit a small number n=19) did plan and reflect in the journals, and did set goals, for that reason CMC's can facilitate and support at least part of SRL. The specific LMS utilized in this study provided private student-teacher journals allowing students a space to confidentially plan for the upcoming unit and reflect on previous units progress. As discussed in chapter 2, the planning and reflecting journals were (according to Devolder et al.'s, 2012 criteria): embedded, hard, indirect, and most importantly static. Planning and reflecting journals were deliberately left as static (providing no feedback to students) as to not provide unequal interventions either between participants or 42 between experimental groups. So while the CMC was able to support SRL it was not necessarily able to fully support participants, without significant human intervention, by providing feedback on their planning, goal setting, and reflection. Certainly the CMC provided a complete platform for planning and reflecting, but required human intervention to fully support and facilitate SRL, a theme that will be discussed more thoroughly later. Generally CMC's would likely suffer from the same limitations as the LMS utilized in this study. Given the highly detailed and specific nature of individual courses (including content, assignments, and exams) providing the kind of specific targeted SRL feedback that is associated with student learning gains (Azevedo et al., 2005) is difficult without significant human intervention. While sophisticated study software like iHomework and examtime can offer a platform to input and manage learning goals and tasks it does not make evaluative judgments and provide real-time feedback to students. Performance control. Performance control scaffolds were perhaps the most difficult to support and facilitate through CMC's. Performance control is comprised of two primary processes self-control and self-observation (Cleary & Zimmerman, 2004). While some subcomponents of self-control and self-observation can be easily achieved through CMC's (tracking mouse clicks, search terms, recording and timing page views, etc.) the difficulty arises from providing as-needed or on-demand feedback on learner progress towards a goal, identifying misconceptions, and suggesting appropriate and effective strategies. Researcher observations of designing and implementing reflect this difficulty. A significant barrier to designing and implementing performance control scaffolds, particularly in the CMC utilized in this study, was the physical and temporal separation of participant and researcher. Additionally, even in an online course delivered through an LMS, students do not necessarily always stay within the LMS 43 or complete all of the work within the LMS. The interventions and data collection methods utilized in this study relied on participant self-report after completing the learning activity which presents two issues, first it relied on participants remembering to complete interventions and data collections tasks after the primary learning activity (homework, assessment, etc.), something many students did not do, and it relied on participants accurately identifying and remembering their own, perhaps subtle or routine behaviors. Given the many challenges that persisted in designing scaffolds to support SRL in CMCs, this study concludes that CMCs can facilitation and support some aspects of SRL but as of yet cannot fully support SRL for large numbers of online K-12 students. Changes in Self-Regulated Learning Initial group differences. To check that random assignment of participants to experimental groups in fact controlled for participant differences I ran a one-way multivariate analysis of variance (MANOVA), having three levels of between group factors (experimental vs. control I vs. control II) with mean scores on the pre-survey PISA overall and sub-tests as dependent variables. The multivariate between-subjects omnibus for group (i.e. experimental vs. control I vs. control II) was not significant overall (Wilk's λ=.809, F(18,102)=.634, p>.05) nor was it significant for any of the sub-tests. Additionally given the high attrition and differential group sizes based on dependent variable a participant flow chart was created. 44 6 courses selected for participation Consent, assent, and pre-survey completed (N=69) Randomized Experimental condition (N=30) Control I condition (N=24) Control II condition (N=15) Exposed to experimental interventions Exposed to control interventions Exposed to no interventions Completed journals (N=9) Completed journals (N=10) Completed post-survey (N=10) Completed post-survey (N=3) Completed post-survey (N=6) Figure 3. Chart: Participant flow chart. Research question 2. Research question 2 asked, does the presence of self-regulated scaffolding in online K-12 courses lead to gains in adolescent learners' self-efficacy and motivation, and individual and environmental control? Originally I had intended to run a repeated measures MANOVA, however given the high attrition sample sizes were not sufficiently large for such an analysis. It was thus necessary to use non-parametric tests. Further, given that this study contains three conditions, it was necessary to use the Kruskal-Wallis test which allows the comparison of more than two independent groups. Importantly the Kruskal45 Wallis test does not assume normality in the data and as such normality assessments for the dependent variables in research question 2 are unnecessary. The Kruskal-Wallis test does have two important assumptions, first that the dependent variable is ordinal or interval, second, that the independent variable consists of two or more categorical and independent groups. Both assumptions were met and the analysis for research question 2 proceeded with the KruskalWallis test. It is important however to note that the Kruskal-Wallis cannot test for pre/post differences between groups. As a result, in the following analyses the Kruskal-Wallis test will be used for analyze post-survey differences only. While not ideal, the concerns over a post only test are significantly mitigated by random group assignment and the non-significant results of the MANOVA (discussed above) that tested for initial group differences. Self-efficacy and motivation by experimental group. A Kruskal-Wallis test was run to determine if there were differences in self-efficacy and motivation as measured by the PISA Student Characteristics Questionnaire between experimental conditions. Overall self-efficacy and motivation sub-test scores on the post-survey varied slightly between groups for each subtest but the differences were not statistically significant: motivation strategies x2(2)=1.120, p=.571; effort and persistence in learning x2=.504, p=.777; interest in mathematics x2=5.148, p=.076; self-related beliefs x2=.212, p=.900; academic self-concept x2=.931, p=.628. Results indicate that there was no difference in self-efficacy and motivation as measured by the post-survey between experimental groups. 46 Table 6 Descriptive statistics for self-efficacy and motivation by experimental group Motivation Strategies Experimental Control I Control II N M SD N M SD N M SD Effort and Persistence Interest in Mathematics Pre 11 2.55 1.08 3 2.11 1.17 5 2.67 1.43 Pre 11 3.23 .666 3 3.33 .946 5 3.15 .454 Pre 11 2.21 1.04 3 1.89 1.54 5 3.00 1.18 Post 10 3.15 .747 3 2.33 1.20 5 3.07 .983 Post 10 3.15 .747 3 3.08 1.01 5 3.40 .652 Post 10 2.13 1.20 3 1.67 1.56 5 3.33 .408 SelfRelated Beliefs Pre Post 11 10 2.70 2.50 .849 .892 3 2 2.44 2.83 1.07 1.18 5 5 2.80 2.73 .837 .925 Academic SelfConcept Pre Post 11 10 3.06 3.13 .786 .958 3 2 3.11 3.83 .962 .236 5 5 3.13 3.47 1.26 .767 Individual and environmental control by experimental group. A Kruskal-Wallis test was run to determine if there were differences in individual and environmental control as measured by the PISA Student Characteristics Questionnaire between experimental conditions. Overall individual and environmental control sub-test scores on the post-survey varied slightly between groups for each subtest but the differences were not statistically significant: learning strategies x2(2)=1.649, p=.438; memorization strategies x2=.247, p=.884; control strategies x2=.285, p=.879. Results indicate that there was no difference in individual and environmental control as measured by the post-survey between experimental groups. 47 Table 7 Descriptive statistics for individual and environmental control by experimental group Experimental Control I Control II N M SD N M SD N M SD Learning Strategies Pre Post 11 10 2.82 2.90 .822 .699 3 3 2.75 2.42 .901 .629 5 5 2.8 2.55 .671 .622 Memorization Strategies Pre Post 11 10 2.52 2.67 .697 1.14 3 3 2.50 2.44 .928 1.02 5 5 3.23 2.40 .641 .435 Control Strategies Pre Post 11 10 3.11 3.16 .635 .704 3 3 3.20 3.27 .800 .611 5 5 3.28 3.04 .363 .477 Self-efficacy and motivation by initial preparedness. A Kruskal-Wallis test was run to determine if there were differences in self-efficacy and motivation as measured by the PISA Student Characteristics Questionnaire between initial preparedness groupings. Groups followed Roblyer's categorization wherein "yes" to 23-25 items indicates excellent preparedness for online learning; "yes" to 20-22 items indicates some preparedness but needs additional development of required qualities; "yes" to fewer than 20 items indicates minimal preparedness for online learning and suggests possible difficulties. Overall self-efficacy and motivation sub-test scores on the post-survey varied slightly between groups for each subtest but the differences were not statistically significant: motivation strategies x2(2)=2.255, p=.324; effort and persistence in learning x2=5.725, p=.057; interest in mathematics x2=2.933, p=.231; self-related beliefs x2=2.280, p=.320; academic self-concept x2=3.124, p=.210. Results indicate that there was no difference in self-efficacy and motivation as measured by the post-survey between initial preparedness groups. 48 Table 8 Descriptive statistics for self-efficacy and motivation by initial preparedness Motivation Strategies Minimal Prepared ness Some Prepared ness Excellent Prepared ness N M SD N M SD N M SD Effort and Persistence Pre 11 1.94 1.02 3 3.22 .839 4 3.67 .471 Pre 11 2.93 .593 3 3.58 .382 4 3.88 .144 Post 10 2.67 .994 3 2.67 .667 4 3.42 .319 Post 10 2.80 .654 3 3.67 .577 4 3.69 .473 Interest in Mathemati cs Pre Post 11 10 1.88 2.03 1.08 1.19 3 3 2.89 2.44 1.35 1.26 4 4 3.42 3.25 .500 .957 SelfRelated Beliefs Pre Post 11 9 2.30 2.48 .737 .884 3 3 3.33 2.22 .333 1.35 4 4 3.42 3.17 .687 .430 Academic SelfConcept Pre Post 11 9 2.76 2.89 .990 .957 3 3 3.78 3.89 .192 .192 4 4 3.50 3.67 .638 .471 Individual and environmental control by initial preparedness. A Kruskal-Wallis test was run to determine if there were differences in individual and environmental control as measured by the PISA Student Characteristics Questionnaire between initial preparedness groupings. Again, groups followed Roblyer's categorization wherein "yes" to 23-25 items indicates excellent preparedness for online learning; "yes" to 20-22 items indicates some preparedness but needs additional development of required qualities; "yes" to fewer than 20 items indicates minimal preparedness for online learning and suggests possible difficulties. Overall individual and environmental control sub-test scores on the post-survey varied slightly between groups for each subtest but the differences were not statistically significant: learning strategies x2(2)=5.193, p=.075; memorization strategies x2=.393, p=.822; control strategies x2=1.396, p=.498. Results indicate that there was no difference in individual and environmental control as measured by the post-survey between initial preparedness groups. 49 Table 9 Descriptive statistics for individual and environmental control by initial preparedness Minimal Preparedness Some Preparedness Excellent Preparedness N M SD N M SD N M SD Learning Strategies Pre Post 11 10 2.41 2.48 .735 .640 3 3 3.5 2.58 .250 .520 4 4 3.31 3.13 2.79 .323 Memorization Strategies Pre Post 11 10 2.83 2.53 .891 .958 3 3 2.44 2.56 .839 .694 4 4 2.63 2.25 .285 1.03 Control Strategies Pre Post 11 10 2.96 2.96 .662 .587 3 3 3.33 3.27 .115 .611 4 4 3.65 3.30 .100 .622 Domain Knowledge and Retention Research question 3. Research question 3 asked, does the presence of self-regulated learning scaffolding lead to greater gains in domain knowledge and student retention in online K-12 courses? Domain knowledge and retention by group. Before moving forward with the analyses for research question 3 I ran a normality assessment on final course grade for each of the three experimental groups. Table 10 Normality of final course grade by experimental group Statistic df Experimental .850 30 Control I .908 24 Control II .799 15 *indicates significance at the .05 level Sig. .001** .032* .004* **indicates significance at the .001 level 50 The results of the normality analyses indicate that the final course grades are not normally distributed for any of the three experimental groups. Upon further analyses of the results there were no significant outliers rather each group presented a unique distribution. The final course grades for the experimental condition were unimodal and negatively skewed; the final course grades for the control I condition were relatively flat with no obvious mode; and the results for the control II condition were bimodal with clusters at the low and high end of the distribution. The non-normal distribution was a minimal concern with regards to moving forward with the statistical analyses for research question 3 as research has demonstrated that false positive rates are not affected by a violation of normality with large sample sizes (Glass et al. 1972). Moving forward with research question 3 and in order to understand if the three groups differed in terms of domain knowledge as measured by final course grade score and retention measured by date of last course access, I ran a one-way between subjects analysis of variance (ANOVA), having two within-subject factors (final course grade and date of last course access) and one between-subject factor (experimental vs. control I vs. control II). The analysis of variance was not significant for course completion score (F(2,65)=1.749, p>.05) or retention (F(2,66)=.995, p>.05). Neither course completion or retention differed significantly by initial preparedness (F(2,60)=.208, p>.05; F(2,59)=.497, p>.05). Given the results of the statistical analysis the null hypothesis could not be rejected. 51 Table 11 Descriptive statistics for final grade and date of last access by experimental group Final Grade Date of Last Access Experimental Condition Experimental Control I Control II Experimental Control I Control II N M SD 30 24 15 30 23 15 60.65 47.66 58.07 14.23 12.74 16.27 31.724 34.074 40.266 5.513 7.105 2.815 Table 12 Descriptive statistics for final grade and date of last access by initial preparedness Final Grade Date of Last Access Initial Preparedness Minimal preparedness Some preparedness Excellent preparedness Minimal preparedness Some preparedness Excellent preparedness N 33 14 15 34 14 15 M 58.27 59.04 51.95 14.76 13.14 14.93 SD 32.806 38.952 33.847 4.542 7.294 5.725 Domain knowledge by intervention completion for the experimental condition. In order to understand if the SRL interventions had any relationship to domain knowledge as measured by final course grade score I ran a one-way between subjects analysis of variance (ANOVA), having one within-subject factors (final course grade) and one between-subject factor (No Interventions vs. Minimal Interventions vs. Some Interventions vs. Several Interventions). This analysis although not explicitly part of the original research questions was done given the low levels of participation in the research interventions. As is seen in table 3 more than half of the participants in the experimental condition did not complete any of the interventions and as such could serve as their own control group. The analysis of variance was significant for final course grade (F(3,26)=3.258, p<.05). 52 Table 13 Descriptive statistics for final grade by intervention completion in the experimental condition Intervention Completion N M SD No Interventions Minimal Interventions 18 7 47.40 79.80 33.05 18.83 Some Interventions Several Interventions 3 2 84.34 77.43 5.204 21.44 Research question 4. Research question 4 asked, how does self-regulated learning scaffolding influence adolescent learners' ability to successfully to regulate their learning to produce improved achievement outcomes in computer-mediated contexts? According the results of research questions 2 and 3 self-regulated learning scaffolding in this study did not influence adolescent learners' ability to regulate their own learning as measured on the PISA Student Characteristics Questionnaire. That is not to say that self-regulated learning scaffolding does not in any condition influence learners' ability to regulate their learning in contexts mediated by technology, only that the scaffolds utilized in this intervention did not, in any significant way, influence participants SRL. Journal data provided by participants in the experimental and control I groups were also used to answer research question 4. What few journal responses there were did demonstrate that the journals could support and facilitate planning and self-reflection as participants could, and did set learning goals, plan for upcoming units and reflect on their original plans and learning goals. While many of the journals were not sequential (meaning participants did either the planning or reflecting but not always both for a particular unit), the three participants below are typical examples of utilizing the planning and reflecting journals to scaffold self-regulated learning. The names used below are all pseudonyms and each participants journals are from the 53 same course unit. Abby. Planning journal: I wish to get better test and score grades in this unit. I plan on asking questions and getting reviews from other math teachers in my school. I look over notes. I'm 50% sure I can if I work hard and put my mind too it. 40 because math is really hard for me so it is hard for me to retain all the information. Reflection journal: I  somewhat  met  my  goals   this  unit,  I  spent  more  time  getting  help  and  I  studied  more  than  I  did  my  last  test. I  got  14/18   on  my  math  test  and  that's  the  best  I've  gotten  on  my  tests  so  far. I  feel  100. I  studied  a  lot   more  than  I  usually  do. Put  more  effort  into  my  work  and  notes.   Carrie.  Planning  journal:  My goal is to work as hard as I can to get through this pretty good. For me to achieve this I will study hard, keep all distractions away, and keep my mind set on it. To decide on which strategy to use it depends on what I am working on at the time. on a scale from 1-100 i at least would want to get an 80 or higher. For me to study its about an 65 because its sometimes hard for to study with a lot of distraction that are around me.  Reflection   journal:  For this unit I say yes to reaching at my goal at getting done what I had to get done. I seen that harder I worked at it the easier it was for me.   James. Planning journal: This unit I am trying to complete everything by the 15th and get an A on the unit test. I plan on taking notes when reading the eTextbook. I don't know how I decide which strategy I use. I am a 75 on how sure I am I can get the grade I want. I am an 80 on interest in studying. Reflection journal: I did very well on this unit! I am 100% percent satisfied with how I did. The reason I did so well is because I took notes well during the tutorial videos, and made sure I understood everything. As is clear from the journals they did support self-regulated learning, however completion of the journals (regular or sporadic) had no effect on participants self-regulated 54 learning. When participants were grouped according to their involvement in research interventions (frequent, sporadic, none) there was no significant difference between the three groups on any of the PISA self-regulated learning sub-tests, final course grade or retention. While this analyses was outside of the original scope of research it was conducted as a result of the low overall participation and to determine if completing the scaffolds--any scaffolds, had an effect. Given this the conclusions regarding research question 4 stand, that self-regulated learning scaffolds did not influence learners ability to regulate their learning in contexts mediated by technology. 55 CHAPTER 6 Discussion As this study was intended and designed as a proof of concept, outcomes and implications of this study encompass more than the results of statistical analyses. Therefore this section will begin by discussing each of the explicit design goals detailed in Chapter 1, focusing on how the design goal was carried out in this research study and the intended and unintended consequences of such a consideration concluding with a discussion on the implications of the analyses presented in Chapter 5. While the results were largely absent of any statistical significance and subsequently significant experimental group differences, there are significant implications born out of the insignificance. As much of the statistical insignificance is likely attributed to research design implications a combined discussion of both serves to better inform each and provide a more robust discussion. Design Implications Interventions based on previous research. As stated previous there are decades of respected and informative research on self-regulated learning (SRL) with all age groups and in several educational contexts. This research served as a valuable resource and was not ignored during the design of this study. Many of the established foundational strategies for effective SRL were incorporated including (but not limited to): • Introducing students to general and proximal learning goals through the SRL videos and having students set clear, attainable and proximal learning goals (Bandura, 1986; Zimmerman, 2002) in the planning journals. • Introducing and encouraging the use of effective performance control strategies for 56 learning in CMCs (Azevedo & Cromley, 2004; Azevedo, Cromley, & Seibert, 2004) such as self-questioning, taking notes, summarization and deliberate environmental control through the SRL videos. Additionally, effective processes for facilitating and supporting SRL in CMCs were also incorporated into the study including (but not limited to): • Externally facilitated SRL learning tutoring script which prompted students to activate prior knowledge, plan their time, and monitor their learning goals (Azevedo et al., 2008). • Specific SRL phase targeting in the Self-Regulation Empowerment Program (Cleary & Zimmerman, 2004). • Exposure to SRL through an initial orientation and introduction which produces short-term changes in SRL and is intended to set the stage for long-term SRL strategy use (Azevedo and colleagues). Collectively these strategies, processes and design interventions represent careful consideration of previous research on SRL and attention to what experimental conditions have demonstrated gains in SRL. Given the exploratory nature of this study none of these elements were identified as individual experimental variables, rather they contributed to the larger experimental interventions and data collection processes. As such there were no explicitly obvious unintended consequences associated with these design considerations nor where they particularly difficult to implement in a CMC. Instead they served as a solid empirical foundation upon which to build this study. Collectively these interventions which had previously demonstrated effective SRL strategy use or gains in SRL (depending on the type of study) did not lead to significantly different SRL knowledge (on any subtest) or lead to significant gains in final course grade or retention in this study. Given the effective history of these specific 57 interventions it is likely that the experimental context in which they were embedded and how they were actually carried out in the study contributed to their overall ineffectiveness. This study did not demonstrate any adverse effects as a result of exposure to the interventions nor was it designed to determine the effectiveness of individual interventions. Design and interventions should remain pedagogically sound. Another important consideration that drove the design of this study was to keep all interventions pedagogically sound, that is to restrict the design of the study to what could realistically be done in a typical K12 online class. Easily accessible interventions. This study was strongly driven by the belief that educational research should be directly applicable to both researchers and educators and it was with this in mind that as much as possible interventions were designed and implemented without additional costly software or high-level software programming. Interventions were designed to be accessible to large numbers of teachers and students without significant technological knowledge and overly significant initial time investments. The implication of this decision was that the interventions were designed in a context not necessarily originally designed to support said activities. That is the interventions had to be retro-fitted into the existing LMS and further into how the Michigan Virtual School was currently utilizing the LMS. While the Blackboard LMS is designed to do many things it is not without limitations and many features that perhaps are better suited to SRL scaffolding are not standard or not commonly utilized (such as advanced user tracking and system data). The obvious result of this was the imperfect performance control measure and interventions. Scaffolding this aspect of SRL was extremely difficult within the context of the CMC and collecting real-time data was not an option. This certainly affected development of SRL (and subsequently measuring that development) but also likely affected 58 response rates. This is likely due to the delay of the performance control measure as well as the quality of what little data was collected as it was based primarily on participant recall. Long study duration. With the goal of pedagogical soundness in mind this study was designed to take place over the course of a full academic term. While tightly controlled experimental studies have provided invaluable information regarding effective SRL strategy, facilitation, and support, these types of experiments represent atypical educational settings. Learning is not typically defined to highly structured and intensive 1-hour training sessions followed immediately by an assessment, rather learning is a process that varies in intensity and duration of both instruction and assessment. This study aimed to embed itself in typical K-12 education practice with interventions and assessment paced appropriately with course progression. Additionally, the interventions were designed as to not place undue requirements on participants’ time or attention that is to not interfere negatively with participants learning of the course content. Each intervention was deliberately placed within each unit, available to participants at the appropriate and intended time through adaptive release. It is likely that the low-demand and long duration of the intervention (18 weeks) in conjunction with this course being one of a likely 6 for each study resulted in a low impact of the interventions on SRL knowledge. Designed to maximize group differences. The third major design consideration was in keeping with the proof-of-concept theme and it was to design all interventions to maximize group differences. For this reason (among others) the second control group, control II, was included. This group was given only the pre and post assessments and no subsequent interventions. While control I was intended to isolate the SRL interventions as the unique variable, control II was intended to maximize the effects of the SRL interventions on the 59 experimental group. As there were no significant differences on SRL, final course grade, and retention between any of the three groups, it seems that not only were the SRL intervention ineffective compared to the general interventions, but that neither interventions were significant compared to no intervention. Implications of Analyses and Results While the statistical analyses may not have been very significant overall, three noteworthy themes emerged out of the insignificant. The three themes are presented below with particular emphasis on the implications for this study and the self-regulated and K-12 online learning research communities. CMCs can support self-regulated learning but effective support requires significant human intervention. While it may not seem evident given the quantitative results of the study, qualitative data and researcher observations suggest that CMC's can in fact support at least parts of self-regulated learning but also that as of yet, widely utilized (in K-12 online education) and low-cost options do not possess the capability to fully support all areas of self-regulated learning (SRL) without significant human intervention. Certainly CMC's can work in conjunction with skilled online teachers to facilitate and support SRL, however this still requires significant investment by the teacher both in terms of upfront design and development as well as ongoing monitoring and support. It is not the "humanity" of teachers that provides them the ability to serve as effective external regulators rather it is certain features such as dynamic student monitoring and feedback (explain in more detail in the next section) that can be learned and applied to different situations that CMCs have yet to fully realize. This monitoring and support is undoubtedly something that many teachers already do implicitly or could begin to do however, 60 the goal of this study was to determine if CMC's could bear the brunt of support and allow teachers to focus as content experts for larger numbers of online K-12 students. The CMC utilized in this study provided ample support for the planning and reflection phases of Zimmerman's (1989) theory of self-regulated learning in the form of the private student-teacher journals where teachers were able to set a series of journal prompts for all students (or a subset through membership groups), receive private student responses and provide private student feedback. However as stated previously, this still required a significant effort on the part of the teacher. Unsurprisingly, there has yet to be a CMC that can fully replicate highly effective and responsive teaching on a large scale. The CMC was significantly less able to facilitate and support the performance control phase of self-regulated learning through systematic monitoring of performance intended to maximize learning. There are of course CMCs and a large and ever growing market for productivity software that are designed to do just this (such as RescueTime), monitor ones activity over a given time, however additional add-ons like this present two issues. First, while they can accurately measure activity in a CMC easily, they are less able to provide direct and effective feedback independently, and second combing several pieces of software begins to resemble a piece-meal solution in which users (students or teachers in this case) would have to remember, and have the initiative and drive to utilize many nonembedded resources. The answer to this question then becomes quite muddled and complex, yes CMCs can help facilitate and support self-regulated learning but evidence suggests that K-12 online students, those just beginning to develop self-regulated learning knowledge and skills, need more responsive support beyond what an accessible, integrated CMC can currently offer. There are a number of digital technologies that can help students plan and organize their learning, many 61 provide integrated calendars, mobile notifications, grade tracking, and scheduling information but, like performance control software discussed above, these also possess significant drawbacks. First, most of the technologies rely on students to input the necessary information, significant barrier to full use and second, these technologies only help plan, they do not provide sufficient responsive support or feedback should the demonstrate ineffective strategy use. Embedded, hard, indirect, and static interventions are ineffective at supporting selfregulated learning. Given the limitations of the CMC utilized in this study a learning management system (LMS), experimental interventions and data collection procedures were embedded (within the LMS), hard (fixed, non-customizable), indirect (guiding learning but not providing direct instruction), and static (fixed procedures). While this was largely determined by the LMS these classifications were not without unintended consequences. Azevedo et al. (2008) demonstrated success with the use of human tutors to scaffold and facilitate SRL, however in that particular study it may have been less about the human tutor specifically and potentially more about having access to an adaptive and dynamic scaffold, particularly when reviewed in conjunction with the findings of Azevedo et al. (2005). The tutors in Azevedo's (2008) study followed a strict script, however written into that script were a series of potential responses and branching options which in many ways resembled basic coding schemes. Azevedo’s findings themselves are relatively unsurprising, that students benefit from adaptive and responsive SRL scaffolding, but this combined with the results of this study suggest some important implications for SRL support in online K-12 courses. First, all scaffolds are not equally effective at facilitating and supporting SRL. The scaffolds utilized in this study (embedded, hard, static and indirect) were designed as such given the limitations of the LMS and to maintain truly equal experimental conditions. While there is not gold standard scaffold type, 62 Devolder et al. (2012) states that all scaffolds differ in effectiveness based on method of delivery and based on this assertion; it is likely that ideal scaffold types are highly contextual and based on the needs of the student. That being said however, there are more ideal scaffolds for the particular population targeted in this study. The types of scaffolds utilized in this study will be discussed below, focusing on the decision to utilize that type of scaffold, implications associated with use, and if applicable more advantageous options. Embedded scaffolds. Embedded scaffolds are those that are integrated into the learning environment and available to students alongside course content, requiring little additional initiative on the part of the student to seek out the resource. The CMC that was utilized in this study, the Blackboard LMS, did allow for scaffolds to be embedded alongside course content within the unit modules, which given the age of the population (13-18 years of age) is advantageous given that this age group, or at least part of it, are still developing SRL knowledge and skills. Given previous research on scaffold types and effectiveness it is unlikely that having the scaffold embedded in the LMS contributed negatively to SRL outcomes. Hard scaffolds. Hard scaffolds are static or fixed whereas soft scaffolds are customizable and negotiable. While exposure to scaffolds was customizable in the LMS though membership groups and adaptive release (based on a number of criteria), the scaffolds themselves were fixed. This was both a limitation of the LMS, the research design, and a desire to maintain consistency of intervention across experimental conditions. Devolder et al. suggest that hard scaffolds are typically those presented through CMC's as they depend on a pre-determined set of criteria. Fixed scaffolds are not necessarily ineffective, as evidenced by Azevedo et al. (2008) wherein human tutors utilized a fixed scaffolding script but it may be that fixed combined with static is an ineffective combination, at least for the population of learners utilized in this particular study. 63 Static scaffolds. Static scaffolds offer fixed guidelines and procedures as opposed to dynamic scaffolds that are interactive and provide learner feedback. The scaffolds utilized in this study were static. Planning and reflecting journals were static in that participants did not receive feedback regarding their journal entries, nor were journals adaptive based on prior entries, this was done deliberately to maintain absolute consistency between experimental groups. While providing feedback to the experimental condition could have followed guidelines set by Cleary and Zimmerman (2004), feedback for the control I group would have been difficult to keep strictly general (not related to SRL) yet responsive, particularly when several students noted difficulties with the course material, pacing, and test preparation. In a strictly educational setting this would not have been an issue, however in an experimental setting maintaining consistency between groups was paramount. As mentioned previously, Azeveo et al. (2008) demonstrated gains in SRL in groups with a human tutor acting as an external regulation agent. It is likely that these gains did not necessarily result from the humans specifically, but from the dynamic script the tutors were able to utilize. This suggests it wasn't any particular feature of "humanness" that impacted learning, rather it was the tutors ability to provide dynamic feedback and appropriately guide learning. While the script was static in that it did not change during the intervention, it was dynamic in that it was responsive to where participants were at any moment and provided a number of branching opportunities in response to participant input. Additionally, end of unit surveys used in this study were static as Blackboard does not allow branching on its internal survey tools. The decision was made to keep the surveys embedded and static over nonembedded and dynamic to encourage participation through lower barrier of access. Given the results of this study, that the ineffectiveness of the scaffolds utilized in this study evidenced by the absence of any significant differences between the experimental and control 64 groups on a number of measures, previous work by Cleary and Zimmerman, and work by Azevedo and colleagues, it is reasonable to suggest that perhaps the types of scaffolds were ineffective for this population in this context. Given the well-established (Schunk, 1996; Wood, Bandura & Bailey, 1990) academic benefits of self-regulated learning training and scaffolding in face-to-face courses perhaps the reason these interventions were ineffective was not that SRL does not impact learning, but rather that the scaffolds used in this study were not the right type of scaffolds. Further research is needed to investigate and determine effective scaffolds for online K-12 learners, as well as how these scaffolds can fit within, and be supported and facilitated by CMCs and LMSs. There is a relationship between completing interventions and success in online courses. Analysis within the experimental condition of participants who completed varying levels of interventions demonstrated that there is a relationship between completing interventions and final course grade, however the specific nature of this relationship remains unknown. The analysis was not able to determine causality so it is unknown whether completing interventions positively influences final course grade, or if participants who completed interventions were already more motivated and did better in their course. Whatever the nature of the relationship students who completed any interventions demonstrated significantly higher final course grades than students who did not. Again, the relationship is not clear at this point in time but the finding opens up more questions about how to motivate students to participate in online courses and how this participation interacts with course success. Participants that completed the journals did seem to be typical of the overall sample of the study. As a group participants who completed at least one journal had a mean course completion 65 percentage of 69.36% which marginally higher than the overall course completion mean of 66.5%. Journal responses were evenly split (N=9 for each group) between those participants earning 80% or more of the total course points and students earning 79.9% or less of the final course points and journal entries were analyzed for total and average word count. The experimental condition was over represented in the over 80% group (N=6) and the control I group was over represented in the less than 80% group (N=6). The over 80% group had approximately 2,200 total words, with an average of 244 words per participant. The less than 80% group had approximately 1,200 total words, with an average of 133 words per participant. Following this analysis journals only from participants in the experimental condition were qualitatively analyzed to determine if there were differences in quality of response by grouping (over 80%; less than 80%). There were not any overt differences in quality by group overall, the three responses from participants in the less than 80% group were similar to the six from the over 80% group in respect to planning, reflection, and goal setting. Successfully retaining and completing online courses depends on more than initial preparedness for online leaning. Perhaps that most surprising insignificant finding was that preparedness for online learning had no effect on final course grade or retention in the online course. This finding was surprising for two reasons, first this was the largest possible sample size in the study (N=69) as it did not rely on completion of the post survey (only the pre), in fact it did not even rely on successful completion of the course, variance in final course grade was a welcome and an integral part of the analyses. This result was surprising given the common thinking and work by Roblyer and colleagues which suggests that students who are more prepared will persist longer in online courses, which as discussed in Chapter 1 present unique contextual learning challenges, and develop greater domain knowledge. This however was not 66 the case in this study as there were no significant differences between preparedness groups on retention and final course grade. Regardless of the results, preparedness for online learning is undoubtedly important, if students struggle to operate the technology there is already a barrier to learning before even reaching the course material. However, like with many other educational variables, preparedness may be one of many acting on students at any given time. Within the lack of significant results there are three conclusions. First, it seems that retention specifically in online courses depends on much more than simply preparedness for online learning. Students may leave an online course for a variety of reasons beyond not being prepared for the course including: changes in students’ life situation, technology issues at home or school, and/or wrong course for the particular student. Just as there are a number of reasons a student may leave a course, there are a number of factors that influence why a student stays in a particular course and how well that student does overall. Second, it may be that initial preparedness for online learning may help in less obvious ways but it has a weak relationship with learning outcomes. It may be more likely that readiness for online learning works with (or against) readiness for specific course content and the difficulty of the course content overall. The courses utilized in this study, mathematics courses, rely heavily on students possessing a solid foundation of mathematical knowledge. It is impossible to know from the data collected in this study but may be that while some students were ready for online learning, they may not have been ready for academic content of the course. A domain specific pre-assessment may have provided additional insight into this possibility, this will be further discussed in the limitations section of this chapter. Third, upon further analyses of the online learning readiness scores it seemed that scores went down for all three groups. This was merely an observation, not based on statistical 67 significance and should be tempered by noting that less than half of participants completed the post-assessment. However, the results seem to suggest that perhaps participants initially overestimated their readiness for online learning and only after the course was completed understood the difficulties associated with learning online. This observation may also interact with students past experience with online courses and warrants future consideration and study. 68 CHAPTER 7 Limitations There were four significant limitations to this study that came about either as a result of the design of the study or emerged unforeseen during the data collection. Each limitation will be discussed in detail below. The sections will start by describing each limitation, identifying possible causes, and concluding with implications for this study and if applicable possible ways to address these limitations in future research. Attrition. Perhaps the most obvious and significant limitation arouse unforeseen during data collection. While high attrition is always an unfortunate and distinct possibility during any study and can be to some degree expected, this study suffered from extremely high attrition rates for all three experimental groups. The attrition for this study is really two-fold, high fluidity and attrition in the courses overall, and high attrition from pre to post assessment. Both types of attrition will be discussed in detail below. Historically MVS experiences high fluidity during the first few weeks of a course. There is a significant "settling" period at the start of a course wherein large numbers of students are enrolling or dropping the course. Typically this evens out around the third or fourth week and this study attempted to control for this high fluidity by placing interventions not at the start of the course, but rather in the first or second unit. This may have helped control for some of the fluidity in the course however the courses overall, and subsequently the sample in this study, still experienced high attrition, total attrition for each individual course is provided in table 3. Overall of the 234 students who enrolled in the courses 35 officially or unofficially (earned less than 10% of total course points) dropped the course, resulting in an overall attrition rate of 0.15. Further most courses had attrition rates over 10%, with the average rate of attrition at 0.13. For 69 study participants specifically, of the total 69 participants 13 would be considered to have "unofficially" dropped the course having received a final course grade of less than 10%. Four more students received higher than 10% of the total course points however did not log into the course past the second month (October for semester 1, March for semester 2), combined both groups represent an attrition rate of .24, or approximately the loss of 25% of the study participants. Given this it seems that this initial fluidity and typical add/drop cycle accounts for some of the overall attrition but not all. Attrition was also extremely high for all three experimental groups from the pre to post test. Only 22 participants completed at least one research intervention (journal or survey) and only 19 of the original 69 participants completed the post assessment despite frequent reminders to do so. Some attrition was expected however it was not expected to have an attrition rate over .5. To account for the expected attrition the maximum number of classes available while keeping the teacher constant were included accounting for an initial possible population of 234 students. This study did reasonably well recruiting from that initial possible population with approximately 30% (N=69) students consenting to participate. Unfortunately this initial number was not enough given the attrition rates for each course and experimental group (for specific number see table 3) and final participation numbers did not large enough samples for robust statistical analyses. While there is no definitive cause for the attrition it is likely that the reminders placed in the course announcements, without a physical presence, did not provide enough social pressure to compel students to participate. Solutions to address this limitation are difficult in that one obvious solution, making this sort of intervention required as part of the course, cannot ethically be mandated, as there is no evidence that any of these interventions benefit students. Perhaps the interventions themselves were not interesting enough for students to compel participation, this 70 leads to the second limitation. Participant feedback. As discussed at length previously in this chapter, the decision was made not to provide feedback to participants on any of the interventions, however the only intervention that provided for feedback was the journals. This was done to maintain absolute consistency between experimental groups. As stated previously, providing feedback to the experimental group could have been modeled off of prompts from Azevedo and colleagues and Cleary and Zimmerman (2004), however no such model existed for providing feedback to the control group. Maintaining feedback that was responsive (as is the ultimate goal with feedback, and as it would have been for the experimental group) yet general would have been extremely difficult. This decision however likely contributed to the ineffectiveness of the interventions. By not providing feedback the journals were fixed (not dynamic), however both Azevedo et al. (2005) and Azevedo et al. (2008) demonstrated effective SRL scaffolding through the use of dynamic interventions (among other experimental variables and conditions). It is not known with complete certainty that dynamic journals would have contributed to differences between groups however based on previous research it remains a plausible possibility. Future research should investigate interactions between scaffold types and SRL phases to determine the best scaffolds for each phase of learning. Low overall response rate. Closely related to, yet a distinct limitation, was the low overall response rate on the interventions including the planning and reflection journals and end of unit surveys. Despite regular reminders to complete all research activities and the embedded position alongside required course content, very few students regularly completed the interventions. The journals had the highest rate of completion with 19 participants completing at least one journal. The end of unit surveys had very low participation with only 6 participants completing a total of 71 10 end of unit surveys, with a total of 22 (of 69; 32%) participants completing at least one research intervention (journal or survey). The qualitative data from the journals was used to answer research question 4 as there was a relatively even split between the experimental group and control I regarding completed journals. Given the low response rate of the end of unit surveys they ultimately were not utilized. Additionally, the final exam planning journal intended to allow for comparison between the experimental and control groups on planning skills and knowledge was not utilized given the low response rate, only three completed final exam planning journals. Again, it seems reasonable that the low response rate can be attributed to two primary factors. First, as discussed previously, there was not adequate social pressure on the participants to complete the research interventions. Second, it may be that the interventions were not engaging enough or seen as useful and therefore largely ignored by a majority of participants. No domain knowledge pre-test. While a domain knowledge pre-test was not necessary to measure learning outcomes (one of the aims of this study), such an assessment could have provided insight into gains in domain knowledge by group over the course of the semester. Additionally, a domain knowledge pre-test could have been used to identify if the experimental groups, although randomly assigned, had equal domain knowledge at the start of the course. This assessment could also have been used, and should be used in future similar studies alongside initial preparedness for online learning to see if students are truly ready for their respective course. It may be, as discussed in chapter 6, that initial preparedness is only a small part of retention and successful completion in online courses and that in particularly those courses requiring a solid foundation of domain specific knowledge initial preparedness can only do so much towards success. It is reasonable to suggest that if students are not ready for the course content they may struggle more than students who are ready, this potentially compounded with 72 initial readiness for online learning could mean the difference between students that succeed and those that struggle. Pragmatically speaking a domain knowledge pretest would also provide online teachers with additional information about students and allow them to more closely monitor those students that lacked foundational knowledge. This information is invaluable especially since with supplemental online providers like the Michigan Virtual School the teachers most likely have no prior experience with their students, formal or informal. In traditional school settings teachers are able to identify struggling students and collaborate with other (or upcoming) teachers to provide the best learning environment for that student. This is not possible to the same degree with online courses, particularly those from supplemental providers and a domain knowledge pre-test could help supplement the information that online teachers do receive. Conclusions and Implications Regardless of the specific results of this study, it is hard to argue against the assertion that online learning, for learners of any age, presents new and unique challenges apart from those present in face-to-face classrooms. Contextually the two environments are very different and offer different affordances and challenges. This is highlighted by work previously discussed by Azevedo and colleagues which suggests that young learners or those new to online learning are less able to take full advantage of the affordances of online learning (Azevedo, 2005). This study was guided by the belief (based on work by Devolder, et al., 2012) that online learning can be facilitated and challenges mitigated by well-designed SRL scaffolds embedded with CMCs. While this belief may not be supported by the results of this study, there remains a recognizable need to maximize the affordances of online learning and to minimize the constraints. 73 This study attempted to do just this through the use of SRL scaffolds that served as the experimental interventions. It is unknown specifically why the interventions were ineffective at supporting SRL but certainly one contributing factor was the low overall completion rate of the interventions. It is hypothesized that the interventions may have been seen as extra work that students were not motivated or sufficiently compelled to complete. For any type of scaffold or support to be effective students must actually engage with said support and to do so students must be motivated either internally or have external pressure or perceived value associated with the support. With the goal of developing intrinsically motivated life long learners, those that seek learning beyond obvious external motivators and rewards, we must design effective interventions that students willingly complete with a good-faith effort, figuring out how to do this within existing LMS for large numbers of students will be a challenge but one worth pursuing. An important conclusion that has been mentioned previously but warrants final discussion is that the results of the study indicate that preparing students for online learning requires more than providing access and basic technological skills. There was no relationship between the preparedness measure and final course grade suggesting that students with superior technological skill or online learning preparedness overall fared no better than the least skilled or prepared students. Learning in any context is incredibly complex, even more so with the significant contextual demands of online learning. As online K-12 education becomes more widely utilized across the country as a supplement to face-to-face instruction for any student (not just homebound or credit-recovery students) the educational community needs to better understand the processes and demands of online learning to ultimately better serve our students. This study extended beyond research by Azevedo and colleagues, taking research design elements that were successful in short duration, individual learning task studies and 74 incorporating them into a long duration study. The results in this long-duration study with multiple goal pursuits (both proximal and general within each module and within the course overall) did not demonstrate the same significance as the short duration studies. While the work by Azevedo and colleagues has done much to inform self-regulated learning in certain contexts, expanding and applying that to authentic learning contexts proves challenging. Many of the strategies that are successful over short periods are likely to lose effectiveness when expanded over longer periods of time and must compete for attention within not only an 18-week course, but one of many 18 weeks courses a student is taking simultaneously. It is likely that there are different types of strategies and supports that students must utilize on micro and macro level goal pursuits that operate concurrently to support students development of SRL and facilitate successful completion of online courses. Future Research While the results of this study were largely not significant these results present several opportunities to continue this line of research. First and foremost this research needs to be simplified and replicated with much larger sample sizes. While several compromises were made in an effort to maintain absolute experimental consistency, this may have ultimately diluted the quality of results. The follow-up should utilize a condensed pre/post-survey and shorter, intrusive, and real-time interventions in an effort to determine interventions that are effective at supporting and facilitating SRL in online courses. Additionally, this research should be extended to other domains, this study deliberately chose mathematics because the relatively low (compared to other domains) average final course grade allowed variation within the sample and room for statistical growth. It was also deliberatly 75 limited to mathematics to control for teacher and course effects. That being said, research on self-regulated learning in online K-12 needs to be expanded to all domains, as all domains, particularly world languages, are seeing unprecedented growth in enrollment across the country. Finally, further research needs to be conducted to identify empirical markers of K-12 online learning readiness. While several informal "readiness assessments" exist from national online course providers, few are built from actual research identifying makers or readiness. Markers may include learner characteristics and attributes at the start or end of a course, learning trajectories during a course, or levels of technological access. Knowing these markers will allow online K-12 course providers and teachers to better serve their students and intervene earlier. 76 APPENDICES 77 APPENDIX A Assessments Pre/Post-Survey PISA Student Characteristics Questionnaire Learning Strategies The following items are rated as either: almost never, sometimes, often, or almost always. Elaboration Strategies When I study, I try to relate new material to things I have learned in other subjects. When I study, I figure out how the information might be useful in the real world. When I study, I try to understand the material better by relating it to things I already know. When I study, I figure out how the material fits in with what I have learned. Memorization Strategies When I study, I try to memorize everything that might be covered. When I study, I memorize all new material so that I can recite it. When I study, I practice by saying the material to myself over and over. Control Strategies When I study, I start by figuring out what exactly I need to learn. When I study, I force myself to check to see if I remember what I have learned. When I study, I try to figure out, as I read, which concepts I still haven't really understood. When I study, I make sure that I remember the most important things. When I study, and I don't understand something, I look for additional information to clarify the 78 point. Motivation Strategies The following items are rated as either: almost never, sometimes, often, or almost always. Instrumental Motivation I study to increase my job opportunities. I study to ensure that my family will be financially secure. I study to get a good job. Effort and Persistence in Learning When studying, I work as hard as possible. When studying, I keep working even if the material is difficult. When studying, I try to do my best to acquire the knowledge and skills taught. When studying, I put forth my best effort. The following items are rated as either: disagree, disagree somewhat, agree somewhat, agree. Interest in Mathematics When I do mathematics, I sometimes get totally absorbed. Mathematics is important to me personally. Because doing mathematics is fun, I wouldn't want to give it up. 79 Self-Related Beliefs The following items are rated as either: almost never, sometimes, often, or almost always. Self-efficacy I'm certain I can understand the most difficult material presented in readings. I'm confident I can do an excellent job on assignments and tests. I'm certain I can master the skills being taught. The following items are rated as either: disagree, disagree somewhat, agree somewhat, agree. Academic Self-Concept I learn things quickly in most school subjects. I do well in tests in most school subjects. I'm good at most school subjects. Post-Survey Only Questions I feel better prepared for future courses and learning in general. I have new strategies that I can use in future courses in all subject areas. I have new strategies that I can use in future mathematics courses. From now on, I will do well most school subjects. 80 Self-Regulated Learning Questionnaire In what classes do you excel? In what classes do you struggle? On which type of assignments do you perform well? On which type of assignments do you not perform well? What strategies do you use to prepare for and complete the assignments that you perform well on? What about on the ones you do not perform as well? Do you typically set learning goals when completing your homework and course assignments? When taking an exam in a course that performed well in, how sure were you (before getting your grade) that you would perform well in that course? What made you sure or unsure? How did you prepare for this exam? When taking an exam in a course that you did not perform well in, how sure were you that you would not perform well on that exam? What made you sure or unsure? How did you prepare for this exam? Is this course required? How interested are you in this course? Do you anticipate having to motivate yourself to complete the required coursework? How do you motivate yourself when you don't feel like studying? Do you keep track of where and when you study, and for how long? How do you determine if you performed well on a course assignment? Do you typically know why you performed well or not well on a course assignment? How do you know? 81 What do you do to improve your performance on course assignments that you do not perform well on? 82 Online Learning Self-Assessment All questions are Yes or No. Scoring is as follows: "yes" to 23-25 items indicates excellent preparedness for online learning; "yes" to 20-22 items indicates some preparedness but needs additional development of required qualities; "yes" to fewer than 20 items indicates minimal preparedness for online learning and suggests possible difficulties. Technology Skills and Access I have an email account and frequently communicate through emails. I know how to use an Internet browser to navigate web sites. I have fairly good typing skills. I know how to save a document to my desktop or to a folder on my hard drive. I have some access to a high-speed Internet connection. Risk Taking I am not afraid of making mistakes if I am learning new things. It does not bother me if I answer incorrectly in a new learning situation. I don't mind asking someone for help if I don't know how to do something. I don't mind displaying my work in front of others in a learning situation. I am not easily frustrated when I am learning new things. Organization and Self-Disciplined Learning I tend to start new tasks early to get a head start on them When I have a lot of things to do, I schedule them to make sure I get them done. 83 I almost never let things go until the last minute. I tend to plan my daily activities to allow enough time to accomplish them. I know I can make a weekly commitment of time to complete my coursework. Responsibility for Learning I believe it is rewarding to be a high achiever. My successes have always been because of my efforts; luck rarely played a role. I try to achieve in all my classes, regardless of their level of difficulty. I usually reach the goals I set for myself. I tend to persist at things I start, even when it takes more time than I thought. Communication and Study Skills I have good reading comprehension. I'm comfortable with expressing myself in writing. I am fairly good at following written directions. I can focus my attention on learning even when there are distractions around. I like working on tasks independently; I don't need face-to-face contact. 84 APPENDIX B Consent/Assent Consent We are inviting your child to participate in a research study designed to help us understand how to support students in online courses. If you agree to allow your child to participate in this study we will ask your child to complete two questionnaires designed to better understand your learning process. If you agree to participate your name will be entered into a raffle for the chance to win one of two new Apple iPads. There are no correct or incorrect answers, and your child's responses here will in no way impact their standing in their online course. The researchers are interested only in their general learning strategies. Their participation is voluntary and they may decline to answer the questionnaire or may skip any items that they feel uncomfortable answering. Your child's unit test scores, final course grade, and date of last course activity will also be collected. All data, including grades and responses are confidential and their privacy will be protected to the maximum extent allowable by law. They will be given a unique identifier and following the completion of the online course, all documents will contain only this unique identifier. There are no direct benefits to participating in this study, although we hope that they will gain more insight into their learning through participation. If you have any concerns of questions about this research study, such as scientific issues or how to do any part of it, or to report an injury, please contact the following investigators: 85 • Dr. Punya Mishra, responsible project investigator, 509A Erickson Hall, Michigan State University, East Lansing, MI, 48824. punya@msu.edu (517) 353-7211. • Kristen Kereluik, secondary investigator, Erickson Hall, Michigan State University, East Lansing, MI, 48824. kereluik@msu.edu (269)621-5217. If you have any questions or concerns about your role and rights as a research participant, would like to obtain information or offer input, or would like to register a complaint about this research study, you may contact, anonymously if you wish, the Michigan State University’s Human Research Protection Program at 517-355-2180, FAX 517-432-4503, or e-mail irb@msu.edu, or regular mail at: 408 W. Circle Dr. Rm. 207 Olds, East Lansing, MI 48824. Your time, insights, and perceptions are valuable resources. Thank you for sharing them! By entering your initials and clicking the "I agree" button you mark your voluntarily agreement to allow your child to participate. 86 Assent We are inviting you to participate in a research study designed to help us understand how to support students in online courses. If you agree to participate in this study we will ask you to complete two questionnaires designed to better understand your learning process. If you agree to participate your name will be entered into a raffle for the chance to win one of two Apple iPads. There are no correct or incorrect answers, and your responses here will in no way impact your standing in your online course. The researchers are interested only in your general learning strategies. Your participation is voluntary and you may decline to answer the questionnaire or may skip any items that you feel uncomfortable answering. All responses are confidential and your privacy will be protected to the maximum extent allowable by law. You will be given a unique identifier and following the completion of your online course, all documents will contain only this unique identifier. There are no direct benefits to participating in this study, although we hope that you will gain more insight into your learning through participation. If you have any concerns of questions about this research study, such as scientific issues or how to do any part of it, or to report an injury, please contact the following investigators: • Dr. Punya Mishra, responsible project investigator, 509A Erickson Hall, Michigan State University, East Lansing, MI, 48824. punya@msu.edu (517) 353-7211. • Kristen Kereluik, secondary investigator, Erickson Hall, Michigan State University, East Lansing, MI, 48824. kereluik@msu.edu (269)621-5217. 87 If you have any questions or concerns about your role and rights as a research participant, would like to obtain information or offer input, or would like to register a complaint about this research study, you may contact, anonymously if you wish, the Michigan State University’s Human Research Protection Program at 517-355-2180, FAX 517-432-4503, or e-mail irb@msu.edu, or regular mail at: 408 W. Circle Drive, 207 Olds Hall, MSU, East Lansing, MI 48824. Your time, insights, and perceptions are valuable resources. Thank you for sharing them! By entering your initials you are voluntarily agreeing to participate in this research study. 88 APPENDIX C Course Mapping Table 14 Course week number by semester Week # Semester 1 Semester 2 Week 1 9/4/12 1/28/13 Week 2 9/10/12 2/4/13 Week 3 9/17/12 2/11/13 Week 4 9/24/12 2/18/13 Week 5 10/1/12 2/25/13 Week 6 10/8/12 3/11/13 Week 7 10/15/12 3/18/13 Week 8 10/22/12 3/25/13 Week 9 10/29/12 4/8/13 Week 10 11/5/12 4/15/13 Week 11 11/12/12 4/22/13 Week 12 11/26/12 4/29/13 Week 13 12/3/12 5/6/13 Week 14 12/10/12 5/13/13 Week 15 12/17/12 5/20/13 Week 16 1/7/13 5/27/13 Week 17 1/14/13 6/3/13 Week 18 1/21/13 6/10/13 89 BIBLIOGRAPHY 90 BIBLIOGRAPHY Artelt, C., Baumert, J., Julius-McElvany, N., & Peschar, J. (2003). Learners for life: Student approaches to learning results from PISA 2000. Organization for Economic Co-Operation and Development. Azevedo, Roger. (2005). Using Hypermedia as a Metacognitive Tool for Enhancing Student Learning? The Role of Self-Regulated Learning. Educational Psychologist, 40(4), 199-209. Routledge. doi:10.1207/s15326985ep4004_2 Azevedo, Roger, & Cromley, J. G. (2004). Does Training on Self-Regulated Learning Facilitate Students’ Learning With Hypermedia? Journal of Educational Psychology, 96(3), 523-535. doi:10.1037/0022-0663.96.3.523 Azevedo, Roger, Cromley, J. G., Thomas, L., Seibert, D., & Tron, M. (2003). Online process scaffolding ands students’ self-regulated learning with hypermedia. Annual Meeting of the American Educational Research Association. Azevedo, Roger, Cromley, J. G., Winters, F. I., Moos, D. C., & Greene, J. A. (2005). Adaptive human scaffolding facilitates adolescents’ self-regulated learning with hypermedia. Instructional Science, 33(5-6), 381-412. Springer Netherlands. doi:10.1007/s11251-0051273-8 Azevedo, Roger, Moos, D. C., Greene, J. A., Winters, F. I., & Cromley, J. G. (2008). Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research & Development, 56(1), 45-72. Springer. doi:10.1007/s11423-007-9067-0 Azevedo, Roger, & Jacobson, M. J. (2008). Advances in scaffolding learning with hypertext and hypermedia: a summary and critical analysis. Educational Technology Research & Development, 56(1), 93-100. doi:10.1007/s11423-007-9064-3 Azevedo, R., Witherspoon, A., Chauncey, A., Burkett, C., & Fike, A. (2009). MetaTutor: A metacognitive tool for enhancing self-regulated learning. Cognitive and Metacognitive Educational Systems: Papers from the AAAI Fall Symposium (FS-09-02). Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. (E. Cliffs, Ed.)PrenticeHall series in social learning theory (Vol. 1, p. 617). Prentice-Hall. Retrieved from http://www.amazon.com/dp/013815614X Bandura, Albert. (1982). Self-efficacy mechanism in human agency. American Psychologist, 37(2), 122-147. American Psychological Association. doi:10.1037/0003-066X.37.2.122 Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P. A., . . . Huang, B. (2004). How does distance education compare with classroom 91 instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74(3), 379–439. Boekaerts, M., & Cascallar, E. (2006). How Far Have We Moved Toward the Integration of Theory and Practice in Self-Regulation? Educational Psychology Review, 18(3), 199-210. Springer Netherlands. doi:10.1007/s10648-006-9013-4 Brown, A. L. (1987). Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In F. E. Weinert & R. H. Kluwe (Eds.), Metacognition motivation and understanding (Vol. 65, pp. 65-116). Erlbaum. Retrieved from http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Metacognition,+Executiv e+Control,+Self-Regulation,+and+Other+More+Mysterious+Mechanisms#0 Cleary, T. J., & Zimmerman, B. J. (2004). Self-regulation empowerment program: A schoolbased program to enhance self-regulated and self-motivated cycles of student learning. Psychology in the Schools, 41(5), 537-550. Wiley Online Library. doi:10.1002/pits.10177 Corno, L., & Mandinach, E. B. (1983). The role of cognitive engagement in classroom learning and motivation. Educational Psychologist, 18(2), 88–108. Routledge. doi:10.1080/00461528309529266 Dabbagh, N., Kitsantas, A. (2005). The role of Web-based pedagogical tools in supporting student self-regulation in distributed learning environments. Instructional Science, 25, 2437. Devolder, A., van Braak, J., & Tondeur, J. (2012). Supporting self-regulated learning in computer-based learning environments: systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted Learning, no--no. Blackwell Publishing Ltd. doi:10.1111/j.1365-2729.2011.00476.x Dillon, A., & Gabbard, R. (1998). Hypermedia as an Educational Technology: A Review of the Quantitative Research Literature on Learner Comprehension, Control, and Style. Review of Educational Research, 68(3), 322-349. rer.sagepub.com. doi:10.3102/00346543068003322 Dinsmore, D. L., Alexander, P. A., & Loughlin, S. M. (2008). Focusing the Conceptual Lens on Metacognition, Self-regulation, and Self-regulated Learning. Educational Psychology Review, 20(4), 391-409. Springer. doi:10.1007/s10648-008-9083-6 Flavell, J. H. (1979). Metacognition and cognitive monitoring. American Psychologist, 34(10), 906–911. Retrieved from http://psycnet.apa.org/journals/amp/34/10/906.pdf Glass, G.V., Peckham, P.D., & Sanders, J.R. (1972). Consequences of failure to meet the assumptions underlying the fixed effects analysis of variance and covariance. Review of Educational Research, 42, 237-288. 92 Hannafin, M., Land, S., & Oliver, K. (1999). Open learning environments: Foundations, methods, and models. In C. M. Reigeluth (Ed.), Instructionaldesign theories and models Volume II A new paradigm of instructional theory (Vol. 2, pp. 115-140). Lawrence Erlbaum Associates. Retrieved from http://books.google.com/books?hl=en&lr=&id=AbJc4Kg6XQoC&oi=fnd&a mp;pg=PA115&dq=Open+learning+environments:+Foundations,+methods,+and+mod els&ots=WAjws9OeoC&sig=vzyGRxhiOwfzAeJ7XMhX5yWn_S4 Horn, M. B., & Staker, H. (2011). The rise ok K-12 Blended Learning. Innosight Institute Retrieved on September (Vol. 21, p. 2011). Retrieved from http://www.innosightinstitute.org/media-room/publications/education-publications/the-riseof-k-12-blended-learning/ Hypermedia. (n.d.). In Wikipedia. Retrieved November 15, 2013, from http://en.wikipedia.org/wiki/Hypermedia Kauffman D.F., Xun G., Kui X. & Ching-Huei C. (2008) Prompting in web-based environments: supporting selfmonitoring and problem solving skills in college students. Journal of Educational Computing Research 38, 115–137. Kereluik, K., & Mishra, P. (2011). Adolescents’ activities online and how their notions of learning shape strategies and expectations. Proceedings of Society for Information Technology & Teacher Education (pp. 33-42). Chesapeake, VA: AACE. Kereluik, K., & Mishra, P. (2012). Getting the digital generation to integrate technology in their learning: A framework for disciplinary learning with technology for adolescents. In P. Resta (Ed.), Proceedings of Society for Information Technology & Teacher Education International Conference 2012 (pp. 2883-2892). Austin, Texas, USA: AACE. Retrieved from http://www.editlib.org/p/40027 Lajoie, S. P., & Azevedo, R. (2006). Teaching and learning in technology-rich environments. In P A Alexander & P. H. Winne (Eds.), Van Weert TJ Munro RK eds Informatics and the Digital Society Social Ethical and Cognitive Issues (pp. 803-821). Erlbaum. Retrieved from http://www.sociotech-lit.de/Herr03-LaT.pdf Learning management system. (n.d.). In Wikipedia. Retrieved November 15, 2013 from http://en.wikipedia.org/wiki/Learning_management_system Means, B., Toyama, Y., Murphy, R., Bakia, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115(3), 147. Michigan Virtual University. (2012). Recommendations for full time online learnders. Lansing, MI: Author. Retrieved from http://www.mivu.org/LinkClick.aspx?fileticket=T%2fdeTj6Qdsw%3d&tabid=373 93 Michigan Virtual University. (2012). A report to the legislature. Lansing, MI: Author. Retrieved from http://www.mivu.org/LinkClick.aspx?fileticket=aKm6ti3JdnQ%3d Paris, S. G., & Winograd, P. (1990). How Metacognition Can Promote Academic Learning and Instruction. Dimensions of Thinking and Cognitive Instruction, 15–51. Beau Fly Jones et Lorna Idol ed., Hillsdale, Lawrence Erlbaum Associates. Retrieved from http://books.google.co.uk/books?hl=en&lr=&id=CJtasg9ODr8C&oi=fnd&pg=PA15&dq=m etacognitive+skills+academic+success&ots=zZTqA59zde&sig=nQkPMVRejaMTZwlHyL_ NeqqRtjw Roblyer, M. D., & Davis, L. (2008). Predicting Success for Virtual School Students: Putting Research-based Models into Practice. Online Journal of Distance Learning Administration, 11(4), 1-9. University of West Georgia Distance and Distributed Education Center. Retrieved from http://www.westga.edu/~distance/ojdla/winter114/roblyer114.html Roblyer, M. D., & Marshall, J. C. (2003). Predicting Success of Virtual High School Students: Preliminary Results from an Educational Success Prediction Instrument. Journal of Research on Technology in Education, 35(2), 1-16. Rosenthal, T. L., & Bandura, A. (1978). Psychological modeling: Theory and practice. In S. L. Garfield & A. E. Bergin (Eds,), Handbook of psychotherapy and behavior change: An empirical analysis (2nd ed., pp. 621-658). New York: Wiley. Schunk, D.H. (1989). Self-efficacy and achievement behaviors. Educational Psychology, 11, 347-369. Schunk, D.H. (1996). Goal and self-evaluative influences during children’s cognitive skill learning. American Educational Research Journal, 33, 359–382. Sharma, P., & Hannafin, M. (2007). Scaffolding in technology-enhanced learning environments. Interactive Learning Environments, 15(1), 27-46. Smith, R., Clark, T., & Blomeyer, R. L. (2005). A Synthesis of New Research on K – 12 Online Learning. Research Studies, (November), 1-92. Watson, J., Murin, A., Vashaw, L., Gemin, B., & Rapp, C. (2010). Keeping pace with K-12 online learning: An annual review of policy and practice. Learning (Vol. 13, p. 148). Retrieved from http://kpK-12.com/cms/wp-content/uploads/KeepingPace2011.pdf Waxman, H. C., Connell, M. L., & Gray, J. (2002). A Quantitative Synthesis of Recent Research on the Effects of Teaching and Learning With Technology on Student Outcomes. Synthesis (p. 28). North Central Regional Education Laboratory. Waxman, H. C., Lin, M.-fen, & Michko, G. M. (2003). A meta-analysis of the effectiveness of teaching and learning with technology on student outcomes. Technology, (December), 1–30. Learning Point Associates. Retrieved from 94 http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:A+MetaAnalysis+of+the+Effectivenesss+of+Teaching+and+Learning+With+Technology+on+Stud ent+Outcomes#0 Winters, F. I., Greene, J. A., & Costich, C. M. (2008). Self-Regulation of Learning within Computer-based Learning Environments: A Critical Analysis. Educational Psychology Review, 20(4), 429-444. Springer Netherlands. doi:10.1007/s10648-008-9080-9 Wood, R., Bandura, A., & Bailey, T. (1990). Mechanisms governing organizational performance in complex decision making environments. Organizational Behavior and Human Decision Processes, 46, 181–201. Zimmerman, B. J. (1989). A social cognitive view of self-regulated academic learning. Journal of Educational Psychology, 81(3), 329-339. American Psychological Association. doi:10.1037/0022-0663.81.3.329 Zimmerman, B. J. (2002). Becoming a Self-Regulated Learner: An Overview. Theory Into Practice, 41(2), 64-70. Routledge. doi:10.1207/s15430421tip4102_2 Zimmerman, B. J., & Pons, M. M. (1986). Development of a Structured Interview for Assessing Student Use of Self-Regulated Learning Strategies. American Educational Research Journal, 23(4), 614. Sage Publications. doi:10.2307/1163093 95