SEARCHING THE WEB TO SOLVE AN ILL-STRUCTURED PROBLEM: ONLINE PLANNING AND ITS CONNECTION TO EPISTEMIC BELIEFS By Cui Cheng A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Educational Psychology and Educational Technology—Doctor of Philosophy 2020 ABSTRACT SEARCHING THE WEB TO SOLVE AN ILL-STRUCTURED PROBLEM: ONLINE PLANNING AND ITS CONNECTION TO EPISTEMIC BELIEFS By Cui Cheng Planning is an important contributor to open and deep learning on the Web, which is crucial for learning and problem solving in ill-structured domains. Situated in the context of online ill-structured problem solving, this study explores (a) the planning processes undergraduate learners develop during the course of online searching and learning, (b) the possible connections between learners’ online planning processes and their epistemic beliefs, (c) how learners’ planning processes connect to their actual search moves, and (d) how online planning and epistemic beliefs shape learners’ problem-solving performance. Think-alouds of learners holding different epistemic beliefs were analyzed to address the research questions. The results show that (a) learners developed three primary levels of plans for their online searching and learning; (b) learners holding expansive epistemic beliefs exhibited more complexity in a wider range of aspects involved in plan development compared to learners holding reductive epistemic beliefs; (c) learners demonstrated different search approaches to executing their plans; (d) online planning and epistemic beliefs exerted a combined influence on learners’ problem-solving performance. The findings contribute to our understanding of planning in online searching and learning and its connection to epistemic beliefs. The proposed three-level model of planning lays the groundwork for future studies to delve more deeply into the role of planning in online searching and learning, especially in complex and ill-structured learning contexts. The findings also provide practical implications for the instruction of online searching and learning strategies. Copyright by CUI CHENG 2020 To my parents, my husband, and my son. v ACKNOWLEDGMENTS Throughout the course of this dissertation and my whole doctoral experience, I have received tremendous support from many individuals and organizations. I would like to take this opportunity to publicly acknowledge and express my sincere gratitude to them. Foremost, I thank my advisor and dissertation committee chair, Dr. Rand Spiro, for his thoughtful guidance, encouragement, and support over the years, without which I would not have been able to navigate through this ill-structured, challenging, and rewarding process. His expertise, vision, and creativity in the field have always challenged and inspired me to deepen my thinking and will continue to be inspiring. I also thank each of the other members of my committee, Dr. Douglas Hartman, Dr. Cynthia Okolo, and Dr. Christine Greenhow, for the time and insightful feedback they gave to my work. I am truly grateful to the students who participated in this research, in particular those who completed the final online searching and learning task. I thank them for their time and effort, for trusting me enough to share their thoughts and beliefs related to learning, and for giving me the privilege of analyzing and presenting those thoughts and beliefs. I would like to thank the entire Educational Psychology and Educational Technology (EPET) program in the College of Education at Michigan State University, all the faculty, staff, and peer doctoral students I have worked with. Particularly, I extend a vi special thanks to Dr. John Bell, former director of the Design Studio in our program, for welcoming me into his team and providing me with great research opportunities to explore the innovative use of technologies in different educational settings. I am also grateful for the financial support, both fellowships and assistantship opportunities, I have received throughout my doctoral studies. I thank the Graduate School, the College of Education, and the EPET program for funding my dissertation research. I am deeply indebted to my friends for their encouragement, friendship, and the fun times together. Finally, I owe my sincere thanks to my family. I am especially grateful to my parents and my sister for loving me and believing in me throughout my life, to my husband for encouraging and supporting me with love every step along the way, and to my son for serving as a source of joy and inspiration. I love you all. vii TABLE OF CONTENTS LIST OF TABLES ........................................................................................................... xii LIST OF FIGURES ......................................................................................................... xiii CHAPTER 1 ........................................................................................................................ 1 Introduction ......................................................................................................................... 1 CHAPTER 2 ........................................................................................................................ 4 Literature Review ................................................................................................................ 4 Problem Solving and the Web ......................................................................................... 4 Epistemic Beliefs ............................................................................................................. 6 Epistemic Beliefs and Online Searching and Learning ................................................... 8 Planning in Online Searching and Learning .................................................................... 9 Individual Differences in Online Searching and Learning ............................................ 12 Purpose and Research Questions ................................................................................... 13 CHAPTER 3 ...................................................................................................................... 15 Method ............................................................................................................................... 15 Participant Selection Procedures ................................................................................... 15 Ill-Structured Problem-Solving Task ............................................................................ 17 Instruments and Measures ............................................................................................. 18 General Epistemic Beliefs ......................................................................................... 18 The Cognitive Flexibility Inventory ...................................................................... 19 Schommer’s Epistemological Questionnaire ........................................................ 19 Prior Content Knowledge .......................................................................................... 20 Verbal Ability ............................................................................................................ 20 Ill-Structured Problem-Solving Performance ............................................................ 20 Data Collection Procedures ........................................................................................... 21 Data Analysis Procedures .............................................................................................. 22 viii Qualitative Analysis of Data ..................................................................................... 22 Qualitative Analysis of Online Planning Processes .............................................. 23 Qualitative Analysis of Search Terms ................................................................... 26 Qualitative Analysis of Sources of Information Explored .................................... 26 Qualitative Analysis of Ill-Structured Problem-Solving Performance .................. 26 Quantitative Analysis of Measures ............................................................................ 26 Quantifying General Epistemic Beliefs ................................................................. 26 Quantifying Online Planning ................................................................................. 27 Quantifying Prior Knowledge ............................................................................... 27 Quantifying Verbal Ability .................................................................................... 27 Quantifying Problem-Solving Performance .......................................................... 27 CHAPTER 4 ...................................................................................................................... 29 Results ............................................................................................................................... 29 Addressing the Research Questions .............................................................................. 29 Learner Profiles ............................................................................................................. 30 Online Planning Processes ............................................................................................ 31 Global Plan ................................................................................................................ 31 Identifying or Addressing General Learning Goals .............................................. 32 Developing a Stance on the Topic ......................................................................... 32 Sub-Plan .................................................................................................................... 33 Web-Directed Sub-Plan ......................................................................................... 33 Web-Directed Immediate Sub-Plan ................................................................... 33 Web-Directed Deferred/Reflective Sub-Plan .................................................... 34 Learner-Directed Sub-Plan .................................................................................... 35 Learner-Directed Top-Down Sub-Plan ............................................................. 35 Learner-Directed Reciprocally Adaptive Sub-Plan ........................................... 35 Immediate Specific Search Plan ................................................................................ 36 Generating Search Terms or Defining Search Range ............................................ 37 Developing Exploration Paths of Search Results .................................................. 37 ix Evaluating Information Helpfulness, Currency, or Credibility ............................. 37 Controlling Reading Depth or Pace ....................................................................... 38 Taking Notes ......................................................................................................... 38 Reflecting on Plan Execution ................................................................................ 38 Comparing Information From Different Sources .................................................. 39 Online Planning and Epistemic Beliefs ......................................................................... 39 Planning Trajectories and Epistemic Beliefs ............................................................. 39 Global Plans and Epistemic Beliefs .......................................................................... 45 Sub-Plans and Epistemic Beliefs ............................................................................... 46 Immediate Specific Search Plans and Epistemic Beliefs .......................................... 55 Online Planning and Actual Search Moves ................................................................... 60 Sub-Plan Execution ................................................................................................... 61 Use of Search Terms ................................................................................................. 61 Sources of Information Explored .............................................................................. 65 Online Planning, Epistemic Beliefs, and Problem-Solving Performance ..................... 68 CHAPTER 5 ...................................................................................................................... 73 Discussion .......................................................................................................................... 73 Planning in Online Searching and Learning .................................................................. 73 Global Plans ............................................................................................................... 73 Sub-Plans ................................................................................................................... 75 Immediate Specific Search Plans .............................................................................. 77 Online Planning and General Epistemic Beliefs ........................................................... 78 Online Planning and Actual Search Moves ................................................................... 82 Online Planning, Epistemic Beliefs, and Problem-Solving Performance ..................... 83 Implications ................................................................................................................... 85 Limitations ..................................................................................................................... 87 CHAPTER 6 ...................................................................................................................... 89 Conclusion ......................................................................................................................... 89 x APPENDICES ................................................................................................................... 91 APPENDIX A The Revised Cognitive Flexibility Inventory ....................................... 92 APPENDIX B Schommer’s Epistemological Questionnaire ........................................ 97 APPENDIX C Prior Content Knowledge Test ............................................................ 102 APPENDIX D Ill-Structured Problem-Solving Performance Grading Rubric ........... 104 APPENDIX E Think-Aloud Instructions .................................................................... 107 APPENDIX F Think-Aloud Prompts and Questions .................................................. 110 APPENDIX G Coding Scheme of Online Planning Processes ................................... 113 REFERENCES ................................................................................................................ 119 xi LIST OF TABLES Table 1 Learner Profiles………………………………………………………………………....31 Table 2 Frequencies of Web-Directed and Learner-Directed Sub-Plans………………………...46 Table 3 Sub-Plans Created by the Four Learners………………………………………………..50 Table 4 Frequencies and Proportions of Immediate Specific Search Plans……………………...57 Table 5 Search Terms Generated by the Four Learners………………………………………….64 Table 6 Information Sources Explored by the Four Learners…………………………………....65 Table 7 Ill-Structured Problem-Solving Performance Grading Rubric………………………...104 Table 8 Coding Scheme of Online Planning Processes…………………………………………113 xii LIST OF FIGURES Figure 1 Planning Trajectory Created by Learner H1…………………………………………...43 Figure 2 Planning Trajectory Created by Learner H2…………………………………………...43 Figure 3 Planning Trajectory Created by Learner L2……………………………………………44 Figure 4 Planning Trajectory Created by Learner L1……………………………………………44 xiii CHAPTER 1 Introduction Ill-structured problem-solving skills are crucial in our everyday and professional life, as many problems we encounter in the real world are ill-structured (e.g., organizing a community event, buying a home, designing an online course). In fact, in today’s rapidly changing and increasingly complex world, ill-structured problem-solving skills have become even more vital than ever before. In contrast to well-structured problems (e.g., solving a quadratic equation), which are clearly defined and have a single correct solution, ill-structured problems are vaguely defined, have no prepackaged correct solutions, and typically possess multiple possible solutions, each of which has its own advantages and disadvantages in the context of their application (Jonassen, 1997). Therefore, solving ill-structured problems, especially problems that are novel or fall outside one’s area of expertise, requires open and deep learning of related topics, featured by recognition of divergent perspectives, consideration of different contexts, and creative and flexible synthesis of one’s own understanding in response to the specific problem-solving contexts (Jonassen, 1997; Spiro et al., 2004). The Web has great potential for facilitating this type of open and deep learning necessary for solving ill-structured problems, as it enables learners to search for multiple perspectives, to criss-cross complex learning content from different directions or contexts, and to make flexible connections of ideas through quick traversals of related topics (Spiro et al., 2003; Spiro & Jehng, 1990). In the meantime, however, the nonlinear structure and the openness of the Web present 1 learners with additional (meta)cognitive challenges compared with traditional learning with printed texts (Hartman et al., 2010; Shapiro & Niederhauser, 2004; Spiro et al., 2015). To address these challenges, a constellation of strategies (e.g., searching and locating, evaluating, planning, monitoring, synthesizing) has been identified in accomplished online searching and learning (Afflerbach & Cho, 2009; Coiro & Dobler, 2007; DeSchryver, 2014; Goldman et al., 2012; Leu et al., 2013). Among these strategies, planning is a crucial contributor to open and deep learning on the Web because without effective and flexible use of planning, learners may easily get distracted by irrelevant information or even get lost in the nonlinear and boundless Web learning environment, thus failing to make use of the great potential of the Web. Prior studies have identified the presence of planning processes (Coiro & Dobler, 2007; Zhang & Duke, 2008) in online searching and learning, but little is known about the different types and characteristics of planning processes learners develop, especially in the context of ill-structured problem solving. To address this research gap, this study made an in-depth analysis of the planning processes undergraduate learners develop in online searching and learning for investigating an ill-structured problem. Given the importance of online planning, a natural question arises: what factors would influence how learners develop their planning processes? A growing number of studies (e.g., Castek et al., 2012; Coiro, 2011; Ulyshen et al., 2015) have been conducted to investigate the roles of different individual differences (e.g., prior knowledge, age, and epistemic beliefs) in online searching and learning in general. This study focused on exploring how one potential influencing 2 factor – epistemic beliefs – shapes learners’ online planning processes. The reason for looking at epistemic beliefs is that learners may hold different beliefs about understanding and approaching complex and novel phenomena (Spiro et al., 1996), and therefore, examining possible connections between epistemic beliefs and online planning will help us better understand the characteristics of the latter in complex and ill-structured learning contexts. 3 CHAPTER 2 Literature Review Problem Solving and the Web Problem-solving is generally regarded as the most important and meaningful cognitive activity in everyday and professional life (Gagne, 1964; Jonassen, 2000). People are constantly challenged by problems of different natures and difficulty levels. Based on problems’ structure, complexity, and abstractness, researchers have classified problems into two major types - well-structured problems and ill-structured problems (Jonassen, 1997; Simon, 1973), sometimes referred to as well-defined problems and ill-defined problems (Reitman, 1964; Schraw et al., 1995). Well-structured problems have “convergent solutions that engage the application of a limited number of rules and principles within well-defined parameters” (Jonassen, 1997, p. 65). They are often found in standardized academic achievement tests or at the end of textbook chapters (e.g., solving a quadratic equation, calculating the distance traveled by an object using velocity and time). In contrast, ill-structured problems “possess multiple solutions, solution paths, or no solutions at all” and “contain uncertainty about which concepts, rules, and principles are necessary for the solution or how they are organized, and which solution is best” (Jonassen, 1997, p. 65). These problems are typically situated in real-world settings (e.g., buying a car with financial constraints, designing an online course from scratch). Well-structured problems and ill-structured problems call on different problem-solving skills and mindsets because of the fundamental differences between the two (Schraw et al., 1995; 4 Spiro et al., 2004). To solve well-structured problems, a constrained set of knowledge (e.g., identifying the well-defined concepts, recalling the appropriate algorithms or procedures, and correctly applying them to the problem at hand) is required to find the correct answer. For ill-structured problem solving, however, there are no prepackaged solutions for learners to recall or search for, and it requires a broader range of knowledge related to multiple topic domains, identification and comparison of different or even contradictory perspectives, flexible and situation-specific construction and application of one’s own solution (Spiro et al., 2004). In today’s world that is full of change, complexity, and irregularity, learning and problem-solving skills in ill-structured domains are central to success in our everyday and professional life (Spiro et al., 2017). The Web has become an indispensable information searching and learning environment for ill-structured problem-solving. On the one hand, the Web possesses great potential for facilitating the deep and flexible learning necessary for ill-structured problem solving, as it enables learners to search for multiple perspectives, to criss-cross the complex learning content from different directions or contexts, and to make flexible connections of ideas through quick traversals of the topic (Spiro et al., 2003; Spiro & Jehng, 1990). On the other hand, the nonlinear structure and the openness of the Web present learners with additional (meta)cognitive challenges, when compared with traditional learning with printed texts (Hartman et al., 2010; Shapiro & Niederhauser, 2004). These issues have led to a small but growing number of studies on the use of the Web for ill-structured problem-solving (e.g., DeSchryver, 2014; Ulyshen et al., 2015), and this study aimed to contribute to this body of research by investigating 5 the planning processes learners engage in for solving ill-structured problems on the Web and how the planning processes connect to their epistemic beliefs. Epistemic Beliefs The term “epistemic beliefs” has been generally used to refer to learners’ beliefs about the nature of knowledge and knowing (Hofer & Pintrich, 1997). There are multiple perspectives on learners’ epistemic beliefs, each emphasizing different facets of the construct. Some studies focused on the developmental nature of personal epistemology, proposing that personal epistemology develops from simple to complex stages as an integrated cognitive structure (e.g., King & Kitchener, 1994; Perry, 1968). Some (e.g., Schommer, 1990) questioned the uni-dimensional concept in developmental perspectives, arguing for a multidimensional perspective of epistemic beliefs. For instance, Schommer (1990) identified five more or less independent dimensions of epistemic beliefs: (a) structure of knowledge (beliefs about whether knowledge is simple rather than complex), (b) certainty of knowledge (beliefs about whether knowledge is certain rather than tentative), (c) source of knowledge (beliefs about whether knowledge is handed down by authority rather than derived from reason), (d) control of knowledge acquisition (beliefs about whether the ability to learn is innate rather than acquired), and (e) speed of knowledge acquisition (beliefs about whether learning takes place quickly). Some studies were concerned with other dimensions, such as flexibility of knowledge and learning (Spiro et al., 1996) and justification for knowing (Hofer, 2004). Yet some other studies (Buehl et al., 2002; Hofer, 2004) focused on the domain-specific nature of epistemic beliefs, arguing that learners’ epistemic 6 beliefs may differ across domains or contexts. For instance, Buehl et al. (2002) found that learners’ beliefs about knowledge in a more well-structured learning domain (e.g., math) differed from those reflected in a more ill-structured learning domain (e.g., history). This study was informed by Spiro et al.’s (1996) perspective on epistemic beliefs. Gearing towards ill-structured learning domains, Spiro et al. focused on a particular dimension of epistemic beliefs - the flexibility of knowledge and learning, which is crucial for advanced knowledge acquisition in ill-structured domains. Along with this dimension, they presented two kinds of epistemic worldviews: the expansive and flexible worldview and the reductive worldview. Specifically, learners holding the expansive and flexible epistemic worldview tend to perceive the world as complex and disorderly, prefer to examine complex phenomena with multiple frameworks or perspectives, consider components of phenomena as highly interconnected, have a high degree of tolerance of ambiguity, and actively construct knowledge in a flexible and situation-adaptive way. In contrast, learners holding the reductive epistemic worldview tend to account for complex phenomena with a single framework or perspective, consider components of phenomena as independent, perceive the world as simple and orderly, have a low degree of tolerance of ambiguity, and passively receives information presented by authorities. According to Spiro et al., the expansive and flexible epistemic worldview is “conducive to the processing of complexity” in ill-structured domains, whereas the reductive worldview is “associated with various kinds of oversimplification of complexity known to be related to learning failure in 7 ill-structured domains” (p. S51). They further validated an assessment instrument – the Cognitive Flexibility Inventory (CFI) – for assessing the two epistemic worldviews. Spiro et al.’s (1996) perspective on epistemic beliefs is helpful in framing this study because (a) this study is particularly situated in the context of ill-structured problem-solving, and (b) the activity of searching and learning on the open and nonlinear Web itself is complex and ill-structured. Epistemic Beliefs and Online Searching and Learning The connection between learners’ epistemic beliefs and their online searching and learning practices is an emerging area of research. Some studies revealed a positive relationship between learners’ complex epistemic beliefs and the sophistication of cognitive strategies (e.g., Ulyshen et al., 2015), or their knowledge gain and transfer performance (e.g, Cho et al., 2018). Learners with more complex epistemic beliefs were found to be more likely to (a) compare and integrate multiple online sources for knowledge construction (Barzilai & Zohar, 2012; Ulyshen et al., 2015), (b) evaluate information they encounter on the Web (Hofer, 2004; Ulyshen et al., 2015), (c) set and complete learning goals (Ulyshen et al., 2015), (d) gain greater topic knowledge from online searching (Cho et al., 2018), (d) benefit from a nonlinear, multidimensional hypertext learning environment and transfer what they have learned in new ways or to new situations (Jacobson & Spiro, 1995), (d) engage in Internet-based discussion and communication (Bråten & Strømsø, 2006), and (e) prefer Internet-based learning environments that encourage inquiry learning and reflective thinking (Tsai & Chuang, 2005). However, some studies found no evidence for the 8 connection between epistemic beliefs and the use of source evaluation strategies (Barzilai & Zohar, 2012) or inconsistent findings on the relationship between epistemic beliefs and reflective judgment in online information seeking (Whitmire, 2004). These inconsistent findings revealed more subtleties of the connections between epistemic beliefs and online searching and learning, which require further investigations. Besides, few studies have explored the particular connection between epistemic beliefs and learners’ use of online planning strategies. Although Ulyshen et al. (2015) identified a positive relationship between complex epistemic beliefs and goal setting and completing in general, little is known about how epistemic beliefs shape the particular patterns of online searching and learning plans. Planning in Online Searching and Learning When learners move from printed texts to the open Web learning environment, the complexity of the learning environment increases in both the scope of information and the nonlinearity of structure. This increased complexity presents additional metacognitive challenges to learners. One crucial metacognitive challenge is the greater demand for planning. Researchers have identified the use of planning strategies in successful online searching and learning (Coiro & Dobler, 2007; Steffens, 2012). The planning strategies discussed in these studies primarily focused on such specific processes as deciding what search terms to use to locate information and what links or websites to look at for details. Hartman and his colleagues (2010), however, posited that a broader set of goal knowledge is needed to guide learners through the nonlinear, boundless online 9 learning environment. There are so many choices on the Web competing for attention that without explicit reading goals online learners may easily get distracted, or even get lost. According to Hartman et al. (2010), planning should enable an online learner to “formulate relevant and realistic goals, categorize and evaluate Web content in relation to these goals, adjust goals in response to what the reader finds is available and relevant, monitor progress, and determine when the goal has been attained” (p. 149). Integrating the above interpretations of the planning process involved in online searching and learning, Cheng and Spiro (2016) proposed in a preliminary study that an intermediate level of planning could help bridge the broad level and the specific level of planning, especially in the context of solving ill-structured problems that require examination of multiple aspects or perspectives related to a topic. Specifically, a three-level model of planning was developed in the context of accomplished online searching and learning for solving ill-structured problems. The first level contains a global plan (or global plans) for the purpose of learning and problem solving in the context of a particular task. The second level consists of a series of sub-plans focusing on particular goals within the larger task. The third level comprises immediate specific search plans to fulfill higher levels of plans. It was found that higher levels of plans inform lower levels of plans, affecting the specifics of the search for information on the Web, and that information on the Web feeds back to the formation and change of plans at different levels. Since this study builds upon this model, more detailed examples are provided below to illustrate the dynamics across the three levels. Take the ill-structured problem-solving context 10 used in the preliminary study (Cheng & Spiro, 2016) - investigating whether genetically modified foods should be served in school cafeterias – for example. A learner can first develop a global plan – exploring different perspectives on the issue and forming his or her own understanding. Under this global plan, the learner can develop multiple sub-plans, such as looking for what genetically modified foods are, finding some examples of most common genetically modified foods, benefits of genetically modified foods, disadvantages of genetically modified foods, and how to identify genetically modified foods. When the learner is focusing on a particular sub-plan such as finding out possible disadvantages of genetically modified foods, he or she needs to decide on what specific search terms to use (e.g. disadvantages or drawbacks of genetically modified foods), what particular websites to look at, how much time to spend on each website, etc., which constitute the immediate specific search plans. What the learner encounters on the Web (e.g., information about the short-term influences of genetically modified foods) may lead him or her to a specific search plan (e.g., spending more time to read that information for accurate understanding) or affect what the learner decides to explore for the next sub-plan (e.g., long-term influences of genetically modified foods). Seeing the short-term and long-term influences may cause a change in the learner’s opinion on the given topic, which is part of the global plan. This study aimed to further develop and test the efficacy of this three-level model of online planning with a different and more diverse group of students. 11 Individual Differences in Online Searching and Learning To examine the connection between online planning and epistemic beliefs, some other individual difference factors that may influence learners’ online searching and learning should be taken into account. However, a challenge is that our understanding of individual differences in online searching and learning is limited by a relatively small number of studies in this area and that what is known about individual differences in print-based learning cannot be simply applied to online searching and learning, as the two are not necessarily isomorphic (Leu et al., 2016). Based on the findings of emerging studies that have examined the role of individual differences in online searching and learning, this study took into consideration several individual difference factors: prior content knowledge, verbal ability, and learning time. Prior content knowledge has been found to be a crucial contributor to print-based learning (e.g., Spilich et al., 1979). However, findings of the role of prior content knowledge in online searching and learning are inconsistent. Some studies found that learners with high levels of prior content knowledge performed better on navigating, using, and comprehending information in hypertext environments (e.g., Dee-Lucas, 1999; Potelle & Rouet, 2003). Results of some other studies showed that learners’ content knowledge did not predict their success in search-required online learning environments (Coiro, 2011; Willoughby et al., 2009), suggesting that prior content knowledge may be somehow less important in online searching and learning. Despite these inconsistent findings, this study examined learners’ prior content knowledge to account for its possible effects on learners’ online searching and learning. 12 Generally, previous research findings support the notion that verbal ability helps facilitate reading comprehension (Anderson & Freebody, 1979; Qian, 1999) and related to general intelligence (Anderson & Freebody, 1979). To take into consideration the possible effects of verbal ability on online planning, this study assessed learners’ verbal ability for analysis. There is little research specifically examining the effects of learning time on online searching and learning. Intuitively, learning time would influence learners’ engagement and use of strategies in online searching and learning. Ulyshen et al. (2015) found that learning time was positively connected to learners’ self-perceived thoroughness of knowledge exploration and actual learning complexity in online searching and learning for complex problem solving. It should be acknowledged that these factors are by no means an exhaustive list of individual differences that may influence learners’ online planning processes and overall online learning performance. More possible influencing factors should be taken into account for future studies. Purpose and Research Questions The purpose of this study was twofold: (a) to further develop and test the efficacy of a three-level model of planning proposed in a pilot study (Cheng & Spiro, 2016) by identifying the planning processes that a more diverse group of students develop in online searching and learning for ill-structured problem-solving and (b) to investigate how learners’ planning processes connect to their epistemic beliefs, actual search moves, and problem-solving performance. Specifically, 13 this study aimed to address two primary research questions (Research Questions 1 and 2) and two ancillary research questions (Research Questions 3 and 4): 1. What planning processes do undergraduate learners develop in online searching and learning for solving a complex, ill-structured problem? 2. How do the online planning processes developed for ill-structured problem solving connect to learners’ general epistemic beliefs? 3. How do the online planning processes developed for ill-structured problem solving connect to learners’ actual search moves in terms of their plan execution, generation of search terms, and exploration of webpages? 4. How do learners’ online planning processes and epistemic beliefs shape their ill-structured problem-solving performance? 14 CHAPTER 3 Method Participant Selection Procedures An online survey, which combined the Cognitive Flexibility Inventory (Spiro et al., 1996) and Schommer’s Epistemological Questionnaire (Schommer, 1990), was first administered to identify a group of students with expansive and flexible epistemic beliefs and another group of students with relatively reductive epistemic beliefs. About 1400 undergraduate students, who enrolled in a research participant pool and took English as their native language, received an email invitation to take the survey. A limit of 80 participants was set on the survey, and 70 students completed it. Instead of averaging the scores of the Cognitive Flexibility Inventory (CFI) and Schommer’s Epistemological Questionnaire (SEQ), the CFI was used as the primary instrument for measuring learners’ general epistemic beliefs, and SEQ as a secondary one. This choice was based on two considerations: First, the CFI was designed particularly for assessing beliefs and preferences about learning in complex and ill-structured learning domains, in which this study was situated. Second, in this study, the CFI performed better in reflecting the heterogeneity of learners’ epistemic beliefs (the average score for the CFI was 4.54 and the standard deviation of the CFI scores was 0.54; the average score for the SEQ was 4.86, and the standard deviation of the SEQ scores was 0.37). 15 Students’ scores in the CFI were ranked from low to high, with low scores representing relatively reductive epistemic beliefs and high scores indicating relatively expansive and flexible epistemic beliefs. In cases where two or more students obtained the same CFI score, those learners’ scores in SEQ were further compared to determine the ranking of their epistemic beliefs. Ten students at the two extremes of the ranking (five at each extreme) were selected to form two contrastive groups - the expansive epistemic belief group and the reductive epistemic belief group. Before entering the next stage of the study, two students who ranked the lowest in the reductive epistemic belief group opted out of the study. Another two students who ranked the lowest among the remaining 60 participants were then invited for substitution, and both students accepted the invitation. Each of the 10 selected students completed a 2.5-hour online searching and learning session to solve an ill-structured problem. Considering the scarcity of our knowledge about the planning processes involved in online searching and learning, this study further selected four students from the aforementioned 10 students as the final participants to allow for an in-depth analysis of learners’ online planning and the associated concepts raised in the research questions. Of the four final participants, two students ranked the highest in the expansive epistemic belief group and the other two ranked the lowest in the reductive epistemic belief group, that is, the four final participants were at the two extremes of the overall ranking (two from each extreme). Although the learner who ranked the second highest was not able to use the full learning time allotted because of Internet issues, this learner was still included in the study for two reasons: First, the primary aim of the study was to make an in-depth 16 qualitative analysis of learners’ online planning processes and actual search moves, and it was anticipated that having that learner spend 15 minutes less on the task would not exert substantial influence on the qualitative analysis. Second, for analysis involving quantitative comparisons, the length of learning time would be taken into consideration. Ill-Structured Problem-Solving Task The problem-solving task asked participants to search the open Web to investigate an ill-structured problem, which was described as follows: Considering the increasing interest regarding genetically modified foods, the Food Services Office in your university will discuss at an upcoming meeting whether and how genetically modified foods should be served at the school cafeterias and dining halls. As one of the student representatives at the Food Services Office, you are expected to be an active part of the discussion. Specifically, you are responsible for writing a report to present a thorough perspective on this topic and your suggested solution to the problem. In your report, make sure to include specific evidence to support your statements. You have two 45-minute learning sessions (90 minutes in total) to search the Web for any information you need, and then you have 20 minutes to write up your report. This problem-solving task was adapted from the one used in a prior study (Cheng & Spiro, 2016) to meet the specific needs of this study. Specifically, the task was (re)designed based on the following considerations: 17 1. It should be an ill-structured problem that has no pre-existing solutions. This task is complex and ill-structured in that it involves multiple issues (e.g., scientific, economic, ecological, environmental, and even ethical issues), multiple stakeholders, and multiple perspectives, and that it can be legitimately approached in different ways. 2. The task description should avoid priming learners about any perspectives that can be taken on the topic. 3. It was anticipated that participants would have some familiarity and interest in this topic. This topic has been widely discussed on the Web and in TV shows and newspapers, and it is closely related to people’s daily life. In the meantime, it was anticipated that participants would not possess a high level of knowledge on this topic and thus need to explore the Web for necessary information. 4. The design of the task aimed to resemble a complex, real-world problem that typically requires flexible and context-adaptive assembly and synthesis of information. 5. Previous experience with task design in a pilot study (Cheng & Spiro, 2016) indicated that learners should be able to learn reasonably sufficient information to construct their responses to the task within the given time constraints. Instruments and Measures General Epistemic Beliefs As explained in the participant selection section, the Cognitive Flexibility Inventory (CFI) and Schommer’s Epistemological Questionnaire (SEQ) were administered to measure learners’ 18 general epistemic beliefs, with the former as the primary instrument and the latter as a secondary one. The Cognitive Flexibility Inventory. The Cognitive Flexibility Inventory (CFI) was designed particularly for measuring beliefs and preferences about learning in complex and ill-structured learning domains. The CFI comprised 15 pairs of oppositional statements on a 7-point Likert scale. Some statements were edited to be more accessible to undergraduate students. The revised CFI was administered in two parts, each containing a balanced number of expansive and reductive statements of epistemic beliefs (see Appendix A for the revised CFI). An overall CFI score, obtained by averaging the scores of 30 ratings, was used to measure each learner’s general epistemic beliefs. Higher scores represent more expansive epistemic beliefs, and lower scores indicate more reductive epistemic beliefs. Schommer’s Epistemological Questionnaire. Schommer’s Epistemological Questionnaire (SEQ) has been commonly used for measuring general epistemic beliefs. Although Schommer’s perspective, as presented in the literature review section, does not directly map onto the expansive and reductive dimensions of epistemic beliefs, some items in SEQ are inferentially related to those in the CFI. For instance, if a learner believes that learning is fast, he or she is less likely to examine multiple perspectives on a topic. Therefore, SEQ was used as a secondary instrument for measuring learners’ epistemic beliefs. SEQ assesses epistemic beliefs on five dimensions (see Appendix B): (a) structure of knowledge (ranging from isolated bits and pieces to integrated concepts), (b) stability of 19 knowledge (ranging from unchanging to continually changing), (c) source of knowledge (ranging from handed down by authority to derived from empirical evidence and reasoning), (d) speed of learning (ranging from quick all-or-none to gradual), and (e) ability to learn (ranging from fixed at birth to improvable over time and experience). The original SEQ consists of 63 items on a 5-point Likert scale. To be consistent with the design of the CFI, SEQ used in this study employed a 7-point Likert scale as well. Prior Content Knowledge Although current findings of the role of prior content knowledge are inconsistent, a factual knowledge test (Appendix C) about the learning topic was administered to assess learners’ prior content knowledge to account for its possible effects on online planning. Verbal Ability To take into consideration the possible effects of verbal ability on learners’ online planning processes, a 6-minute extended range vocabulary test (Ekstrom et al., 1976) was used to assess participants’ verbal ability. Ill-Structured Problem-Solving Performance Participants’ problem-solving performance was measured by their written responses to the ill-structured problem. Assessment of responses was based on a grading rubric (see Appendix D), which consisted of eight grading criteria with a 4-point grading scale for each criterion. Four criteria were adapted from Toy’s (2007) study, which drew on the ill-structured problem-solving process proposed by Jonassen (1997); two criteria (situation-adaptiveness of response and 20 synthetic integration of ideas) were created based on the characteristics of expansive or reductive epistemic beliefs depicted by (Spiro et al., 1996); two criteria of “clarity, accuracy, and concision of information representation” and “completeness of response” were added to assess the quality and completeness of responses. The design of the grading rubric aimed to reflect the specific requirements of the given problem-solving task in this study and the levels of expansiveness/flexibility of the written responses. Data Collection Procedures Each participant met with the researcher individually to complete a 2.5-hour online problem-solving session, which consisted of five phases: (a) the pre-task test-taking phase (15 minutes) (b) the think-aloud training phase (10 minutes), (c) the online exploration phase (two 45-minute sessions and a 5-minute break in between), (d) the solution write-up phase (20 minutes), and (e) the post-task interview phase (10 minutes). In the first phase, participants were asked to complete the prior knowledge test and the verbal comprehension test. For the think-aloud training phase, instructions and a sample were provided to familiarize participants with the techniques of thinking aloud during online searching and learning (see Appendix E). During the online exploration phase, each participant had two 45-minute sessions to search the Web for investigating the ill-structured problem. While they were exploring the Web, they were asked to think out loud about their searching and learning plans and explain why they made those plans. Participants’ think-alouds synchronized with their screen navigations were 21 video-recorded. If they forgot to explain their navigational plans or decisions (e.g., selecting a search result, clicking an embedded link, or entering a new search term), they were probed to think out loud about what they were thinking, what they planned to search for, and why they made those plans/decisions. The during-task questions or prompts (see Appendix F) aimed to help participants think aloud about and explain their planning moves rather than bias their searching and learning process by leading them to certain information. For questions that might bias participants’ planning process, the researcher took notes first and asked them later at the post-interview phase (see Appendix F). After completing the online exploration phase, participants had 20 minutes to write up their solutions to the problem-solving task. In the final phase, participants were asked (a) what information they would search for if they had more time on the task, (b) why they made particular planning moves rather than others, and (c) how their search moves might relate to their beliefs about learning. Data Analysis Procedures Qualitative Analysis of Data Learners’ think-alouds, online navigations, and responses to during-task prompts and post-task interviews were transcribed into texts for qualitative analysis of their online planning, use of search terms, and exploration of different sources of information. The transcribed texts were then segmented into meaningful units of analysis based on learners’ expressed intentions of or actual navigational changes (e.g., searching for information with a new term, selecting a search 22 result to read, moving to a different aspect of the topic by clicking an embedded link, re-reading the instructions of the task, shifting to note-taking, etc.). Qualitative Analysis of Online Planning Processes. An initial coding scheme was developed based on codes generated from a three-level interactive model of planning (Cheng & Spiro, 2016) and codes developed from the researcher’s preliminary coding of the data in this study. To ensure the reliability of coding, a second round of coding was conducted by the researcher and a second coder based on the coding scheme. Before starting the second-round coding, the second coder took a training session about the concept of online planning, the coding scheme (i.e., codes, code definitions, examples for each code, and coding guidelines), and a hands-on coding exercise using a sample verbal protocol (the protocol of a learner who was excluded in this study). Based on the questions, feedback, and coding issues raised in the training session, the coding scheme was further refined. To identify and categorize all instances of planning and reduce possible bias caused by pre-existing codes toward identifying relevant data, a two-step coding process (Hsieh & Shannon, 2005) was employed for the second round of coding. At the first step, instead of directly assigning codes to the segmented units of data, the two coders individually highlighted all text units that on first impression appeared to represent learners’ planning process, checked the consistency of the highlighted texts between the two coders, and resolved discrepancies through discussion. The discussion of discrepancies was limited to whether particular texts were related to planning in general, aiming to avoid any hints for what specific codes to assign to the texts. Going through the 23 initial segmentation of data and the step of identifying codable units ensured that the two coders were coding the same texts, which was necessary to facilitate accurate inter-coder reliability assessment (Campbell et al., 2013). For the second step, the two coders went back to all the highlighted texts and coded them using an abductive approach (Morgan, 2007), first deductively coding the texts with the codes in the coding scheme and then inductively assigning a new code to any text that could not be categorized with existing codes. This approach allows for examination of both pre-determined codes and identification of new codes emerging from the data. In the second-round coding, one new code within the third level of planning was identified by the researcher and added to the coding scheme (see Appendix G for the final coding scheme of online planning processes). It is worth noting that the coding examples provided in the initial coding scheme were selected either from the protocols excluded in this study or protocols in a prior study (Cheng & Spiro, 2016) to avoid biasing the second coder’s coding decisions; the examples presented in Appendix G are from this study’s protocols. In the actual coding process, there were cases where two or more codes were assigned to the same unit of analysis. Inter-coder reliability was calculated by dividing the number of coding agreements by the number of coding agreements and disagreements combined – the proportion agreement method. Although the proportion agreement method does not take into account the possibility of agreement by chance, it was selected over other methods based on the following considerations suggested by Campbell et al. (2013): First, a relatively large number of codes in this study leads to a low 24 likelihood that the coders agreed by chance. Second, no existing and reliable coding schemes are available for this study, and some researchers have argued that the proportion agreement method is an acceptable approach for exploratory studies (Kurasaki, 2000). Third, in this study, multiple codes can be assigned to the same coding unit (see the second example presented in the next paragraph), which makes it inappropriate to use some other statistics that require only one code for a unit of analysis. The two examples below demonstrate how the proportion agreement method was implemented. The two coders assigned different codes to this excerpt of verbal protocol: “[skimming the webpage] okay, the trends and future avenues. To view the table…, okay, there’s something that caught my attention, which is five big GM seed companies. Monsanto, DuPont…” One coder used the code “Web-directed immediate sub-plan” for the plan of viewing “five big GM seed companies”, whereas the other coder used the code “controlling depth or pace of reading.” In this coding instance, one coding disagreement was counted. Take another coding instance with this excerpt for example: “I’m checking my notes to see how much I followed up on, and what to follow up on next. So I already looked at the concerns on safety, now I’m gonna look at the anti-GM campaign.” One coder assigned two codes to the excerpt - “reflecting on plan execution” and “Web-directed deferred/reflective sub-plan”, whereas the other coder assigned one code - “Web-directed deferred/reflective sub-plan.” In this coding instance, there was one coding agreement and one coding disagreement. The overall inter-coder reliability for these two coding instances would be 33.33% (one divided by three). 25 Calculated by this proportion agreement method, the overall inter-coder reliability for all the four verbal protocols was 85.40% and the levels of inter-coder reliability for the protocols of Learners H1, H2, L1, and L2 were 85.71%, 83.82%, 86.90%, and 84.78%, respectively. It is worth noting that the one newly identified code occurred only twice, leading to two coding disagreements. Therefore, the addition of the new code did not lower the inter-coder reliability in a significant way. Qualitative Analysis of Search Terms. The search terms generated by each learner were summarized for analysis. Qualitative Analysis of Sources of Information Explored. The webpages explored and time spent on each webpage were summarized for each learner and compared across the four learners. Qualitative Analysis of Ill-Structured Problem-Solving Performance. Qualitative analysis of the written responses was conducted based on each grading criterion presented in Appendix D. Quantitative Analysis of Measures Descriptive statistics were used to analyze learners’ general epistemic beliefs, online planning, prior knowledge, verbal ability, and problem-solving performance. Quantifying General Epistemic Beliefs. Learners’ responses to the statements of CFI and SEQ were scored on a range of 1 to 7, with higher scores representing more expansive epistemic beliefs and lower scores more reductive epistemic beliefs. An overall CFI score was obtained by 26 averaging the scores of all the CFI statements, and a similar calculation was conducted for getting an overall SEQ score. As explained in the measures section, the overall CFI scores were used as the primary measurement of learners’ general epistemic beliefs, and the overall SEQ scores were used as a secondary measurement. Quantifying Online Planning. The occurring frequencies of all the codes for searching and learning plans were counted for each participant. Quantifying Prior Knowledge. Learners’ performance on the prior knowledge test was graded. The overall test score was the sum of the number of correct responses to the true or false questions and multiple-choice questions and the number of correct themes demonstrated in the responses to the open-ended question. Quantifying Verbal Ability. Learners’ verbal ability was measured by their performance on the verbal comprehension test. The verbal comprehension test score was the total number of correct responses to 24 multiple-choice questions. Quantifying Problem-Solving Performance. As stated in the measures section, learners’ ill-structured problem-solving performance was assessed according to a grading rubric (Appendix D). The researcher and a second rater independently graded learners’ written responses. In cases where raters’ scores under a grading criterion differed by two or more points, the two raters discussed their ratings on that particular criterion. If the rating difference remained the same or became greater, a third rater would be introduced to grade the response based only on that particular criterion, and the final score for that criterion would be the average of three ratings. In 27 the actual grading process, only one grading instance called for discussions between the two raters, and the introduction of a third rater was not triggered. 28 CHAPTER 4 Results This section presents how the research questions were addressed, the profiles of the four selected learners (e.g., verbal ability, prior knowledge, general epistemic beliefs), and the results of each research question. Addressing the Research Questions The first research question explores what planning processes undergraduate learners develop in online searching and learning for solving an ill-structured problem. To address this question, qualitative analyses of online planning were conducted to present the different levels/types of plans learners engaged in during the course of online searching and learning. The second research question examines how learners’ planning processes connect to their general epistemic beliefs. Both descriptive statistics and qualitative analyses were conducted to compare the patterns of plans developed by the two contrastive groups. In these data analyses, learners’ prior knowledge, verbal ability, and learning time were taken into account. The third research question investigates how learners’ planning processes relate to their actual search moves in terms of execution of sub-plans, the use of search terms, and information sources explored. Qualitative analysis of learners’ search moves was conducted to address this research question. The fourth research question looks at the connection between learners’ planning processes, epistemic beliefs, and their problem-solving performance. Both descriptive statistics and 29 qualitative analyses of learners’ problem-solving performance were conducted to address this research question. Learner Profiles Based on the ranking of general epistemic beliefs, the four learners were named as Learner H1, Learner H2, Learner L2, and Learner L1. Learner H1 ranked the highest, Learner H2 the second highest, L2 the second lowest, and L1 the lowest in terms of their epistemic belief scores. In other words, Learners H1 and H2 demonstrated more expansive and more flexible general epistemic beliefs than Learners L1 and L2. Table 1 shows the profile information of each learner. At the time of data collection (a summer session), the four learners had just completed their junior year of college. They came from different major backgrounds. With respect to their verbal ability scores, Learner H1 scored the lowest, and Learner H2 scored the highest. However, it is worth noting that the scores of Learners H2, L2, and L1 all fell on a medium-high range (50.0% - 62.5% of the total score). In terms of prior knowledge, Learners H1, H2, and L2 self-reported a medium level of prior knowledge, and their actual scores were slightly lower than a medium level and demonstrated no substantial difference. Learner L1 self-reported a slightly medium-low level of prior knowledge and gained a noticeably lower score than the other three learners. Learners L1, L2, and H1 spent the same amount of time (45 minutes for each of the two learning sessions) on online searching and learning, and Learner H2 only spent 35 minutes on the first learning session and 40 minutes on the second learning session due to Internet issues. Further analysis is provided in the section of online planning and 30 epistemic beliefs about how these individual differences might influence the interpretations of the connections between online planning and epistemic beliefs. Table 1 Learner Profiles Learner Age Major Year (to enter) in program Verbal ability (out of 24) Prior knowledge Self-reported (out of 5) General epistemic beliefs CFI SEQ 5.70 5.21 5.63 3.73 3.73 5.16 4.68 4.14 Actual (out of 11) 5 5 4 1 H1 H2 L2 L1 23 Actuarial Senior science Senior Zoology English Senior Journalism Senior 21 20 21 10 15 13 12 3 3 3 2 Online Planning Processes This section presents the findings of the first research question – what planning processes undergraduate learners develop in their online searching and learning for addressing the given ill-structured problem. All learners developed three primary levels of plans in their online planning – global plans, sub-plans, and immediate specific search plans. Within each of the three primary levels of plans, secondary and tertiary types of plans were further identified. Global Plan Two types of global plans were identified across the four learners: (a) identifying or addressing general learning goals and (b) developing a stance on the topic. 31 Identifying or Addressing General Learning Goals. This type of global plan includes plans or decisions related to the general goals or purposes of learning in the context of a particular task. It entails the following situations: when a learner (a) determines, analyzes, or refines the goals or purposes of the task, (b) reminds himself/herself of the goals or purposes, or (c) proposes measures to address the goals or purposes. For example, seeing the statement that alternatives to genetically modified foods should remain available, Learner H1 explained how he/she would use the information to address the task, “Considering the report that I want, there might be some people that might be against it at the, at my university, so I’d rather have that information out there for them to also be able to choose.” Another example is about how Learner L2 analyzed the goal of the task (defending perspectives at a town hall meeting) and how to address it, “That’s really good to know that foods from genetically engineered plants are safe to eat, but they don’t really go into a lot of explaining that… if I’m supposed to be defending this in a sort of town hall, I might want a little more information than that. So maybe I’ll research that later to like… see the credible evidence.” Developing a Stance on the Topic. This type of global plan refers to plans or decisions related to forming or changing one’s own opinion on the learning topic for solving the problem. For example, Learner L2 articulated a plan about developing his/her own opinion on the topic: “Maybe I’ll look up also later the biggest viewpoints surrounding GMOs, so I can kind of know what people like and don’t like, or are skeptical about, so I can kind of know which angles to take since it seems like GMOs are like a relatively good thing.” A second example is about Learner H2 32 talking about which stance he/she planned to take to address the task, “I’m trying to see if there’s anything that would make me want to say ‘no,’ this would be like an actual risk to students or whoever is eating in our cafeteria, so no, you don’t want it. But a lot of it is environmental, which obviously is not a good thing. But then if there’s no direct physical risk to students and there’s only the environmental risk, then I think that’s definitely more of a personal decision.” Sub-Plan Sub-plans are plans about what particular aspects to explore within the larger goals of the task. Learners developed a number of sub-plans both during the course of online searching and in the post-task interview. Two secondary types of sub-plans were identified across the four learners: (a) Web-directed sub-plan and (b) learner-directed sub-plan. Within each secondary type, tertiary types of sub-plans were further identified. These sub-plans are illustrated in detail below. Web-Directed Sub-Plan. Web-directed sub-plans are defined as plans or decisions related to particular aspects of the topic within the larger task and primarily influenced by information learners encounter on the Web. Under Web-directed sub-plan, two sub-types of plans were identified: Web-directed immediate sub-plans and Web-directed deferred/reflective sub-plans. Web-Directed Immediate Sub-Plan. This type of sub-plan occurs when a learner (a) plans or decides to shift to a new or different aspect of the topic by clicking a link or (b) makes a new search using the exact information encountered at the moment. For example, Learner L1 decided to shift his/her exploration from the definition of genetically modified foods to “what foods are genetically modified” by clicking the “people also ask” dropdown menu provided by Google 33 search. The formation of this sub-plan was guided by the “people also ask” questions presented on the Web. Another example is that Learner H1 encountered a referenced article about a review of the health risks of genetically modified foods and then searched the title of the article for further exploration. In this example, Learner H1 generated a new search about a different aspect of the topic, which was directed by the exact information encountered on the Web. It is worth noting that it does not count as a Web-directed immediate sub-plan if a learner shifts to a new or different aspect of the topic simply by following a linear reading path of a webpage (e.g., reading a report section by section, skipping a paragraph or section to read further about the next paragraph or section within the same webpage). Instead, these are examples of “controlling reading depth or pace” – a type of immediate specific search plan, which is demonstrated in the immediate specific search plan section. Web-Directed Deferred/Reflective Sub-Plan. This type of sub-plans occurs when a learner (a) forms a sub-plan based on the exact information encountered on the Web but decides to carry out the plan at a later stage of learning, or (b) decides to revisit a previously formed sub-plan created based on the exact information encountered on the Web. For example, Learner H1 formed a new search plan about food labeling based on the exact information on the Web: “This article talks about ‘[could] the future of good labeling be changing for good?’ I’m just gonna write a note to go back on food labeling… I’ll go back and look for that information.” At a later stage of the online learning, Learner H1 decided to go back to this previously formed Web-directed plan: “The 34 first thing I’ll do is go back to labeling and just search on Google about labeling because that’s one of the first things I said I’ll go back to look at.” Learner-Directed Sub-Plan. Learner-directed sub-plans are defined as plans or decisions related to particular aspects of the topic within the larger task and are essentially guided by learners rather than information encountered on the Web. Within learner-directed sub-plan, two sub-types of plans were identified: learner-directed top-down sub-plans and learner-directed reciprocally adaptive sub-plans. Learner-Directed Top-Down Sub-Plan. The creation of this type of sub-plans is primarily based on learners’ prior knowledge or understanding of the task, that is, top-down learner guidance. For example, Learner H1 generated a search plan about what aspect of the topic to explore based on his/her prior knowledge: “So I think one of the last searches I’d like to do is look for GMOs and the World Health Organization because they usually give out public opinions about anything pertaining to human and animal health.” Learner-Directed Reciprocally Adaptive Sub-Plan. In creating this type of sub-plans, learners allow themselves to employ information they encounter on the Web (bottom-up Web influence) to make connections to their prior knowledge or understanding of the task and then creatively generate their own search plans based on the synthetic connections they have made (top-down learner guidance). Through this reciprocal bottom-up and top-down process, learners can generate creative, context-adaptive search plans that may lead to new understanding or discoveries of the topic. This type of plan differs from Web-directed sub-plans in that it 35 synthesizes the information on the Web and learners’ knowledge or understanding of the topic rather than simply follow the exact information presented on the Web. For example, after reading the economic and environmental benefits of genetically modified crops/foods, Learner L2 made a plan to explore if there are any nutritional benefits to the human body: “So the first thing that caught my attention was cheaper. Second, which literally falls on the cheaper, is that you could use a single rapidly decaying herbicide. Okay still wanna see anything about the benefits to the human body in terms of what nutrients are they providing that are better or are they not providing any better nutrients.” Another example is that after viewing a website about how GMOs are assessed in the U.K., Learner L2 formed a plan of exploring how GMOs are assessed in the U.S. to address the context of the task: “It says UK down here, so this might be, not be how America says about GMOs… the next website I go to, I will probably look for like a US authority on how they assess GMOs.” Immediate Specific Search Plan This type of plan refers to specific plans about how to search for information and construct knowledge or how to fulfill higher levels of plans within the given time limit. Within this level of planning, seven types of immediate specific search plans were identified, which are presented below in detail. The first five types were identified across the four learners; the sixth type was identified among three learners (Learners H1, H2, L2); the last type was identified only within one learner (Learner H2). 36 Generating Search Terms or Defining Search Range. It refers to plans about what search engine or search term to use or what type of information sources to explore. For example, Learner H1 made a plan to search for scholarly articles via Google Scholar: “So I’ll go back to my search (genetically modified food in general), but instead of using just regular Google, I’ll go to Scholar Google and try to find any articles.” Developing Exploration Paths of Search Results. This type of plan occurs when learners develop the paths of exploring search results, e.g., which search results to explore, in what order, or what search results not to explore. For instance, after skimming a list of search results for genetically modified foods in the U.S., Learner L2 purposefully selected a website to explore: “The first thing is a little Time article called ‘genetically modified foods: what is grown and eaten in the U.S.’ That might be interesting, but I would like to see something from like FDA or like something like that. So I’m scrolling down. There is some Wikipedia pages. There is something [down below the Wikipedia page] that says the U.S. regulation of genetically modified crops. [scrolling further down] There is also another one called ‘consumer info about food from GM plants.’ That one is from the FDA. So I might read that one. It’s a government administration.” Evaluating Information Helpfulness, Currency, or Credibility. This type of plan refers to plans or decisions related to evaluating the helpfulness, currency, or credibility of the information. For example, Learner H1 planned to evaluate the helpfulness of the information encountered and explained that “I’m going to read on and see if this is important and how it factors into my presentation, my report, and if it’s necessary for me to look further later.” Learner L2 37 evaluated the credibility of encountered information: “That quote seems to be just from an interview with the researcher, and not a result of actual research but that still seems credible…he is a plant molecular biologist, so it seems like he is pretty connected with what this issue actually is.” It is worth noting that it does not count as a plan of this type if a learner simply states that the encountered information is credible/interesting/helpful or simply mentions the authors or publication dates of the information without making any justification for their evaluation plans or decisions. Controlling Reading Depth or Pace. It refers to plans or decisions about controlling the depth or pace of reading. For example, Learner L2 decided not to continue reading a webpage, saying “that might just be more of the last paragraph that I read, um very wordy, the consultation process things like that. So I might not go into that yet especially cuz I have other things I wanted to research.” Another example is that Learner L2 made a plan about the pace of reading: “I think I’m just kind of gonna look through the article super quick for the history part.” Taking Notes. The four learners made specific plans or decisions related to when to take notes and what notes to take. Learner H1 took handwritten notes, and the other three learners took online notes in Google Docs. The notes included (a) verbatim quotes or summaries of information encountered for facilitating understanding of the topic or for directly addressing the report and (b) aspects the learners planned to further explore at a later stage of the online learning. Reflecting on Plan Execution. This type of plan includes plans or decisions about monitoring or evaluating the execution of plans. For example, Learner H1 decided to check how 38 well the plans were executed after completing a search of a particular aspect of the topic: “So that helped me learn a lot about some of the challenges in commercial agriculture, solutions that [are provided by] GM foods and crops, problems and controversies. So I'm gonna check that off, antibiotic resistance. I'm checking my notes to see how much I followed up on, and what to follow up on next.” Comparing Information From Different Sources. This type of plan entails plans or decisions about comparing the information presented from different sources. For example, Learner H2 made a plan for an explicit purpose of comparing different information sources: “I think I’m gonna go back to the other page (a Purdue webpage) and then maybe come back to this one (a Harvard webpage) and keep cross comparing if they, to see what they are both saying.” Online Planning and Epistemic Beliefs This section presents the findings of the second research question – how learners’ planning processes relate to their epistemic beliefs. Learners’ overall planning trajectories, global plans, sub-plans, and immediate specific search plans were analyzed in connection with their epistemic beliefs. Planning Trajectories and Epistemic Beliefs Based on the qualitative analysis of online searching and learning plans, visual representations were created to demonstrate the four learners’ planning trajectories, that is, how their online planning paths progressed as a function of time (see Figures 1-4). The figures depict 39 the different types of plans developed in the first half learning session, the second half learning session, and the post-task interview section. It is worth noting that the four visuals were designed to depict a general picture of online planning trajectories. They are simplified abstract sequential representations of different levels of plans along the dimension of time, and thus do not reflect the actual complexity of having multiple codes assigned to the same text unit or the actual time intervals between plans. In the four figures, squares represent global plans, circles denote sub-plans, and triangles are immediate specific search plans. Specifically, a square with the letter G inside denotes a global plan of developing or addressing general learning goals; a square with the letter S inside stands for a global plan of developing a stance on the topic; a circle with W1 inside represents a Web-directed immediate sub-plan; a circle with W2 inside is for a Web-directed deferred/reflective sub-plan; a circle with L1 inside denotes for a learner-directed top-down sub-plan; a circle with L2 inside means a learner-directed reciprocally adaptive sub-plan; the number inside a triangle represents the number of consecutive immediate specific search plans (e.g., a triangle with the number 3 inside means three consecutive immediate specific search plans). The next paragraph explicates an exemplary planning trajectory (Figure 1), followed by a summary of the four trajectories. More detailed comparisons of the plans are presented in the remaining sections related to online planning and epistemic beliefs. In Figure 1, when looking closely at the first several plans created by Learner H1, we can see that Learner H1 started with a global plan (analyzing and identifying the goals of the task), 40 generated a specific search plan (planning to use Google search), developed a learner-directed top-down sub-plan (deciding to search for general information on genetically modified food), and then used several immediate specific search plans to fulfill this sub-plan (generating a search term, developing a specific path of exploring the search results, controlling depth of reading). When screening only the sub-plans in the first learning session, we can see that Learner H1 developed relatively a large number of sub-plans, including 11 web-directed sub-plans and five learner-directed sub-plans. This reflects that at the initial stage of learning when his/her understanding of the topic was limited, Learner H1 allowed the Web to guide him/her to different aspects of the topic, but still actively engaged in selecting which aspects to explore (Web-directed sub-plans) and synthesizing knowledge from different sources to generate new aspects of exploration (learner-directed sub-plans). Learner H1 even developed a go-back-to list to keep track of all his/her sub-plans. When comparing the sub-plans of the two learning sessions, we can see that although Learner H1 developed fewer Web-directed sub-plans in the second session, he/she created a balanced number of learner-directed sub-plans across the two sessions (five for each session). It indicates that Learner H1 exerted constant effort to take active control of his/her searching and learning. The four figures show that all the four learners started with a global plan or multiple global plans, developed learner-directed sub-plans to search the Web for information, allowed the Web to guide them in forming new sub-plans (Web-directed sub-plans), and generated immediate specific search plans to explore information on the Web and fulfill higher levels of plans. In the meantime, 41 however, there were observed differences (which are presented in detail in the sections below) among the learners, especially between the two learners with expansive general epistemic beliefs (Learners H1 and H2) and the two learners with relatively reductive epistemic beliefs (Learners L1 and L2). 42 Figure 1 Planning Trajectory Created by Learner H1 Figure 2 Planning Trajectory Created by Learner H2 43 Figure 3 Planning Trajectory Created by Learner L2 Figure 4 Planning Trajectory Created by Learner L1 44 Global Plans and Epistemic Beliefs All learners attempted to identify the global learning and problem-solving goals of the task, refined or reminded themselves of the goals, and assessed what information would help address the goals. No qualitative differences in developing global plans were observed between the two contrastive groups. However, Learner L2 developed global plans in a considerably more frequent manner (see Figures 1-4) compared to the other learners. For instance, Learner L2 constantly assessed Web information in relation to his/her goals of the task or demonstrated his/her evolving understanding of the task: “My school wanted to put GMO foods in our cafeteria, so it might be helpful to know how those foods are evaluated;” “It also has a section on how genetically modified foods are labeled. This might not be super relevant to my report, but if people at this meeting have questions, then it might be good to know;” “If I’m supposed to be defending this in a sort of town hall, I might want a little more information than that.” Developing global plans enabled learners to identify learning goals that were not clearly defined in ill-structured problems, adjust learning goals based on encountered information, and avoid getting distracted by irrelevant or unhelpful information. The results showed that Learner L2, who held relatively reductive epistemic beliefs, still demonstrated complex thinking in developing global plans for addressing the given task. 45 Sub-Plans and Epistemic Beliefs The frequencies of the two types of sub-plans (Web-directed and learner-directed) throughout the two online learning sessions for each learner are presented in Table 2 (and Figures 1-4). The actual content of the sub-plans created by each learner is presented in Table 3. Table 2 Frequencies of Web-Directed and Learner-Directed Sub-Plans Learner Type of sub-plan Session 1 Session 2 Two sessions H1 Web-directed Learner-directed Web-directed & Learner-directed H2 Web-directed Learner-directed Web-directed & Learner-directed L2 Web-directed Learner-directed Web-directed & Learner-directed L1 Web-directed Learner-directed Web-directed & Learner-directed 11 5 16 4 5 9 6 5 11 7 2 9 7 5 12 0 3 3 1 0 1 1 1 2 in total 18 10 28 4 8 12 7 5 12 8 3 11 As shown in Table 2, Learner H1 developed considerably more sub-plans than Learner L2 and Learner L1 (with Learner H1 having developed 28 sub-plans, Learner L2 12 sub-plans, and Learner L1 11 sub-plans), and this is especially the case for the second learning session (12 sub-plans created by Learner H1, one sub-plan by Learner L2, and two 46 sub-plans by Learner L1). This pattern remains the same when looking at Web-directed sub-plans or learner-directed plans individually, that is, H1 developed noticeably more Web-directed sub-plans (18 Web-directed sub-plans by Learner H1, seven by Learner L2, and eight by Learner L1) as well as more learner-directed sub-plans (10 learner-directed sub-plans by Learner H1, five by Learner L2 and three by Learner L1). It indicates that compared to the two learners with relatively reductive epistemic beliefs, Learner H1 was more engaged in developing sub-plans, either Web-directed or learner-directed. Despite the less learning time, Learner H2 developed the same number of sub-plans as Learner L2 (12 sub-plans) and slightly more than Learner L1 (11 sub-plans). It is reasonable to project that Learner H2 would develop more sub-plans than Learners L1 and L2 if he/she invested the full time allowed in online learning. Besides, Learner H2 developed more learner-directed sub-plans than Learner L1 and Learner L2 (eight created by Learner H2, five by Learner L2, and three by Learner L1). When examining learner-directed sub-plans between the two learning sessions, Learner H1 generated a balanced number of learner-directed sub-plans (five for each session); Learner H2 developed slightly fewer in the second session (five in the first session and three in the second session); Learner L2 developed five learner-directed sub-plans in the first session but none in the second session; Learner L1 created two in the first session and only one in the second session (see Table 2). The results suggest that Learners H1 and H2 tended to exert more constant effort to take active control of their 47 learning. In contrast, Learner L2 made active efforts to guide his/her online learning only in the first session but became noticeably more passive in the second session (e.g., spending 38 minutes reading an interview transcript almost section by section). Learner L1 demonstrated more passive learning in both learning sessions compared to the other learners. Even though a lower level of prior knowledge might contribute to Learner L1’s relatively passive learning (especially at the initial stage of learning), it is worth noting that this passive tendency became even more prominent as Learner L1 gained more knowledge on the topic (only one learner-directed sub-plan and one Web-directed sub-plan were developed for the second session). When looking specifically at learner-directed reciprocally adaptive sub-plans, it was found that Learners H1 and H2 tended to develop more plans of this type (five created by Learner H1 and seven by Learner H2) than Learners L1 and L2 (two developed by each learner). In the post-task interview session, learners were asked what information they would like to further explore if they had more time for online searching and learning. Learners H1 and H2 generated more sub-plans about what they planned to explore for future learning than Learners L1 and L2 (see Figures 1-4 for the numbers of future learning sub-plans and Table 3 for the actual content of those sub-plans). When examining the content of those plans, it was found that the sub-plans developed by Learners H1 and H2 were more in-depth and more situation-adaptive: (e.g., how other universities introduce 48 GM foods ad responses of their students, faculty, and staff members; ways of making university students, faculty, and staff members aware of GMO facts; criteria FDA, USDA, and EPA use for approving or rejecting GM products, regulations on particular products). In contrast, the sub-plans generated by Learners L1 and L2 were more general (e.g., a layman’s perspective on GMOs and differences between GMOs and non-GMOs). To summarize this section, compared to Learners L1 and L2 (who reported relatively reductive general epistemic beliefs), Learners H1 and H2 (who held expansive epistemic beliefs) tended to develop more sub-plans, more learner-directed sub-plans, and more learner-directed reciprocally adaptive sub-plans, exerted more constant effort to take active control of their online learning, and generated more in-depth and more context-adaptive sub-plans for future exploration of the topic. 49 Table 3 Sub-Plans Created by the Four Learners Sub-plans general information on GM food arguments for or against GMOs food labeling consumer attitudes and purchase intentions benefits of GM food benefits of GM food to human body antibiotic resistance marker L2 W2 W2 L1 L2 Session 1 1 2 3 4 5 6 7 W2 GMOs and labeling W1 50 Learner H1 Learner H2 Type L1 Sub-plans what is a GMO Learner L2 Sub-plans GM food in general Type L1 Type L1 Learner L1 Sub-plans genetically modified foods in general common genetically modified foods benefits of genetically modified foods Type L1 W1 W1 why do we use GMOs do GMOs harm health W1 GM food assessment L2 in the U.S. W1 GM food assessment L2 in the U.S. when we started relying on GMOs L2 how long GMOs have been on the market regulation of GMOs W1 L2 history of GM crops W2 GMO shopping tips W1 evidence for FDA’s statement about GM food safety FDA regulations on GM food (not fulfilled) different viewpoints on GMOs W2 W2 L1 by Whole Foods Market W2 benefits of genetically modified foods non-GMO project W1 USDA list of bioengineered foods W1 Table 3 (cont’d) 8 9 10 11 12 13 14 15 concerns about food safety W2 risks of GMOs anti-GM crops campaign W2 global attitude towards GMOs L2 L2 evidence for FDA’s statement about GM food safety common concerns on GMOs food labeling W2 L2 W2 W2 W1 W1 GM foods labeling requirements in the U.S. antibiotic resistance marker gut bacteria antibiotic resistance health risks of GM foods challenges and impacts in commercial agriculture resistance to golden rice - a GM crop (not fulfilled) arguments against GMOs 51 W2 USDA Bioengineered Foods information additional information on GMO from USDA W1 W2 L1 W1 L2 Table 3 (cont’d) 16 Session 2 1 bans on GM foods/crops Sub-plans anti-GM crops campaign W2 Type W2 Sub-plans global outlook on GMOs Type L2 Sub-plans Type history of GM crops W2 Sub-plans are genetically modified foods cheaper are GM foods more nutritious Type W1 L2 anti-GM campaign organization Greenpeace anti-GM campaign organization who leads the anti-GM campaigns “natural is always better” argument “GMOs allow for more profits” argument (not fulfilled) GMO bans in the U.S. W2 W2 L1 W2 W2 L2 2 3 4 5 6 7 reasons why other countries allow or don’t allow GMOs actual effects of GMOs L2 L2 52 Table 3 (cont’d) 8 9 10 11 12 Interview session 1 GMO bans in the world opinions of World Health Organization on GMOs WHO general information on food safety the “natural is always better” argument nutritional benefits and risks of GM foods Sub-plans role of large agricultural corporations and how they benefit from the GM food industry L2 L1 W1 W2 L1 Type Sub-plans Type Sub-plans Type Sub-plans Type L2 criteria FDA, USDA, and EPA use for approving or rejecting products on the market L2 a meta-analysis study of 20 years data of GMOs W2 differences between GMO and non-GMO L1 53 L1 L1 Table 3 (cont’d) 2 3 4 how other universities introduce GM foods and responses of their students, faculty, and staff members ways of making university students, faculty, and staff members aware of GMO facts specific regulations on particular products L1 specific impacts on human body L1 If research matches up with how the regulations are being formulated L1 a layman’s perspective on GMO L2 Note. In the type columns, L1 denotes learner-directed top-down sub-plan; L2 represents learner-directed reciprocally adaptive sub-plan; W1 is for Web-directed immediate sub-plan; W2 stands for Web-directed deferred/reflective sub-plan. 54 Immediate Specific Search Plans and Epistemic Beliefs The frequencies and proportions of the seven types of immediate specific search plans for each learner are presented in Table 4. Considering that Learner H2 spent less time on learning, comparisons of the frequencies of plans were made primarily among Learners L1, L2, and H1, or took learning time into consideration when making comparisons among the four learners. Comparisons of the proportions or qualitative natures of plans were conducted among all the four learners. The proportions of plans within each learner imply which types of specific plans the learner tended to focus on and which types of plans the learner paid less attention to. Learner H1 paid the most share of attention (32.91%) to plans about controlling reading depth/pace and no explicit attention to plans of information sources comparison. Learner H2 was the only learner who demonstrated the use of all the seven types of plans and paid relatively balanced attention to different plans (with proportions of plans ranging from 4.88% to 26.83%). Learner L2 paid predominant attention to plans of evaluating information, which took a proportion of 57.4%, and little or no explicit attention to the execution of plans and comparison of information sources. Learner L1 paid predominant attention to plans of taking notes, which took a proportion of 69.0%, and little or no explicit attention to plans related to information evaluation, plan execution, and information sources comparison. In summary, when compared to Learners L1 and L2, Learners H1 and H2 demonstrated no predominant reliance on particular types of specific search plans; instead, 55 they paid relatively more balanced attention to the development of different types of specific search plans. The remainder of this section presents in detail the characteristics of the specific search plans developed by each learner (in comparison with other learners). 56 Table 4 Frequencies and Proportions of Immediate Specific Search Plans Immediate specific search plans Generating search terms or defining search range Developing exploration paths of search results Evaluating information helpfulness, currency or credibility Comparing information from different sources Controlling reading depth or pace Reflecting on plan execution Taking notes Total Learner H1 Learner H2 Learner L2 Learner L1 Frequency Proportion Frequency Proportion Frequency Proportion Frequency Proportion 17 11 7 0 26 3 15 79 21.52% 13.92% 8.86% 0 32.91% 3.80% 18.99% 100% 5 9 4 2 8 2 11 41 12.20% 21.95% 7 5 13.21% 9.43% 9.76% 27 50.94% 0 6 1 7 53 0 11.32% 1.89% 13.21% 100% 4.88% 19.51% 4.88% 26.83% 100% 57 5 8 1 0 4 0 40 58 8.62% 13.79% 1.72% 0 6.90% 0 68.97% 100% Learner H1 made considerably more plans about what search terms to use or what types of information to explore (e.g., news articles, academic studies, forums) compared to Learners L1 and L2 (and Learner H2 when controlling learning time). It indicates that Learner H1 was more active in initiating searches. As a result, Learner H1 was more engaged in making plans for developing exploration paths of search results. In addition, Learner H1 made noticeably more plans about how deep to dig into a particular aspect of the topic or a particular website and what reading pace to take, whereas Learners L1 and L2 were more passive in following the information presented on the Web (e.g., Learner L2 read an interview transcript section by section for about 38 minutes without providing explicit justifications for the decision, and in several occasions, Learner L1 decided to leave a website only when reaching the end of the webpage). Another point worth noting is that Learner H1 developed a go-back-to list of the particular aspects that he/she planned to explore further at a later stage of learning, and crossed out a search plan in the list after completing that plan. This explains why Learner H1 developed relatively more plans than the other three learners in terms of reflecting on plan execution. Despite the less learning time, Learner H2 developed a wider range of specific search plans than the other learners. Learner H2 was the only learner who made explicit plans about comparing information from different sources (e.g., “I think I’m gonna go back to the other page (a Purdue webpage) and then maybe come back to this one (a Harvard webpage) and keep cross comparing if they, to see what they are both saying”). 58 Given that Learner H2 already developed relatively more or a comparable number of plans about search term generation, information evaluation, and reading depth/pace control compared to Learners L1 and L2, it is projected that this active plan-developing pattern would be more noticeable if Learner H2 used the full learning time allotted. One major feature of Learner L2’s specific plan development is that he/she made considerably more plans or decisions about evaluating the credibility, currency, and helpfulness of information compared to Learners H1 and L1 (and Learner H2 when controlling learning time). For instance, Learner L2 evaluated the credibility of encountered information with reasonably healthy critical thinking: “That quote seems to be just from an interview with the researcher, and not a result of actual research but that still seems credible…he is a plant molecular biologist, so it seems like he is pretty connected with what this issue actually is.” It indicates that Learner L2, who held relatively reductive general epistemic beliefs, demonstrated rather complex thinking in critically evaluating information. Learner L1 developed five of the seven types of specific search plans. He/she made no explicit plans of comparing different information sources and reflecting on plan execution, and only one plan about evaluating the credibility of the encountered information. Learner L1 made 40 note-taking plans, far exceeding the numbers of note-taking plans made by Learners H1 and L2 (and Learner H2 when controlling learning time) because he/she frequently paused his/her online searching and learning to take notes. 59 All the notes taken by Learner L1 were about information he/she considered as facts or information he/she intended to use in the final report; whereas the notes taken by Learners H1 and L2 also included what aspects of the topic they planned to research at a later stage of their online learning. To summarize the results of this section, compared to learners holding relatively reductive epistemic beliefs (Learners L1 and L2), learners with expansive epistemic beliefs (Learners H1 and H2) tended to demonstrate more complex use of immediate specific search plans. It is worth noting that learners holding relatively reductive epistemic beliefs demonstrated complex thinking in some aspect(s) of specific plan development (e.g., Learner L2’ critical thinking in evaluating information), but the overall results showed that learners holding expansive epistemic beliefs demonstrated more complex thinking in a wider range of aspects involved in specific plan development (e.g., more balanced use of different types of specific search plans rather than predominant reliance on particular plans, use of a wider range of plans, relatively more active engagement in initiating searches, developing exploration paths, controlling reading depth/pace, and reflecting on plan execution). Online Planning and Actual Search Moves This section addresses the third research question (how learners’ planning processes connect to their actual search moves) by presenting learners’ sub-plan execution, use of search terms, and information sources explored. 60 Sub-Plan Execution By definition, a Web-directed immediate sub-plan is fulfilled immediately after forming the plan either by clicking a readily available link or starting a new search based on the exact information encountered. Execution of the other three types of sub-plans (Web-directed deferred/reflective sub-plans, learner-directed top-down sub-plans, and learner-directed reciprocally adaptive sub-plans) was examined. Learner H1 developed a number of deferred sub-plans, and only one deferred sub-plan about exploring whether genetically modified foods allow for more profits, which was at the bottom of his/her go-back-to list, was not fulfilled. As for Learner L2, two of his/her four deferred plans remained unexplored: (a) exploring the regulations of FDA on genetically modified foods and (b) looking into a golden rice (a genetically modified crop) project. Learners H2 and L1 did not develop any deferred sub-plans, and they fulfilled all their sub-plans. The results showed that while Web-directed deferred/reflective sub-plans enabled learners to focus on fulfilling current sub-plans or sub-plans of higher priority, a subsequent challenge for learners was to keep track of their deferred sub-plans and determine whether and when to revisit them. Learner H1’s use of an evolving list of sub-plans, a go-back-to-list as the learner named it, seemed helpful in addressing this challenge. Use of Search Terms Learners generated different search terms (see Table 5) to explore information on the Web to fulfill their sub-plans, which ultimately served for addressing their global plans. 61 Learners’ use of search terms was found to be related to all the four types of sub-plans. Out of all the search terms, only two were used to fulfill Web-directed immediate sub-plans (e.g., Learner H1 searched “health risks of genetically modified foods” on the website of Taylors and Francies Online for an article referenced on a Website that he/she just encountered), and the others were generated to accomplish the other three types of sub-plans. The remaining paragraphs in this section present the characteristics of the search terms generated by each learner, followed by a summary of this section. Learner H1 initiated 12 searches (five searches for the first learning session and seven searches for the second session), covering a relatively wide range of aspects about the topic (e.g., general information about genetically modified foods, production process, general benefits, nutritional value, health risks, labeling requirements in the U.S., bans in the U.S. and in the world, anti-GM campaign, etc.). Learner H2 generated five searches (two in the first session and three in the second session) for exploring general information about GMOs, the history of GM crops, global outlook on GMOs, reasons for GMO bans, and potential risks of GMOs. It is projected that Learner H2 would develop more search terms if he/she used the full allotted learning time, especially considering that he/she developed more plans in the post-task interview session than the other three learners. 62 Learner L2 developed four searches in the first session and only one search in the second session to look into general information about GMO foods, GM foods in the U.S., evidence for the safety of GMOs, arguments against GMOs, and the history of GM plants. Learner L1 conducted four searches (two in each session). The first two searches were similar, with the first one being “GM foods” in Google search and the second one being “GMOs” within the website of the U.S. Department of Agriculture (USDA). When conducting the third search, Learner L1 did not provide a plan about what to search, even after being asked to explain the plan. Instead, he/she tended to base the decision of what to search for on the suggested searched terms listed by Google search: Learner L1 entered the words “are genetically modified foods” in the search engine, skimmed the suggested search terms provided by Google (e.g., are genetically modified foods scary, are genetically modified foods, are genetically modified foods safe), and decided to select the term “are genetically modified foods cheaper.” This is the only instance identified in this study where a learner did not provide a clear plan about what to search for before conducting a search. Learner L1’s fourth search was about whether genetically modified foods are more nutritious. The results showed that learners generated search terms to search for information for fulfilling all the four types of sub-plans. Therefore, it was not surprising to find that when holding time constant, the learners who developed more sub-plans (Learners H1 and 63 H2) tended to generate more search terms. Compared to the other three learners, Learner L1 demonstrated more reliance on the Web in generating search terms. Table 5 Search Terms Generated by the Four Learners Session Learner H1 Learner H2 Learner L2 Learner L1 1 1 2 3 4 5 genetically modified foods GMOs (within the US Department of Agriculture website) genetically modified foods genetically modified foods benefits what is a gmo how long have gmo crops been used genetically modified food genetically modified food usa credible evidence on GMO’s safety to eat Arguments against GMOs is labeling of gm foods required in the us antibiotic resistance marker in gm crops health risks of genetically modified foods (within the website of Taylors and Francis Online) Session Learner H1 Learner H2 Learner L2 Learner L1 2 1 2 3 anti gm campaign gmos around the world head of anti gm campaign reasons for gmo bans history of genetic modification in plants gmo bans in the us potential risks of gmos are genetically modified foods cheaper are genetically modified foods more nutritious 64 Table 5 (cont’d) 4 5 6 7 gmo bans in the world gmo and world health organization Natural is better argument against gm crops gmo nutritional value Sources of Information Explored This section examines how learners’ planning processes connect to the information sources they have explored. Table 6 summarizes the different types of information sources explored by the four learners, the numbers of webpages explored within each type, and the associated sample webpages. Table 6 Information Sources Explored by the Four Learners Types of sources Research articles Learner H1 Learner H2 Learner L2 Learner L1 0 0 1 (International Scholarly Research Notices) 3 (British Medical Bulletin, Critical Reviews in Food Science and Nutrition, GM Crops & Food 65 Table 6 (cont’d) Webpages of governmental institutions or institutions in the UN Webpages of educational institutions News articles Webpages for promoting understanding and discussions of GMOs Wikipedia Webpages of business companies or one-sided websites of non-profit organizations Others Total 2 (Library of Congress, World Health Organization) 6 WebMD, Detroit News, New York Times, The Ecologist, Forbes 3 (biofortified, Genetic literacy project, GMO answers) 2 (Greenpeace international GMO-free Regions) 5 Personal blogs, petition, images 21 0 7 (Purdue and Harvard webpages) 1 (The Nation Magazine) 2 (Live Science & Genetic Literacy Project) 11 66 2 2 (FSA in the UK, FDA in the US) (USDA webpages) 2 (Cornell and Harvard webpages) 1 (New York Times) 5 0 3 (Forbes, Medical News Today, New York Times) 1 (GMO Answers) 1 4 (Whole Foods Market, Non-GMO project, Livestrong, BIO) 11 With respect to the number of sources explored, Learners H1 and H2 tended to explore more information sources compared to Learners L1 and L2, when controlling learning time. When examining closely the sub-plans and explored sources of information of Learners H1 and H2, more subtleties emerged. Learner H1 researched more sources for the purpose of fulfilling a relatively large number of sub-plans; whereas Learner H2 developed a moderate number of sub-plans and tended to explore more sources for each sub-plan. Learner H2 explained why he/she attempted to explore multiple sources for each question (sub-plan), “It’s not always easy to know if a source is biased or not, but I guess that’s why you look at a lot of different sources so that you can kind of see what the general overall opinions are and then you kind of know that everything else that’s lying on the outskirts might be information that’s, you know, might be a little bit less well-founded or something that is a little bit more based on whatever the persuasion is of the person who is writing the article. ” Learner L2 visited the least number of information sources. In terms of types of sources explored, news articles were the only type of information source explored by all learners; webpages of governmental institutions (or institutions in the UN) and webpages for promoting understanding and discussions of GMOs were explored by three learners (although not the same three learners). Learners H1 and H2 were more willing to invest effort in exploring research papers compared to the other two learners. Learners H1 and L1 both visited webpages of business companies or websites that clearly held one-sided opinions on the topic. It is worth noting, however, 67 Learner H1 visited these webpages when purposefully searching for information about anti-GM campaigns and GMO bans, whereas Learner L1 visited these websites for general information and did not make critical evaluations of their credibility. For instance, when asked about the reason for exploring the Whole Foods Market website, Learner L1 responded, “I was just looking at the benefits that are on Google, but then I saw Whole Foods and I, like I said I trust Whole Foods, I’m just, that’s looks like a good place to go.” Learner L1 was the only learner who visited Wikipedia. Overall, compared to Learner L1, the other three learners visited information sources with relatively higher credibility. The results showed that the two learners who demonstrated more complexity in developing plans (Learners H1 and H2) tended to visit more information sources and were more willing to invest effort in exploring research articles. The learner who made extensive plans/decisions about evaluating the credibility of information (Learner L2) primarily focused on exploring a few webpages with relatively high credibility (websites of governmental and educational institutions). The learner who made only one plan/decision on evaluating the credibility of information (Learner L1) tended to explore more websites that presented one-sided opinions (e.g., Whole Foods Market, Non-GMO Projects). Online Planning, Epistemic Beliefs, and Problem-Solving Performance This section presents the findings of the second ancillary research question – how learners’ planning processes and epistemic beliefs shape their problem-solving performance. Learner’s performance was measured by their written responses to the task, 68 which were graded based on the procedures presented in the data analysis section. The performance scores for Learner H1, H2, L2, and L1 were 23, 26, 17.5, and 11, respectively. The major qualitative characteristics of each learner’s written response are presented below in detail. Learner H1’s response presented both opposing and proposing viewpoints on the use of genetically modified foods. Multiple stakeholders (e.g., environmental organizations, organic farming organizations, and consumers) involved in the issue were taken into account. Relatively reliable sources of information (the World Health Organization and two journal articles) were cited to support the statements. However, the response demonstrated a low level of synthetic analysis of the different viewpoints. The proposed solution was somewhat context-adaptive, showed an understanding of different perspectives, and provided some measures to implement the solution: “If we decide to introduce GMO foods into our university’s food system, this should be done carefully and should not eliminate the non-GMO options for individuals opposed to GM foods. Careful labeling of GM foods should be maintained at all times and continued research on the regulation of GMO foods should be maintained.” The response did not provide further details about how to implement the proposed measures or evaluate the feasibility of the measures. The response examined one possible consequence of this solution (backlash from anti-GM movements). 69 Learner H2’s response was concise and took into consideration different perspectives and multiple stakeholders (e.g., regulatory organizations, Food Services, students, food production companies). The response made good synthetic connections of ideas. Information was presented clearly and accurately (e.g., accurate demonstration of the three regulatory organizations). Specific context-adaptive measures were proposed to address the task: “Information should be available to Food Services customers if they have questions or concerns, and the staff should be well educated on the scientific and ethical issues that come along with GMOs.” The proposed solution showed an understanding of different perspectives: “We do not see an immediate need to eliminate the cost-effective use of GMO foods in our facilities. However, because it is a new technology, we respect skepticism from our customer base and want to maintain transparency about the content of our food, which is why labeling and circulation of information will be a priority…we will leave the final decision of consumption up to our clientele.” Similar to Learner H1’s response, the feasibility of the proposed solution/measures was not evaluated. Learner L2’s response presented multiple perspectives on the topic, involved multiple stakeholders (e.g., farmers, students, the university, and FDA), and nicely synthesized possible benefits and risks of GM foods. However, the response misrepresented some factual information. For instance, he/she wrote in the report that “As of 2016, all foods sold that are made with GMOs must be clearly marked accordingly.” In fact, as stated by the USDA (the United States Department of Agriculture), a national 70 bioengineered food disclosure law was passed in 2016 to direct USDA to establish the mandatory disclosure standard, and the standard will only be fully implemented by 2022. Some statements simplified the complexity of the topic and were not well supported with evidence. For instance, Learner L2 stated that “while there is some disagreement, most scientific data supports a stance that GMOs are healthier for the environment and they may be healthier for people, too.” The cited evidence was a quote from the New York Times saying that “about 90% of scientists believe that GMOs are safe,” which did not provide direct support for Learner L2’s statement. While the response recognized different perspectives, it tended to take the side of one perspective: “there is no reason not to support the use of GMOs in our school’s cafeterias and dining halls.” The response did not address how to introduce GM foods in the university’s cafeterias and dining halls. Learner L1’s response recognized different perspectives on the topic and presented possible benefits and risks of genetically modified foods. However, the response made the least synthesis of ideas. Statements were largely based on personal experiences/opinions: “throughout my school days at a cafeteria and my college days at a dining hall, I have always eaten whatever was served to me and I don’t see a problem with it….I don’t think there is a problem with eating GMOs, as long as you are careful and maybe aware of what you are eating…I don’t think milk is good for you at all and I believe humans should drink almond milk, coconut milk, hemp milk, oat milk or other nut milks because diary milk, which can possibly contain GMOs, is not healthy for you.” Statements presented in 71 different parts of the response were inconsistent, and the learner’s stance on the topic was unclear: “I think dining halls and school cafeterias should stick to normal food and not worry about GMO or non-GMO because GMO products are cheaper and cut down prices of production for farmers and I think farmers and certain lower-income school districts would go into crisis if they weren’t allowed to serve GMO products anymore… I think school cafeterias and dining halls should be a little concerned with GMO products and should opt out of using them if they can… I think the solution is keeping an open mind about it.” Similar to Learner L2, Learner L1 did not address the “how” question of the task – how to (or not to) introduce GM foods in the university’s cafeteria or dining halls. 72 CHAPTER 5 Discussion In this section, the findings of this study are discussed in six parts: (a) planning in online searching and learning, (b)online planning and general epistemic beliefs, (c) online planning and actual search moves, (d) online planning, epistemic beliefs, and problem-solving performance, (e) implications, and (f) limitations. Planning in Online Searching and Learning Across the four undergraduate learners, three primary levels of plans (i.e., global plans, sub-plans, and immediate specific search plans) were identified in their planning process developed during the course of online searching and learning for solving an ill-structured problem. This finding provides further empirical support for the efficacy of the three-level model of planning proposed in a preliminary study (Cheng & Spiro, 2016) with a different and more diverse group of learners. The preliminary study worked with skilled online learners of graduate students, and this study researched undergraduate students who had different epistemic beliefs. In addition, within the three primary levels of plans, secondary and tertiary categories of plans were further identified in this study. Discussions of each level of plans are presented in the sections below. Global Plans Learners developed two types of global plans: (a) plans for developing or addressing general learning goals and (b) plans for forming a stance on the topic. These 73 two types of global plans reflect the ill-structured nature of both the given problem and the activity of online searching and learning. Ill-structured problems are complex, multi-faceted, ambiguously defined, and typically situated in a specific context. This means that some aspects of an ill-structured problem are not well specified, and there may be multiple interpretations of the problem. Therefore, to solve an ill-structured problem, learners need to identify the possible causes, key issues, challenges, and constraints associated with the specific context, develop their own learning goals or objectives, and propose measures for addressing them (Jonassen, 1997). As learners gain more knowledge on the learning topic, their understanding of the goals and proposed measures may evolve accordingly. This explains why the four learners developed the first type of global plans and revisited or refined their global plans as they were searching and learning on the Web. Another explanation is that online searching and learning takes place in an open and nonlinear learning environment - the Web and that online searching and learning itself is complex and ill-structured (Leu et al., 2016). Learners thus need to develop learning goals to guide them through the nonlinear, boundless online learning environment and avoid getting distracted by so many exploration options presented by the Web (Hartman et al., 2010). Ill-structured problems have no pre-existing correct solutions, typically involve divergent perspectives and multiple stakeholders, and thus can be legitimately approached 74 from different angles (Jonassen, 1997). Therefore, to address the ill-structured problem, learners developed plans and made deliberate effort to form their own stances on the topic. Sub-Plans Within the second level of plans, two secondary categories of sub-plans were identified across the four learners: Web-directed sub-plan and learner-directed sub-plan. Web-directed sub-plans allow learners to explore different aspects of the topic under the primary guidance of the Web, especially at the initial stage of their learning when they have a low level of knowledge on the topic. However, by definition, passively following a linear reading path of a webpage without making any justifications does not count as Web-directed sub-plans because of the lack of active planning. In this sense, developing Web-directed sub-plans does not simply mean following the Web in a completely passive manner; instead, it requires active engagement in thinking about which of the encountered aspects to explore and when. Learner-directed sub-plans enable learners to take personal control of their learning and initiate searches based on their prior knowledge of the topic, understanding of the problem-solving task, and creative integration of information from different sources. Therefore, learner-directed sub-plans require more active involvement from learners compared to Web-directed sub-plans. Balanced development of these two types of sub-plans thus enables learners to explore some aspects that they would not be able to think 75 of without the guidance of the Web and to make creative syntheses of what they have learned on the Web to meet the specific needs of a problem-solving context. Under Web-directed sub-plans, two tertiary types of sub-plans were identified – Web-directed immediate sub-plan and Web-directed deferred/reflective sub-plan. The former was identified across the four learners, and the latter was identified across three learners (Learners H1, L2, and L1). Web-directed immediate sub-plans allow learners to take immediate action to explore a particular aspect of the topic that they consider helpful for addressing the problem; in the meantime, this type of sub-plans may discontinue the progress of a current sub-plan that has not been adequately fulfilled. Web-directed deferred/reflective sub-plans enable learners to focus on fulfilling current sub-plans or sub-plans of higher priority rather than immediately move forward to a different aspect of the topic; a subsequent challenge for learners is to keep track of their deferred sub-plans and determine when to revisit them or whether to refine them. Compared to Web-directed immediate sub-plans, Web-directed deferred/reflective sub-plans involve more learner control of the pace of plan execution, but both are still primarily informed by the information presented on the Web. Within learner-directed sub-plans, two tertiary types of sub-plans were identified across the four learners – learner-directed top-down sub-plans and learner-directed reciprocally adaptive sub-plans. Learner-directed top-down sub-plans are primarily informed by learners’ prior knowledge or understanding of the problem-solving task. 76 Developing learner-directed reciprocally adaptive sub-plans requires learners’ active and synthetic integration of their prior knowledge/experience, the information they have encountered from different sources, and the specific context of the problem-solving task. Fulfilling learner-directed reciprocally adaptive sub-plans thus allows learners to not only capitalize on the affordances of the Web but also develop novel and context-adaptive ideas for addressing ill-structured problems. Execution of learner-directed reciprocally adaptive sub-plans leads to LICRA (learner-initiated, complex, reciprocally adaptive) searches (DeSchryver & Spiro, 2008), which are viewed as “a core feature to successful deep Web learning in ill-structured domains (p. 143).” Immediate Specific Search Plans Results showed that five types of immediate specific search plans were commonly used by the four learners for searching and learning on the Web: (a) generating search terms or defining search range, (b) developing exploration paths, (c) evaluating information helpfulness, currency or credibility, (d) controlling reading depth or pace, and (e) taking notes. Another type of specific plans (reflecting on plan execution) was identified among three learners, and yet another type (comparing information from different sources) was identified only in the planning process of one learner. Information presented on the Web is of different degrees of credibility or helpfulness. Although all learners demonstrated critical thinking in evaluating the credibility or helpfulness of encountered information, they exerted noticeably different 77 levels of effort on this aspect, which is further discussed in the next section – online planning and epistemic beliefs. It is worth particular attention that only one learner made explicit plans about comparing different information sources. A possible explanation is that most learners were not fully aware of the interconnection among different aspects of the topic or the inherent complexities of the task, which require critical comparisons of ideas from different sources (even sources with high credibility) and synthetic connections. Online Planning and General Epistemic Beliefs Overall, the results showed that learners with expansive epistemic beliefs demonstrated more complexity in a wider range of aspects involved in plan development compared to their counterparts holding relatively reductively epistemic beliefs. The following paragraphs discuss the specific results regarding the connections between online planning and general epistemic beliefs. The two learners holding expansive epistemic beliefs developed more sub-plans than their counterparts when holding their learning time constant. This suggests that the two learners with expansive epistemic beliefs were more actively engaged in exploring different aspects or issues of the topic. This finding supports Spiro et al.’s (1996) assumption that learners holding expansive epistemic beliefs are more likely to seek out complexity and account for complex topics by examining various phenomena and multiple explanations. 78 The two learners holding expansive epistemic beliefs also developed more learner-directed sub-plans than their counterparts throughout the whole learning session and exerted more constant effort to create learner-directed sub-plans across the first and second half learning sessions. This indicates that the two learners holding expansive epistemic beliefs tended to view learning as an active process of constructing knowledge and attempted to take personal control of their learning rather than being primarily guided by others (or information encountered on the Web). In addition, the two learners holding expansive epistemic beliefs developed more learner-directed reciprocally adaptive sub-plans, which require active synthesis of information from different sources (learners’ prior knowledge, understanding of the problem-solving context, and newly learned information from multiple Web sources). This result is consistent with the characteristics of expansive and flexible epistemic beliefs depicted by Spiro et al. (1996): learners with expansive epistemic beliefs tend to assume that phenomena or elements of a phenomenon are highly interrelated, put cognitive emphasis on connections among parts, and actively construct knowledge in learning. In the post-task interview session, the two learners holding expansive epistemic beliefs developed more sub-plans for possible future explorations on the topic. This indicates that they were more willing to spend time and effort exploring additional aspects of the topic or investigating particular aspects in greater depth. This result is consistent with the quick learning dimension of epistemic beliefs proposed by Schommer (1990). 79 According to Schommer, Learners with simple/reductive epistemic beliefs tend to believe that learning is quick and that spending more time on problems that have no clear-cut and unambiguous solutions would not pay off. In contrast, learners with complex/expansive epistemic beliefs tend to believe that learning is a slow process of constructing knowledge and that working on a difficult problem for an extended period of time would promote their understanding. In addition, the future learning sub-plans created by the two learners holding expansive epistemic beliefs were more in-depth and more context-specific, which reflects their tendency of seeking out complexity and constructing knowledge in a context-adaptive manner. With respect to immediate specific search plans, one learner (Learner H2) with expansive epistemic beliefs developed a wider range of immediate specific search plans than the other three learners. Both learners holding expansive epistemic beliefs demonstrated a more balanced use of different types of immediate specific search plans, whereas the two learners holding relatively reductive epistemic beliefs predominantly relied on particular types of specific search plans. The findings imply that learners with expansive epistemic beliefs tend to demonstrate more complexity in developing immediate specific search plans. It was interesting to find that Learner L2, who reported relatively reductive epistemic beliefs, demonstrated more complexity than the other learners in developing global plans and specific search plans for information evaluation, which are two crucial 80 aspects of online searching and learning for problem solving. One possible explanation for this finding is that the measured epistemic beliefs of Learners L1 and L2 were relatively more reductive and that they were not at the extremely low end of an epistemic belief continuum. It is also possible that even students receiving extremely low overall epistemic belief scores may demonstrate complex thinking in certain aspects of online plan development. If possible, future research should be conducted to examine these findings with learners holding extremely low epistemic beliefs. Several individual difference factors that might mediate the connections between epistemic beliefs and online planning were taken into consideration: (a) prior knowledge, (b) verbal ability, and (c) time spent on the task. Learner L1 had a noticeably lower level of prior knowledge than the other three learners. As presented in the literature review section, previous findings of the role of prior knowledge in online searching and learning are inconsistent. A lower level of prior knowledge might influence Learner L1’s initial stage of online searching and learning. It is worth noting, however, learner L1 did not demonstrate more complexity in developing plans as he/she gained more knowledge on the topic. With respect to verbal ability, despite obtaining the lowest verbal ability score, Learner H1 demonstrated a high level of complexity in developing plans; although Learner H2 obtained relatively higher scores than Learners L1 and L2, all of their scores fell on a medium-high range (50.0% - 62.5% of the total score). In regard to learning time, considering that Learner H2 was not able to use the full learning time allowed, analyses of 81 and comparisons associated with Learner H2’s planning processes took into consideration the learning time difference. Online Planning and Actual Search Moves This section discusses the results of an ancillary research question: how learners’ online planning processes connect to their actual search moves in terms of plan execution, use of search terms, and information sources explored. In a complex problem-solving context that required exploration of many undefined aspects, the use of a list of evolving plans seemed helpful in facilitating plan execution, especially in fulfilling deferred sub-plans. It was not surprising to find that when controlling learning time, learners who made more sub-plans also generated more search terms to fulfill their plans. In terms of information sources explored, the four learners demonstrated different patterns. Learner H1developed considerably more sub-plans than the other three learners, and thus made wider-ranging searches and visited more sources to explore the multiple aspects reflected in the sub-plans. Different from learner L1, Learner H2 developed moderately more sub-plans than Learners L1 and L2 (when controlling their learning time) and tended to explore and compare multiple sources for each aspect indicated in the sub-plans. Learner L2 exerted noticeably more effort on evaluating the credibility and helpfulness of information than the other learners and tended to focus on exploring one or two webpages that he/she considered credible and helpful (e.g., spending about 38 minutes reading an interview transcript from an educational institution and about 20 minutes 82 reading a news article). Learner L1 generated the least number of search terms and rarely evaluated the credibility of sources explored. Consequently, although Learner L1 visited a moderate number of sources, he/she tended to make simple sequential reviews of search results or reference links and visited more sources with lower credibility. In summary, given the high complexity of the task and time constraints, the four learners developed different patterns in exploring information sources: researching more sources to fulfill a relatively large number of sub-plans, exploring multiple sources for each of a moderate number of sub-plans, focusing on a few sources with relative high credibility, or making few plans/decisions about evaluating the credibility of information and thus visiting sources with lower credibility. The effectiveness of the first three different approaches needs to be further examined in different contexts, and the fourth approach demonstrated by Learner L1 should raise more concern, as critically evaluating the information on the Web is a major component of successful online searching and learning (Bråten et al., 2009; Leu et al., 2013). Online Planning, Epistemic Beliefs, and Problem-Solving Performance This section discusses the results of the fourth research question – how learners’ online planning and epistemic beliefs shape their problem-solving performance. Learner H1 held expansive general epistemic beliefs, demonstrated a wide range of complexity in developing online searching and learning plans, and explored relatively a large number of aspects of the topic. Learner H1’s response covered multiple aspects of the 83 topic, which were supported with credible sources, but did not demonstrate a good synthetic analysis of the different aspects. Learner H2 held expansive general epistemic beliefs, demonstrated a relatively wide range of complexity in developing searching and learning plans, and tended to explore and compare different information sources for each of a moderate number of sub-plans. Learner H2’s response was concise, synthetic, and situation-adaptive. Learner L2 reported relatively reductive general epistemic beliefs, demonstrated complexity in developing global plans and specific search plans about information evaluation, and tended to focus on exploring a few sources with relatively high credibility. Learner L2’s response was synthetic and somewhat situation-adaptive; however, the response misrepresented some factual information and tended to simplify the complexity of some related issues. Learner L1 reported relatively reductive general epistemic beliefs, demonstrated a relatively low level of complexity in developing plans, and made few specific search plans about evaluating the credibility of information. Learner L1’s response was largely based on personal experience, made few synthetic connections between ideas, and presented unclear and inconsistent stances on the topic. The two learners holding expansive epistemic beliefs, compared to their counterparts, demonstrated more complexity in developing plans and in approaching the ill-structured task. Learner L2, who reported relatively reductive epistemic beliefs but 84 demonstrated complexity in some aspects of planning, exhibited complexity in synthesizing different perspectives and possible benefits and risks. Learners L1 reported relatively reductive epistemic beliefs and demonstrated a low level of complexity in developing plans and in approaching the task. These results seem to suggest that learners’ epistemic beliefs and online planning processes exerted a combined influence on their problem-solving performance. The findings of this study did not reveal how online planning and epistemic beliefs individually factored into learners’ problem-solving performance. Implications The findings of this study provide helpful implications to practitioners for their instruction of the new literacy of online searching and learning, especially online planning strategies. First, this research forms the basis for studies that would investigate how to help educators systematically integrate the three-level planning model into their instruction to facilitate students’ learning of online searching and learning strategies. Second, the findings of the complexity in plan development inform teachers of what planning knowledge to teach for online searching and learning, especially in the context of ill-structured problem solving. Students should be aware of their own learning goals and the necessity to evaluate encountered information in relation to their goals and be open to refine or adjust their goals during the course of online learning and learning. 85 Students need to make a balanced use of learner-directed and Web-directed sub-plans to capitalize on the affordances of the Web and take personal control of their learning rather than passively follow the information on the Web. Instruction should pay intensive attention to the development of learner-directed reciprocally adaptive sub-plans (and their execution, i.e., LICRA searches proposed by DeSchryver & Spiro, 2008), which are essential for ill-structured problem solving and demand for creative synthetic connections between what learners have learned on the Web, their prior knowledge, and the specific requirements of the learning context. Assistance or explicit instruction is required to help students develop particular types of immediate specific search plans that they would be less likely to develop on their own (e.g., comparing information sources, reflecting on plan execution, critically evaluating information encountered). Specific tactics such as developing an evolving list of plans, sub-plans in particular, may help students navigate through the complexity of online searching and learning tasks, and thus should be tested in different practical contexts. Third, the identified connections between online plan development and general epistemic beliefs suggest that instruction of online planning strategies, and possibly online searching and learning strategies in general, should take into account students’ general epistemic beliefs. Instruction should focus not only on teaching students planning strategies but also on developing more expansive epistemic beliefs in students, which would require further research. 86 Limitations Several limitations of this study need to be noted. First, the findings of the study are limited by the small sample size. This study aimed to conduct a fine-grained qualitative analysis of the different levels/types of online plans developed by learners with different epistemic beliefs. The themes and connections identified in this study should be further examined with larger samples in future research. Second, common methodological issues with thinking aloud should be acknowledged. Thinking aloud might not exactly reflect what participants have in their minds. For instance, participants might make some plans that they feel unnecessary to verbalize, even when they are probed to think out loud. The thinking aloud method may pose (meta)cognitive challenges to participants while they are engaged in the demanding problem-solving process, thus affecting their planning processes. Third, the questions and prompts participants received during their online searching and learning might influence how they searched the Web and introduce the possibility of having cognitive processes that they would not develop in a natural Web learning setting. It is worth noting, however, these prompts could help participants think aloud about their planning processes and the rationales for their planning decisions, which constituted the core data set for this study. They could also help avoid having false negatives – missing important reasonings that participants feel unnecessary to mention or 87 fail to verbalize because of their unfamiliarity with the thinking aloud method. Future research can examine this study’s findings under non-prompt conditions. Finally, the design and data analysis of this study took into consideration several individual difference factors, which by no means constitute an exhaustive list of factors that might mediate the connections between online planning and epistemic beliefs. Further studies should be conducted to control for more possible influencing factors. For example, although no connections were revealed between major backgrounds and epistemic beliefs when examining all the survey responses, the two learners holding expansive epistemic beliefs came from science majors (Actuarial Science and Zoology), and the two with relatively reductive epistemic beliefs were from art majors (English and Journalism). Further research should look into the role of majors in shaping the connections between epistemic beliefs and online planning. 88 CHAPTER 6 Conclusion Planning is an important contributor to open and deep learning on the Web, which is crucial for learning and problem solving in ill-structured domains. Situated in the context of online ill-structured problem-solving, this study contributes to the literature in several aspects. First, it further developed and tested the efficacy of a proposed three-level model of planning with a more diverse group of students. This model lays the groundwork for future studies to delve more deeply into the role of planning in online searching and learning, especially in complex and ill-structured learning contexts. Second, this is the first study to examine how different levels/types of online searching and learning plans connect to learners’ epistemic beliefs. It was found that learners with expansive epistemic beliefs tended to demonstrate more complexity in a wider range of aspects involved in online planning compared to learners with relatively reductive epistemic beliefs. Third, this study revealed learners’ different exploration approaches to addressing an online ill-structured problem-solving task: (a) researching more sources to fulfill a relatively large number of sub-plans, (b) exploring multiple sources for each of a moderate number of sub-plans, or (c) focusing on only a few sources with relatively high credibility. 89 Future research can further examine the effectiveness of these different approaches in different learning or problem-solving contexts. Fourth, the findings of this study suggest that online planning and epistemic beliefs exert a combined influence on learners’ ill-structured problem-solving performance. Future studies should be conducted to further investigate the individual effects of online planning and epistemic beliefs on learners’ problem-solving performance. 90 APPENDICES 91 APPENDIX A The Revised Cognitive Flexibility Inventory Please rate how much you agree or disagree with each of the following statements on a scale of 1 to 7 (1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: neither agree nor disagree, 5: somewhat agree, 6: agree, 7: strongly agree). There are no right or wrong answers, and we are just interested in what you really believe about the nature of knowledge and learning. Part I 1. A Greek poet said: “The fox knows many things, but the hedgehog knows one big thing.” I’m more like a hedgehog. When I’m trying to understand and learn about some topic, I generally try to relate what I’m learning to a single central system or viewpoint. 2. Breaking apart complex topics to study each individual part separately is often misleading because components of complex topics tend to interact and affect each other. In most areas of study, the whole is usually not the same as the sum of its parts. 3. Learning works best when I am left with a lot of flexibility regarding what should be learned and how I should learn it. I prefer to figure a lot of things out for myself. 4. Different aspects or sub-topics of knowledge should be separated into different mental “boxes”, so they don’t interfere with each other. Keeping knowledge neatly organized in this way helps to avoid confusion in my later uses of knowledge. 92 5. When phenomena appear inconsistent or disorderly, it is probably because a system for organizing them has not yet been found. But, it’s likely that such a system exists. 6. The notion that ideas can “come to life” makes no sense. Concepts are merely abstractions. 7. I enjoy encountering difficult, conflicting, and disorderly concepts and find them challenging. 8. I have learned some topic best when I have examined its various phenomena using different explanatory systems or perspectives. 9. The parts of a whole system tend to be alike (i.e., systems tend to be homogeneous). Uniformity of explanation throughout a system is very desirable. 10. When placed in a new situation in which previously learned information has to be applied, I usually try to remember, from what my teachers or textbooks have said, some plan for what I should do in the new situation (either from some general rule or from a very similar example I have been taught). 11. Exams are important, but they are not the ultimate goal of my learning. I set my own personal standards; self-evaluation matters most to me. 12. I find it very hard to tolerate ambiguity or inconsistency. I like things to have a clear, uniform answer. 13. Learning works best for me when it is self-directed. 93 14. Learning is essentially a process of receiving information and recording it accurately in memory for later retrieval and use. 15. I am highly motivated by internal factors (e.g., what I intrinsically want to do). Part II 16. I prefer simplicity, consistency, and orderliness. Whenever possible, I prefer not to encounter complex concepts in school (although I deal with complexity when I have to). 17. Complex topics should be best broken down into parts and each individual part should be studied separately. In most areas of study, the whole topic is usually equal to the sum of its parts. 18. Doing well on exams is my most important learning goal. I am very concerned with how others evaluate me. 19. Different aspects or sub-topics of knowledge should be highly interrelated in the mind along a variety of different dimensions so that I can see their connections from different perspectives. Although this is not a very neat way to organize knowledge, keeping knowledge interconnected in this way will be beneficial in my later uses of knowledge. 20. Learning works best for me under the guidance of experts (e.g., teachers). 21. When phenomena appear inconsistent or disorderly, it is probably the case that a single system for organizing them cannot be found. The phenomena of the world are frequently 94 quite hard to account for by grand, unifying systems, and thus multiple explanatory systems should be used to explain them. 22. Ideas can “come to life.” Concepts can be personally experienced in a vital manner. 23. A Greek poet said: “The fox knows many things, but the hedgehog knows one big thing.” I’m more like a fox. When I’m trying to understand and learn about some topic, I try to look at the topic from different perspectives, rather than trying to relate everything to a single central system or viewpoint. 24. Learning is essentially an active process in which I personally construct understandings and acquire the ability to apply my knowledge in new ways to various new situations. 25. I have learned some topic best when I have accounted for its various phenomena using some single, more abstract, explanatory system or perspective. 26. There tends to be a lot of diversity within the parts of any whole system. Many parts are not like the whole. Uniformity of explanation throughout a system, while nice, is not essential. Multiple explanations are all right and sometimes necessary. 27. Learning works best when I am told exactly what I am supposed to learn and how I should learn it. Everything should be made explicit to me. 28. When placed in a new situation in which previously learned information has to be applied, I do not count on remembering some plan for what I should do in the new situation (either from some general rule or from a very similar example I have been taught). Instead, 95 I tend to focus on figuring out how it might be necessary to apply my knowledge in a somewhat new way that fits the new situation. 29. I am highly motivated by external factors (e.g., what other people expect of me). 30. I do not find ambiguity or inconsistency too troubling. It’s all right if things don’t always have a clear answer or cannot be explained uniformly. Yet it is essential that I should know the underlying factors accounting for the ambiguity or inconsistency. Note. The following items were reverse coded in data analysis: 1, 4, 5, 6, 9, 10, 12, 14, 16, 17, 18, 20, 25, 27, and 29. 96 APPENDIX B Schommer’s Epistemological Questionnaire Please rate how much you agree or disagree with each of the following statements on a scale of 1 to 7 (1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: neither agree nor disagree, 5: somewhat agree, 6: agree, 7: strongly agree). There are no right or wrong answers, and we are just interested in what you really believe about the nature of knowledge and learning. 1. If you are ever going to be able to understand something, it will make sense to you the first time you hear it. 2. The only thing that is certain is uncertainty itself. 3. For success in school, it's best not to ask too many questions. 4. A course in study skills would probably be valuable. 5. How much a person gets out of school mostly depends on the quality of the teacher. 6. You can believe almost everything you read. 7. I often wonder how much my teachers really know. 8. The ability to learn is innate. 9. It is annoying to listen to a lecturer who cannot seem to make up his/her mind as to what he/she really believes. 10. Successful students understand things quickly. 11. A good teacher's job is to keep his/her students from wandering from the right track. 97 12. If scientists try hard enough, they can find the truth to almost anything. 13. People who challenge authority are over-confident. 14. I try my best to combine information across chapters or even across classes. 15. The most successful people have discovered how to improve their ability to learn. 16. Things are simpler than most professors would have you believe. 17. The most important aspect of scientific work is precise measurement and careful work. 18. To me studying means getting the big ideas from the text, rather than details. 19. Educators should know by now which is the best method, lectures or small group discussions. 20. Going over and over a difficult textbook chapter usually won't help you understand it. 21. Scientists can ultimately get to the truth. 22. You never know what a book means unless you know the intent of the author. 23. The most important part of scientific work is original thinking. 24. If I find the time to re-read a textbook chapter, I get a lot more out of it the second time. 25. Students have a lot of control over how much they can get out of a textbook. 26. Genius is 10% ability and 90% hard work. 27. I find it refreshing to think about issues that authorities can't agree on. 28. Everyone needs to learn how to learn. 29. When you first encounter a difficult concept in a textbook, it's best to work it out on your own. 98 30. A sentence has little meaning unless you know the situation in which it is spoken. 31. Being a good student generally involves memorizing facts. 32. Wisdom is not knowing the answers, but knowing how to find the answers. 33. Most words have one clear meaning. 34. Truth is unchanging. 35. If a person forgot details, and yet was able to come up with new ideas from a text, I would think they were bright. 36. Whenever I encounter a difficult problem in life, I consult with my parents. 37. Learning definitions word-for-word is often necessary to do well on tests. 38. When I study, I look for the specific facts. 39. If a person can't understand something within a short amount of time, they should keep on trying. 40. Sometimes you just have to accept answers from a teacher even though you don't understand them. 41. If professors would stick more to the facts and do less theorizing, one could get more out of college. 42. I don't like movies that don't have an ending. 43. Getting ahead takes a lot of work. 44. Getting ahead takes a lot of work. 99 45. You should evaluate the accuracy of information in a textbook, if you are familiar with the topic. 46. Often, even advice from experts should be questioned. 47. Some people are born good learners, others are just stuck with limited ability. 48. Nothing is certain, but death and taxes. 49. The really smart students don't have to work hard to do well in school. 50. Working hard on a difficult problem for an extended period of time only pays off for really smart students. 51. If a person tries too hard to understand a problem, they will most likely just end up being confused. 52. Almost all the information you can learn from a textbook you will get during the first reading. 53. Usually you can figure out difficult concepts if you eliminate all outside distractions and really concentrate. 54. A really good way to understand a textbook is to re-organize the information according to your own personal scheme. 55. Students who are "average" in school will remain "average" for the rest of their lives. 56. A tidy mind is an empty mind. 57. An expert is someone who has a special gift in some area. 100 58. I really appreciate instructors who organize their lectures meticulously and then stick to their plan. 59. The best thing about science courses is that most problems have only one right answer. 60. Learning is a slow process of building up knowledge. 61. Today's facts may be tomorrow's fiction. 62. Self-help books are not much help. 63. You will just get confused if you try to integrate new ideas in a textbook with knowledge you already have about a topic. Note. The following items were reverse coded in data analysis: 1, 3, 5, 6, 8, 9, 10, 11, 12, 13, 16, 17, 19, 20, 21, 31, 33, 34, 36, 37, 38, 40, 41, 42, 44, 47, 49, 50, 51, 52, 55, 57, 58, 59, 62, and 63. 101 APPENDIX C Prior Content Knowledge Test 1. How would you rate your knowledge level on the topic of genetically modified/engineered foods on a 5-point scale? (5=very knowledgeable, 1= not knowledgeable at all) 2. What is your opinion on whether we should eat genetically modified/engineered foods? Please explain why you hold this viewpoint. 3. Genetically modified foods, also known as genetically engineered foods, are foods derived from organisms whose genetic material (DNA) has been modified in a way that does not occur naturally, e.g., through the introduction of a gene from a different organism. A. True B. False C. I don’t know 4. Genetically modified/engineered foods can have a higher nutritional value than their traditionally bred counterparts. A. True B. False C. I don’t know 5. In the U.S., there are no national standards on mandatory labeling of genetically modified/engineered foods. In most states, manufacturers voluntarily label their foods as containing or not containing genetically modified ingredients. A. True B. False C. I don’t know 6. In the U.S., foods produced from genetically modified/engineered plants are required to meet the same food safety requirements as foods derived from traditionally bred plants. 102 A. True B. False C. I don’t know 7. In the U.S, no genetically modified/engineered livestock has been approved for human consumption yet. A. True B. False C. I don’t know 8. The Food and Drug Administration is the only institution responsible for regulating genetically modified/engineered foods in the U.S. A. True B. False C. I don’t know 9. Genetically modified/engineered foods were introduced into the American food supply in the 1990s. A. True B. False C. I don’t know 10. Which is NOT among the Top 4 genetically modified/engineered crops in the U.S.(cid:1) A. Tomato B. Cotton C. Corn D. Soybeans E. I don’t know 11. Monsanto, DuPont, and Bayer are major suppliers of genetically modified seeds? A. True B. False C. I don’t know 12. In natural settings, modified genes from genetically modified crops can NOT be transferred to other plants? A. True B. False C. I don’t know 103 APPENDIX D Ill-Structured Problem-Solving Performance Grading Rubric Table 7 Ill-Structured Problem-Solving Performance Grading Rubric Criterion 4 points Perspectives Analyzes the problem from multiple stakeholders’ perspectives. Shows concern to fully understanding positions of the stakeholders. Sources of information Presents a balanced and critical view of multiple sources of knowledge (facts, concepts, personal experience, theory and research, etc.) to make informed judgments. Grading scale 3 points Analyzes the problem from multiple stakeholders’ perspectives but tends to focus on and take side of only one key stakeholder. Shows concern to critically evaluating multiple sources of knowledge (facts, concepts, personal experience, theory and research, etc.) to make informed judgments. Ignores 1 or 2 critical information sources. 2 points Recognizes different perspectives and multiple stakeholders. Focuses on only one perspective partially to analyze the problem. Shows concern to using multiple sources of knowledge to make informed judgments. Ignores 2 or more critical information sources and focuses on trivial or irrelevant ones. Misunderstands some of the issues. 1 point Displays no or little concern for understanding the problem from multiple perspectives. Demonstrates uncritical dependence on certain resources OR on gut instinct and fails to explore and interpret other sources of evidence for better reasoning. 104 Table 7 (cont’d) Developing a solution Reflecting on proposed solution Context- adaptiveness of response Assesses the viability of the solution relative to 3 or more important issues and constraints associated with the causes of the problem. Reflects on possible consequences of proposed solution relative to all of the important issues and constraints. Shows deep understanding of complex and interactive nature of actions and decisions. Assesses the viability of the solution relative to 2 important issues and constraints associated with the causes of the problem. Reflects on possible consequences of proposed solution relative to some important issues and constraints. Shows some understanding of complex and interactive nature of actions and decisions. Response fits with the specific context of the task. Takes careful consideration of multiple stakeholders involved in the context. Response fits with the specific context of the task. Takes into consideration multiple stakeholders involved in the context. Assesses the viability of the solution relative to 1 issue or constraint associated with the causes of the problem. Reflects on possible consequences of proposed solution relative to few issues or constraints. Shows superficial understanding of complex and interactive nature of actions and decisions. Response somewhat fits with the specific context of the task. Takes into consideration only 1 or 2 stakeholders involved in the context. Makes no attempt to assess the viability of the solution. Makes no attempt to reflect on possible consequences of proposed solution. Shows no or little awareness of complex and interactive nature of actions and decisions. Response does not fit with the specific context of the task. Takes no consideration of the stakeholders involved in the context. 105 Table 7 (cont’d) Synthetic connections of ideas Clarity, accuracy, concision of representation Completeness of response Makes good synthetic connections of ideas Makes few synthetic connections of ideas. Makes no synthetic connections of ideas. Presented ideas are consistent. Most ideas are explained with clarity. No misrepresentation of factual information. Addresses whether to introduce GM foods. Proposes one measure for how to introduce GM foods. Presented ideas are unclear and inconsistent. Misrepresents one point of factual information. Addresses whether to introduce GM foods but not the “how” part of the task. Presented ideas are unclear and inconsistent. Misrepresents 2-3 points of factual information. Gives no clear response to whether to introduce GM foods. Fails to address the “how” part of the task. Makes creative and compelling synthetic connections of ideas Presented ideas are clear, succinct and consistent. No misrepresentation of factual information. Addresses the “whether” and “how” parts of the task. Proposes 2-3 measures for how to introduce GM foods. 106 APPENDIX E Think-Aloud Instructions I am interested in how you search the Web to solve the given problem. To help me understand how you search the Web, you will need to say out loud your thoughts while you are making explorations on the Web. This method is called thinking aloud. It simply means verbalizing whatever thoughts occur in your mind as you search, browse, and read on the Internet. Here are some prompts about what you should think aloud. 1. What information you want to search for and why, and what search terms you are going to use. 2. Why you selected a search result or clicked a link, but not other results or links. 3. What you are looking at, what you are thinking about the information you encounter, and why you are thinking that way. 4. Why you decided to read more on a webpage. 5. What you plan to search for next and why. 6. When you take notes, please tell me what you are writing down and why you decide to write them down. 7. How you think the information you are reading helps, or does not help, in solving the problem. 8. Any other things that come to your mind: how you understand the information, prior knowledge or experience you can relate to, etc. 107 If I notice that you are not explaining your online exploration decisions, I will ask questions like “What are you thinking? What do you plan to do next?” Just to help you think aloud. I may also ask some follow-up questions to help you explain why you make certain exploration decisions. Below is a sample think-aloud transcript to help you see what thinking aloud may look like. This transcript shows how a hypothetical learner searches the Web to find out what people can do to reduce the effects of climate change. Please read out loud this transcript and let me know if you have any questions about thinking aloud. “So the first thing, well, people talk about climate change, but to be honest, I don’t really know much about it. I’ll… I’m gonna go to Google. And I’m gonna go with the basic term. Let me start with climate change. [typing in the search term climate change] Okay, I’m looking at the search results…top stories…so I know these are news reports. I’m just gonna skip those for now and go with the definition and facts of climate change. ‘Climate change: How do we know? Evidence and facts’ This seems helpful, so I’m gonna open it. Okay, it’s a NASA website. I’m looking at the top of the page, and this graph is catching my eye. ‘For centuries, atmospheric carbon dioxide had never been above this line’ Okay, this is evidence about climate change. I’m wondering if there is any other evidence. [scrolling down the page]Okay, here is more. ‘Global temperature rise’, ‘warming oceans’, ‘shrinking ice sheets’…So climate change is happening. I would like to read more details later, but now I want to check what else I can find on the page. [scrolling up the page] Now 108 I’m looking at this box on the right side here ‘scientific consensus’ I’m curious what consensus… ’Ninety-seven percent of climate scientists agree that climate warming trends over the past century are very likely due to human activities…This is actually very helpful because I want to figure out what we can do to reduce climate change and it says human activities are indeed related to climate change. This means there are actions we can take to reduce it. Then next, I want to look at what human activities are causing climate change… the causes…” 109 APPENDIX F Think-Aloud Prompts and Questions Below are some sample prompts or questions that were used to help participants think out loud during their online exploration process or to help them explain their online planning moves and underlying epistemic beliefs. During-task prompts/questions: 1. What are you thinking? 2. What do you want to do/search for next? 3. Why did you select this website to look at? Why did you decide to search for that? 4. Why did you say that? Post-task interview questions: 1. What information would like to search for if you had more time to work on the task? 2. Do you think there is a clear answer to whether genetically modified foods are safe to eat? Please explain your perspective. 3. Some websites you visited presented different or even contradictory information about the effects of eating genetically modified foods. How do you think about the inconsistent or contradictory information? What are some possible explanations you can think of for the inconsistencies or contradictions? 110 4. In the questionnaire you filled out earlier, you said you find it very hard to tolerate ambiguity or inconsistency and you like things to have a clear answer. But here it seems you are fine with the inconsistent information you have read. Can you please explain why? 5. After going through this online learning process for solving a problem, do you think the Web is a place for you to learn basic information or is it more an environment that helps you think for yourself about genetically modified foods? 6. For searching the Web to learn about genetically modified foods, you have full control of your own learning (e.g., how you want to go about your searching, what you want to learn in detail). Do you like this kind of flexibility or do you prefer to have me provide you with instructions on what information to search for and how (e.g., giving you a list of websites to explore)? And why? 7. During your online learning, you have encountered some general summaries or conclusions made by people or institutions (e.g., conclusions presented on the World Health Organization website or on the Food and Drug Administration website, summaries of possible effects of genetically modified foods listed as bullet points). Also, you have learned about some individual cases (e.g., consumers talking about their experiences of eating genetically modified foods, nutritionists posting their views on eating genetically modified foods, scholars presenting their specific findings about eating a particular kind of genetically modified food). How 111 do you think about these two types of information – general summaries and individual cases? Which is more helpful for you to develop your understanding or perspective on this topic of genetically modified foods? Or are they equally helpful? And why? 8. I noticed that you changed your opinion. Originally, you said we should not use genetically modified foods. In your report, you obviously became more open to the use of GM foods. Can you please explain what made you change your opinion? 112 APPENDIX G Coding Scheme of Online Planning Processes Table 8 Coding Scheme of Online Planning Processes Plans Global plan Codes Identifying or addressing general learning goals Developing a stance on the topic Description Plans or decisions related to the general goals or purposes of the problem-solving task. Plans or decisions related to forming or changing one’s own opinion on the learning topic for solving the problem. Coding guidelines Use the code when a learner (a) determines, analyzes, or refines the goals or purposes of the task, (b) reminds himself/herself of the goals or purposes, or (c) proposes measures to address the goals or purposes. Use the code when a learner makes explicit plans about how to form his/her own opinion or expresses his/her inclination or decision about what stance to take on the topic. 113 Coding examples “Considering the report that I want, there might be some people that might be against it at the, at my university, so I’d rather have that information out there for them to also be able to choose.” “Maybe I’ll look up also later the biggest viewpoints surrounding GMOs, so I can kind of know what people like and don’t like, or are skeptical about, so I can kind of know which angles to take since it seems like GMOs are like a relatively good thing.” Table 8 (cont’d) Sub-plan Web-directed Sub-plan (two sub-codes: Web-directed immediate sub-plan & Web-directed deferred/reflective sub-plan) Plans or decisions related to particular aspects within the larger task and primarily influenced by information learners encounter on the Web. Use the code Web-directed immediate sub-plan when a learner a) plans or decides to shift to a new or different aspect of the topic by clicking a link or (b) makes a new search using the exact information encountered at the moment. Use the code Web-directed deferred/reflective sub-plan when a learner (a) forms a sub-plan based on the exact information encountered on the Web but decides to carry out the plan at a later stage of learning, or (b) decides to revisit a previously formed sub-plan created based on the exact information encountered on the Web. Do NOT use the codes when a learner shifts to a new or different aspect of the topic simply by following a linear reading path of a website (e.g., planning to read a report section by section, skipping a section to read a new section on the same webpage), but use the code “controlling depth or pace of reading” instead. “It referenced an article that I might be interested in viewing. It’s the review of health risks of genetically modified foods. I’m gonna open this (the referenced article).” --Web-directed immediate sub-plan “This article talks about ‘[could] the future of good labeling be changing for good?’ I’m just gonna write a note to go back on food labeling… I’ll go back and look for that information.” --Web-directed immediate sub-plan “This article talks about ‘[could] the future of good labeling be changing for good?’ I’m just gonna write a note to go back on food labeling… I’ll go back and look for that information.” “The first thing I’ll do is go back to labeling and just search on Google about labeling because that’s one of the first things I said I’ll go back to look at.” --Web-directed deferred/reflective sub-plan 114 Table 8 (cont’d) Learner-directed sub-plan (two sub-codes: learner-directed top-down sub-plan & learner-directed reciprocally adaptive sub-plan) Plans or decisions related to particular aspects within the larger task and essentially guided by learners. Use the code learner-directed Top-down sub-plan when a learner makes a search plan about what aspect of the topic to explore based on his/her prior knowledge or understanding of the task and its goals. Use the code learner-directed reciprocally adaptive sub-plan when a learner creatively generates or refines search plans based on synthesis of information on the Web, prior knowledge, and understanding of the task. Immediate specific search plan Generating search terms or defining search range Developing exploration paths of search results Plans or decisions related to the use of search engines, search terms, or range of searches Plans or decisions related to selecting the paths of exploring search results Use the code when a learner makes plans or decisions on what search engines or search terms to use or what types of information sources to explore. Use the code when a learner explains his/her plans or decisions about which search results to explore, in what order, or what search results not to explore. 115 “So I think one of the last searches I’d like to do is look for GMOs and the World Health Organization because they usually give out public opinions about anything pertaining to human and animal health.” -- learner-directed top-down sub-plan “It says UK down here, so this might be, not be how America says about GMOs… the next website I go to, I will probably look for like a US authority on how they assess GMOs.--learner-directed reciprocally adaptive sub-plan “So I’ll go back to my search (genetically modified food in general), but instead of using just regular Google, I’ll go to Scholar Google and try to find any articles.” “The first thing is a little Time article called ‘genetically modified foods: what is grown and eaten in the U.S.’ That might be interesting, but I would like to see something from like FDA or Table 8 (cont’d) Do NOT use the code if a learner simply views the first search result without giving any justifications or if there is only one targeted search result (e.g., searching the website of an origination like WHO). Evaluating information helpfulness, currency, or credibility Plans or decisions related to evaluating the helpfulness, currency, or credibility of information Use the code when a learner explicitly expresses his/her plans or actions of critically evaluating the helpfulness, currency, or credibility of information. Do NOT use the code when a learner simply says that the information is interesting, helpful, or credible, or simply mentions the authors or publication dates of the information without giving any critical evaluation. 116 like something like that. So I’m scrolling down. There is some Wikipedia pages. There is something [down below the Wikipedia page] that says the U.S. regulation of genetically modified crops. [scrolling further down] There is also another one called ‘consumer info about food from GM plants.’ That one is from the FDA. So I might read that one. It’s a government administration.” “I’m going to read on and see if this is important and how it factors into my presentation, my report, and if it’s necessary for me to look further later.” “That quote seems to be just from an interview with the researcher, and not a result of actual research but that still seems credible…he is a plant molecular biologist, so it seems like he is pretty connected with what this issue actually is.” Table 8 (cont’d) Controlling reading depth or pace Plans or decisions related to controlling the depth or pace of reading Taking notes Reflecting on plan execution Plans or decisions related to when to take notes and what notes to take Plans or decisions related to monitoring or evaluating the execution of plans Use the code when a learner explicitly explains his/her plans or actions about controlling the depth or pace of reading. Do NOT use the code when a learner simply follows a linear reading path of a webpage (reading a paper section by section) without explicitly justifying the action. Do NOT use the code when a learner decides to leave a webpage simply because he or she reaches the end of the webpage. Use the code when a learner expresses his/her plans or decisions about taking notes. “that might just be more of the last paragraph that I read, um very wordy, the consultation process things like that. So I might not go into that yet especially cuz I have other things I wanted to research.” “I think I’m just kind of gonna look through the article super quick for the history part.” I’m gonna write down antibiotic resistance marker genes and GM technology, safe or unsafe?” Use the code when a learner explicitly explains his/her plans or decisions about checking how well the plans have been carried out. “So that helped me learn a lot about some of the challenges in commercial agriculture, solutions that [are provided by] GM foods and crops, problems and controversies. So I'm gonna check that off, antibiotic resistance. I'm checking my notes to see how much I followed up on, and what to follow up on next.” 117 Table 8 (cont’d) Comparing information from different sources Plans or decisions about comparing the information presented from different sources. Use the code when a learner explicitly expresses his/her plans about comparing different information sources. “I think I’m gonna go back to the other page (a Purdue webpage) and then maybe come back to this one (a Harvard webpage) and keep cross comparing if they, to see what they are both saying.” 118 REFERENCES 119 REFERENCES Afflerbach, P., & Cho, B.-Y. (2009). Determining and describing reading strategies: Internet and traditional forms of reading. In H. S. Waters & W. Schneider (Eds.), Metacognition, strategy use, and instruction (pp. 201–225). Guilford Press. Anderson, R. C., & Freebody, P. (1979). Vocabulary Knowledge. Technical Report No. 136. https://eric.ed.gov/?id=ED177480 Barzilai, S., & Zohar, A. (2012). Epistemic Thinking in Action: Evaluating and Integrating Online Sources. Cognition and Instruction, 30(1), 39–85. https://doi.org/10.1080/07370008.2011.636495 Bråten, I., & Strømsø, H. I. (2006). Epistemological beliefs, interest, and gender as predictors of Internet-based learning activities. Computers in Human Behavior, 22(6), 1027–1042. https://doi.org/10.1016/j.chb.2004.03.026 Bråten, I., Strømsø, H. I., & Britt, M. A. (2009). Trust matters: Examining the role of source evaluation in students’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44(1), 6–28. https://doi.org/10.1598/RRQ.44.1.1 https://doi.org/10.1598/RRQ.44.1.1 Buehl, M. M., Alexander, P. A., & Murphy, P. K. (2002). Beliefs about Schooled Knowledge: Domain Specific or Domain General? Contemporary Educational Psychology, 27(3), 415–449. https://doi.org/10.1006/ceps.2001.1103 Campbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013). Coding In-depth Semistructured Interviews: Problems of Unitization and Intercoder Reliability and Agreement. Sociological Methods & Research, 42(3), 294–320. https://doi.org/10.1177/0049124113500475 Castek, J., Coiro, J., Guzniczak, L., & Bradshaw, C. (2012). Examining peer collaboration in online inquiry. The Educational Forum, 76, 479–496. https://doi.org/10.1080/00131725.2012.707756 Cheng, C., & Spiro, R. J. (2016, November). Exploring skilled Web learners’ planning process in online reading and learning for solving ill-structured problems. Literacy Research Association Annual Conference, Nashville, TN. 120 Cho, B.-Y., Woodward, L., & Li, D. (2018). Epistemic Processing When Adolescents Read Online: A Verbal Protocol Analysis of More and Less Successful Online Readers. Reading Research Quarterly; Newark, 53(2), 197–221. https://doi.org/10.1002/rrq.190 Coiro, J. (2011). Predicting reading comprehension on the Internet: Contributors of offline reading skills, online reading skills, and prior knowledge. Journal of Literacy Research, 43, 352–392. https://doi.org/10.1177/1086296X11421979 Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by sixth-grade skilled readers to search for and locate information on the Internet. Reading Research Quarterly, 42, 214–257. https://doi.org/10.1598/RRQ.42.2.2 Dee-Lucas, D. (1999). Information Location in Instructional Hypertext: Effects of Content Domain Expertise. In EdMedia+ Innovate Learning (pp. 242-247). Association for the Advancement of Computing in Education (AACE) DeSchryver, M. (2014). Higher Order Thinking in an Online World: Toward a Theory of Web-Mediated Knowledge Synthesis. Teachers College Record; New York, 116(12), 1. DeSchryver, M., & Spiro, R. J. (2008). New forms of deep learning on the Web: Meeting the challenges of cognitive load in conditions of unfettered exploration in online multimedia environments. In R. Zheng (Ed.), Cognitive effects of multimedia learning. IGI Global, Inc. https://doi.org/10.4018/978-1-60566-982-3.ch133 Ekstrom, R. B., French, J. W., Harman, H. H., & Dermen, D. (1976). Kit of factor-referenced cognitive tests. Educational Testing Service. Gagne, R. M. (1964). Problem Solving. In A. W. Melton (Ed.), Categories of Human Learning (pp. 293–317). Academic Press. https://doi.org/10.1016/B978-1-4832-3145-7.50018-2 Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. https://doi.org/10.1002/RRQ.027 121 Hartman, D. K., Morsink, P. M., & Zheng, J. (2010). From print to pixels: The evolution of cognitive conceptions of reading comprehension. In E. A. Baker (Ed.), The new literacies: Multiple perspectives on research and practice (pp. 131–164). Guilford Press. Hofer, B. K. (2004). Epistemological Understanding as a Metacognitive Process: Thinking Aloud During Online Searching. Educational Psychologist, 39(1), 43–55. https://doi.org/10.1207/s15326985ep3901_5 Hofer, B. K., & Pintrich, P. R. (1997). The Development of Epistemological Theories: Beliefs about Knowledge and Knowing and Their Relation to Learning. Review of Educational Research, 67(1), 88–140. https://doi.org/10.2307/1170620 Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288. https://doi.org/10.1177/1049732305276687 Jacobson, M. J., & Spiro, R. J. (1995). Hypertext Learning Environments, Cognitive Flexibility, and the Transfer of Complex Knowledge: An Empirical Investigation. Journal of Educational Computing Research, 12(4), 301–333. https://doi.org/10.2190/4T1B-HBP0-3F7E-J4PN Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology, Research and Development; New York, 45(1), 65–94. https://doi.org/10.1007/BF02299613 Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology, Research and Development; New York, 48(4), 63. https://doi.org/10.1007/BF02300500 King, P. M., & Kitchener, K. S. (1994). Developing Reflective Judgment: Understanding and Promoting Intellectual Growth and Critical Thinking in Adolescents and Adults. Jossey-Bass. Kurasaki, K. S. (2000). Intercoder Reliability for Validating Conclusions Drawn from Open-Ended Interview Data. Field Methods, 12(3), 179–194. https://doi.org/10.1177/1525822X0001200301 122 Leu, D. J., Kiili, C., & Forzani, E. (2016). Individual differences in the new literacies of online research and comprehension. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text, and context (pp. 259–272). Routledge. Leu, D. J., Kinzer, C. K., Coiro, J. L., Castek, J., & Henry, L. A. (2013). New literacies: A dual-level theory of the changing nature of literacy, instruction, and assessment. In D. E. Alvermann, N. Unrau, & R. B. Ruddell (Eds.), Theoretical models and processes of reading (pp. 1150–1181). International Reading Association. Morgan, D. L. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1), 48–76. https://doi.org/10.1177/2345678906292462 Perry, W. G. (1968). Forms of intellectual and ethical development in the college years: A scheme. Harvard University. Potelle, H., & Rouet, J.-F. (2003). Effects of content representation and readers’ prior knowledge on the comprehension of hypertext. International Journal of Human-Computer Studies, 58(3), 327–345. https://doi.org/10.1016/S1071-5819(03)00016-8 Qian, D. (1999). Assessing the Roles of Depth and Breadth of Vocabulary Knowledge in Reading Comprehension. Canadian Modern Language Review, 56(2), 282–307. Reitman, W. R. (1964). Heuristic decision procedures, open constraints, and the structure of ill-defined problems. In M. W. Shelley & G. L. Bryan (Eds.), Human Judgments and Optimality (pp. 282–315). Wiley. Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82(3), 498–504. https://doi.org/10.1037/0022-0663.82.3.498 Schraw, G., Dunkle, M. E., & Bendixen, L. D. (1995). Cognitive processes in well-defined and ill-defined problem solving. Applied Cognitive Psychology, 9(6), 523–538. https://doi.org/10.1002/acp.2350090605 Shapiro, A., & Niederhauser, D. (2004). Learning from hypertext: Research issues and findings. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (2nd ed., pp. 605–620). Lawrence Erlbaum. 123 Simon, H. A. (1973). The structure of ill structured problems. Artificial Intelligence, 4(3), 181–201. https://doi.org/10.1016/0004-3702(73)90011-8 Spilich, G. J., Vesonder, G. T., Chiesi, H. L., & Voss, J. F. (1979). Text processing of domain-related information for individuals with high and low domain knowledge. Journal of Verbal Learning and Verbal Behavior, 18(3), 275–290. https://doi.org/10.1016/S0022-5371(79)90155-5 Spiro, R. J., Collins, B. P., Thota, J. J., & Feltovich, P. J. (2003). Cognitive Flexibility Theory: Hypermedia for Complex Learning, Adaptive Knowledge Application, and Experience Acceleration. Educational Technology, 43(5), 5–10. Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson, D. K. (2004). Cognitive flexibility theory: Advanced knowledge acquisition in ill-structured domains. In R. B. Ruddell & N. Unrau (Eds.), Theoretical models and processes of reading (5th ed., pp. 640– 653). International Reading Association. Spiro, R. J., Feltovich, P. J., & Coulson, R. L. (1996). Two epistemic world-views: Prefigurative schemas and learning in complex domains. Applied Cognitive Psychology, 10, S51–S61. https://doi.org/10.1002/(SICI)1099-0720(199611)10:7<51::AID-ACP437>3.0.CO;2-F Spiro, R. J., & Jehng, J. C. (1990). Cognitive flexibility and hypertext: Theory and technology for the nonlinear and multidimensional traversal of complex subject matter. In D. Nix & R. J. Spiro (Eds.), Cognition, education, and multimedia: Explorations in high technology (pp. 163–205). Lawrence Erlbaum. Spiro, R. J., Klautke, H. A., Cheng, C., & Gaunt, A. (2017). Cognitive flexibility theory and the assessment of 21st century skills. In C. Secolsky & D. B. Denison (Eds.), Handbook on Assessment, Measurement, and Evaluation in Higher Education (2nd ed.) (pp. 631-637). Routledge. https://doi.org/10.4324/9781315709307-50 Spiro, R. J., Klautke, H., & Johnson, A. K. (2015). All bets are off: How certain kinds of reading to learn on the web are totally different from what we learned from research on traditional text comprehension and learning from text. In R. J. Spiro, M. DeSchryver, M. S. Hagerman, P. M. Morsink, & P. Thompson (Eds.), Reading at a crossroads?: Disjunctures and continuities in current conceptions and practices (pp. 45–50). Routledge. 124 Steffens, C. L. (2012). A collective case study observing the print and online comprehension of four second graders (Order No. 3521821) [Doctoral dissertation, Fordham University]. ProQuest Dissertations Publishing. Toy, S. (2007). Online ill-structured problem-solving strategies and their influence on problem-solving performance (Order No. 3274858)[Doctoral dissertation, Iowa State University]. ProQuest Dissertations Publishing. Tsai, C.-C., & Chuang, S.-C. (2005). The correlation between epistemological beliefs and preferences toward Internet-based learning environments. British Journal of Educational Technology, 36(1), 97–100. https://doi.org/10.1111/j.1467-8535.2004.00442.x Ulyshen, T. Z., Koehler, M. J., & Gao, F. (2015). Understanding the connection between epistemic beliefs and Internet searching. Journal of Educational Computing Research, 53, 345–383. https://doi.org/10.1177/0735633115599604 Whitmire, E. (2004). The relationship between undergraduates’ epistemological beliefs, reflective judgment, and their information-seeking behavior. Information Processing & Management, 40(1), 97–111. https://doi.org/10.1016/S0306-4573(02)00099-7 Willoughby, T., Anderson, S. A., Wood, E., Mueller, J., & Ross, C. (2009). Fast searching for information on the Internet to use in a learning context: The impact of domain knowledge. Computers & Education, 52(3), 640–648. https://doi.org/10.1016/j.compedu.2008.11.009 Zhang, S., & Duke, N. K. (2008). Strategies for Internet reading with different reading purposes: A descriptive study of twelve good Internet readers. Journal of Literacy Research, 40, 128–162. https://doi.org/10.1080/10862960802070491 125