CULTIVATING CRITICAL MINDSETS IN THE DIGITAL INFORMATION AGE: TEACHING MEANINGFUL WEB EVALUATION By Angela Kwasnik Johnson A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Educational Psychology and Educational Technology—Doctor of Philosophy 2017 ABSTRACT CULTIVATING CRITICAL MINDSETS IN THE DIGITAL INFORMATION AGE: TEACHING MEANINGFUL WEB EVALUATION By Angela Kwasnik Johnson This dissertation examines the use of dialogic discussion to improve young adolescents’ ability to critically evaluate web sites. An intervention unit comprised three iterations of an instructional cycle in which students independently annotated web sites about controversial issues and discussed the reliability of those sites in dialogic discussions. Data for examining student change came from an evaluation task measure wherein students evaluated the reliability of web sites before and after the unit. A questionnaire in which students recalled site stance and authorship features for each site measured the strength of the source models that students built during the evaluation task. Students’ evaluation strategies were also examined in an independent inquiry task in which they evaluated potential sources and included justifications for final source choices in an annotated bibliography. Students became more critical after the unit, and their attention to authorship and information sourcing features increased. In the post-unit source model task, students recalled more authorship features, suggesting stronger source models. After the unit, the students’ approach toward evaluation was more investigative. Attention to authorship and information sourcing remained strong. Analysis of the dialogic discussions revealed authorship and information sourcing to be important foci of students’ evaluation processes. In addition, the dialogue suggested that students made important shifts from reductive to flexible epistemic mindsets, as evidenced by student-driven investigation and concept revision during the discussions. The study affirms the value of attention to source features in evaluating web sites. It also affirms the use of instructional methods wherein students encounter the complexities inherent in web evaluation, especially by examining various cases that present multiple complications of reliability in shifting contexts. It suggests that repeated iterations of shared inquiry in the form of dialogic discussion may effectively shift students’ critical mindsets toward a more skeptical and probing approach to web evaluation. Copyright by ANGELA KWASNIK JOHNSON 2017 To Dad, who still inspires me to question assumptions, work with purpose, and seek the truth. v ACKNOWLEDGEMENTS This endeavor is far beyond the scope of others I have attempted, and I could not have completed it without the considerable support of many people. The Education Department faculty at Michigan State University, and especially my dissertation committee, provided thoughtful direction, wise advice, and generous encouragement when I most needed it. I am especially grateful to Dr. Ralph Putnam, whose many hours of patient guidance steered me steadily through the inconstant travails of this journey. Other professional and academic colleagues generously offered their time and expertise at various stages of the project: Valerie Carr, Amy Dirlam, Alma Holtgren, Michelle Humes, Michelle Shira-Hagerman, Laura Veldman, and especially Tracy Russo, who contributed hours of tireless effort to improve the quality of the coding. Finally, I owe much to my family, who has heartened and supported me along the way. Madelyn and Sam, thank you for your patience and understanding when this degree pulled me from greater involvement in your lives, and for your enthusiastic encouragement despite that. Most of all, thank you Michael, for all you do and all you are. I would not be here without you. vi TABLE OF CONTENTS LIST OF TABLES………………………………………………………………………………...x LIST OF FIGURES …………………………………………………………..………………….xi CHAPTER 1: INTRODUCTION...…….………………………………………..………...…...…1 CHAPTER 2: THEORY AND RESEARCH......…………..………..…………………………….6 Comprehension Theories and Critical Evaluation ….…………………………..……..6 Offline Reading Theory ………………………………………………………….6 Multiple Text Processing …………………………………….………………….7 Critical Evaluation in Models of Online Reading ………………………………9 Influences on Evaluating Multiple Conflicting Texts ………………………..…...…..11 Prior Knowledge ………………………………..……………………………....11 Prior Opinion ………………………………..………………………………….12 Metacognitive Skill………………………...……………………………………13 Teaching Critical Evaluation …………………………………….…………………….15 The Interplay of Contrasting Cases and Source Features ……………….…..15 Authentic Tasks and Cognitive Flexibility ……………………………………19 Dialogic Instruction ……………………………………………..……………...23 CHAPTER 3: AN INSTRUCTIONAL UNIT ON CRITICAL EVALUATION ………...……..26 Design Principles ……………...……………..…………………………………………26 Activity Structures ……………………………………………………………………..29 Structure 1: Inquiry-of-the-week ……………………………………………...29 Structure 2: Socratic Seminar …………………………………………………32 Structure 3: Contrasting Cases Scenario …..…………………………………33 Overview of the Instructional Unit ……..……………………………………………..35 Research Questions …….…………….………..…………...…………………………..36 CHAPTER 4: METHOD….…….....…..…………………………………………………………37 Setting ……….………………………………………………………………….………37 Participants ……………….……………………………..……………………………..38 The Instructional Unit ……….……….………………………………………………..39 Critical Evaluation Lesson ……….…………………………………………….39 Annotation Lesson ………………………………………..…………………….40 Inquiry-of-the-week.……………..……………………………………………..41 Socratic Seminar …………………………..……………………………………41 Subsequent Practice Cycles ………………………………………….………...43 Data Sources ……………………...……………...……………………………………..43 Critical Evaluation Task …………………………………………….…………43 Source Model Measure ………………………………………………..………..47 Embedded Evaluation ………………………………………….………………48 vii Process Data …………………………………………………...………………..49 CHAPTER 5: CHANGES IN STUDENTS’ WEB SITE EVALUATION...…………………….51 Site Ratings ………………………..……………...……………………………………. 51 Evaluation Criteria ….....……………………………………………………………….53 Key Criteria Categories ………………………………………………….……. 59 Key Criteria Categories by Class ……………………………………………... 61 Key Criteria Subcategories and Extraneous Criteria ……………………...…63 Authorship ………………………………………..…………………….64 Site qualities …………………………………………...……….……….66 Information sourcing………………………………………….….…….67 Content qualities……..…………………………………………….……68 Extraneous criteria ………………………………………….…….……70 Approach to Task …………………………………….…….…………………………..72 Source Model Construction ……………………………...……….……………………77 CHAPTER 6: EVALUATION THEMES IN DIALOGIC DISCUSSIONS…...…….....……….80 The Socratic Seminars ………………………………………….……………….…….. 81 Complexities of Authorship ………...……………………………….…………………82 Authorship in Seminar 1 ………………………………………….……………83 Authorship in Seminar 2 …………………………………………….…………87 Authorship in Seminar 3 ……………………………………….………………88 Complexities of Information Sourcing ……...…………………………………….…...91 Information Sourcing in Seminar 1 ……………………………………………92 Information Sourcing in Seminar 2 ……………………………………………94 Information Sourcing in Seminar 3 ……………………………………………98 Investigation ………………………………...…………………………………………104 CHAPTER 7: DISCUSSION...….…………………...…………………………………………111 Developing Students’ Evaluation Skills …………………….………………………..112 Attention to Authorship and Sourcing ……………………………………….112 Behaviors Related to Evaluation ……………………………………….…….114 Effective Instruction in Critical Evaluation………………………………….115 Conceptual interdependence……….……………………………...…115 Avoiding reductive bias……………..……..………………...………...116 Multiple variations and contexts……………………………………...117 Multiple perspectives...………………………….…..………………...118 Tolerance for complexity and ambiguity...…….………………...…..119 Student agency and independence……….…………………………...121 Limitations...………………………………………….……………………......124 Implications and Future Research ......…………………...………….………126 APPENDICES……………………………………………………………………………….…129 Appendix A: Contrasting Cases Activity Sheet ……......…….…………………......130 Appendix B: Inquiry-of-the-week Assignment…………………...…………………132 Appendix C: Web Sites for IoWs and Socratic Seminar Discussions………….…..134 viii Appendix D: Template for Socratic Seminar Response………………………….....137 Appendix E: Sites for Pre-unit and Post-unit Evaluation Tasks …………………..138 Appendix F: Directions for Pre-unit and Post-unit Evaluation Tasks ……………..140 Appendix G: Evaluation Form for Pre-unit and Post-unit Evaluation Tasks ….…146 Appendix H: Source Model Measure……………………….. …………………….…147 Appendix I: Source Sheet for Independent Inquiry Task…………………………...148 Appendix J: Annotated Bibliography Instructions for Independent Inquiry Task………………………………………………………………….…………149 Appendix K: List of Codes………………………………………………………….....150 REFERENCES …………………………………………………………………………………153 ix LIST OF TABLES Table 1. Mean site ratings by experts and students..................................................................52 x LIST OF FIGURES Figure 1. Overview of the study………..……………………………………………………….35 Figure 2. Mean site ratings by students ………...………….….………………………………53 Figure 3. Hierarchy of code categories and subcategories for evaluation justifications…….55 Figure 4. Pre-unit, post-unit, and embedded task categories …………………………….….57 Figure 5. Advanced class pre-unit, post-unit, and embedded task categories ……………....58 Figure 6. Regular class pre-unit, post-unit, and embedded task categories….……………...59 Figure 7. Pre-unit, post-unit, and embedded task Key criteria subcategories…….……..….60 Figure 8. Advanced class pre-unit, post-unit, and embedded task Key criteria subcategories..…………………………………………………………………………..61 Figure 9. Regular class pre-unit, post-unit, and embedded task Key criteria subcategories…..………………………………………...……………………………..62 Figure 10. Mean references to Key criteria subcategories per site by each student………..63 Figure 11. Mean references to Authorship criteria subcategories per site by each student…………………………………………………………………………………...64 Figure 12. Percentage of references attributed to each code within the subcategory Authorship in the pre-unit, post-unit, and embedded evaluation tasks…...………...65 Figure 13. Mean references to Site Qualities criteria subcategories per site by each student …….………………………………………………………………..…………...65 Figure 14. Percentage of references attributed to each code within the subcategory Site Qualities in the pre-unit, post-unit, and embedded evaluation tasks………………..66 Figure 15. Mean references to Information Sourcing criteria subcategories per site by each student …………...……………………………………………………………….67 Figure 16. Percentage of references attributed to each code within the subcategory Information Sourcing in the pre-unit, post-unit, and embedded evaluation tasks.....68 Figure 17. Mean references to Content Qualities criteria subcategories per site by each student…………………………………………………………………….………69 xi Figure 18. Percentage of references attributed to each code within the subcategory Content Qualities in the pre-unit, post-unit, and embedded evaluation tasks..….....70 Figure 19. Mean references to Extraneous criteria subcategories per site by each student…71 Figure 20. Percentage of references attributed to each code within the category Extraneous criteria in the pre-unit, post-unit, and embedded evaluation tasks…….72 Figure 21. Approach to Task by category…………………………….………………………..74 Figure 22. Approach to Task by code………………………….………………………………75 Figure 23. Advanced class Approach to Task by code…………………………………….…..76 Figure 24. Regular class Approach to Task by code……….……………………………….….76 Figure 25. Stance recollection frequencies per student in source model measures…………77 Figure 26. Author recollection frequencies per student in source model measures…………78 Figure 27. Mean correct author recollection frequencies per student by class level…….….79 xii CHAPTER 1: INTRODUCTION In today’s digital world, information is no longer the purview of traditional libraries and the print sources they contain (Christensen, Johnson, & Horn, 2008; Reynolds, 2011; Williams, 2006).  Instead, the average information seeker accesses a digital source, most commonly the Worldwide Web, available almost anywhere by almost anyone (Smith, McGeeney, Duggan, Rainie, & Keeter, 2015).  On the web, traditional barriers to information production no longer exist, and anyone with a linked device is free to publish whatever they wish, without vetting or review (Cooke, 2017; Rheingold, 2010). As a result, information disseminated via individually produced web pages appears alongside information disseminated via more traditional publishing routes, resulting in a leveling effect that presents all sources as equally reliable and valid (Burbules, 1997). Conflicting ideas and information appear equally viable, making it difficult to ascertain the relative weight of different perspectives. This is particularly problematic in an era in which the prevalence of fake news, misinformation, and disinformation is amplified; consequently, readers must “take charge of the information allowed into their minds and decision-making schemas” (Cooke, 2017, p. 216). Moreover, the web’s advanced technology affords more than blanket dissemination of information.  Data mining and access to personal web search habits afford corporate and private interests the ability to target audiences with personalized messages specific to their interests or affiliations, which automatically appear in one’s social network stream, in one’s email inbox, or in the margins of one’s browser screen (Rheingold, 2010). Therefore, information users can no longer depend on others to critique and evaluate information sources; they must critically evaluate information themselves, and would do well to approach the web with a default skepticism rather than a default trust. 1 Given these circumstances, a movement toward critical literacy, with its roots in critical theory, has been advocated by some theorists. Fabos (2009) describes critical literacy as a means toward which readers come to understand that texts “do not exist in a vacuum. Texts are interrelated; they are the products of economic and political wrangling” (p. 861).  Her understanding is grounded in media studies and literacy studies, and is informed by Paolo Freire’s critical pedagogy, who advocated for critical literacy to empower individuals to greater social agency (see Giroux, 2010).  To this end, Fabos advances the seeking out of conflicting ideas, the analysis of political perspectives, and the thoughtful critique of any discourse that is politically or economically driven, including the internet. Likewise, Giroux (2014) advocates the inclusion of such discourse in the public school curriculum, proffering an educational agenda which “re-imag[es] teaching as a vital public service and schools as democratic public spheres” (p. 8).  He views the most important role of teachers in this regard: “[Their job] involves both educating students to be critical thinkers and preparing them to be activists in the best sense of the term—that is, thoughtful and active citizens willing to fight for the economic, political, and social conditions and institutions that make democracy possible” (p. 8). From this perspective, the ability to access and use information wisely may be one of the most important methods by which a citizen-informed democracy can be preserved (Gainer, 2010; Kellner & Share, 2007).  It becomes, therefore, a primary responsibility of schools to teach students to approach information from a default critical stance and to support them in learning to do so (Cooke, 2017; Gregory & Cahill, 2009). The drive toward critical literacy is not just the clamor of activist voices.  The Common Core State Standards (CCSSI, 2010) have recognized the importance of critical inquiry and prioritized the rigorous expectation that students evaluate information critically, especially when 2 multiple perspectives conflict.  The standards call for eighth grade students to “gather relevant information from multiple print and digital sources . . . assess the credibility and accuracy of each source; and quote or paraphrase the data and conclusions of others” (p. 66). But they also call for students to “determine an author’s point of view or purpose in a text and analyze how the author acknowledges and responds to conflicting evidence or viewpoints” (p. 39). The Speaking and Listening standards require that students “analyze the purpose of information presented in diverse media formats (e.g., visually, quantitatively, orally) and evaluate the motives (e.g., social, commercial, political) behind its presentation” (p. 49). In the History and Social Studies Standards for Grades 6-8, students are to “identify aspects of a text that reveal an author’s point of view or purpose (e.g., loaded language, inclusion, or avoidance of particular facts)” (p. 61). To do so, students must be able to “provide an accurate summary of the source distinct from prior knowledge and opinions” (p. 61). They are to “identify where the texts disagree on matters of fact and interpretation” (p. 39), and construct their own arguments by “Support[ing] claim[s] with logical reasoning and relevant evidence, using accurate, credible sources” (p. 42). In other words, complex critical reasoning is essential, applied not only to ideas within texts, but to the contexts in which and purposes for which they are produced. Despite the need for critical evaluation skills, studies suggest that students of all ages have difficulty determining and applying effective criteria for evaluating the trustworthiness and reliability of web sources (Coombes, 2008; Eastin, Yang, & Nathanson, 2006; Kiili, Laurinen, & Marttunen, 2007; K. S. Kim & Sin, 2011; Kuiper, Volman, & Terwel, 2005; Makinster, Beghetto, & Plucker, 2002; Walraven, Brand-Gruwel, & Boshuizen, 2009). Students tend to rely on relevance criteria to choose web sites, and many fail to consider the authority of information sources (Coiro, Coscarelli, Maykel, & Forzani, 2015; Coombes, 2008; Hirsh, 1999; Julien & 3 Barker, 2009; Paul, Macedo-Rouet, Rouet, & Stadtler, 2017; Walraven et al., 2009) or to check their accuracy (Flanagin & Metzger, 2010; Scholz-Crane, 1998). In a previous study of sixth graders (Johnson & Putnam, 2012), I found that only a third of participants considered trustworthiness when prompted to evaluate web sites, and in over 70% of those cases evaluation was a hasty response to Wikipedia rather than a thoughtful examination of the source or its contents. In addition, almost none of the students considered the trustworthiness of sites they did not recognize, suggesting a default trust of unfamiliar sites rather than a default skepticism. Moreover, readers often overestimate both their ability to critically evaluate (Metzger, Flanagin, & Nekmat, 2015) and the frequency with which they do so (Colwell, Hunt-Barron, & Reinking, 2013; Flanagin & Metzger, 2010). Indeed, evaluation seems to be one of the more challenging aspects of online reading (Walraven, Brandgruwel, & Boshuizen, 2009). A large study recently examined the critical evaluation skills of seventh graders in several states in the Eastern United States using a digital measure called the Online Reading Comprehension Assessment (Coiro, Coscarelli, Maykel, & Forzani, 2015; Forzani & Markel, 2015). Researchers found evaluation to be the most difficult for students—significantly more challenging than location or synthesis. Just 25% of the students could ascertain the reliability of web sites using accurate and sufficient reasons, and evidence of several challenges emerged: three quarters of the students had trouble articulating evaluation criteria, half made inaccurate or generalized assumptions about the sites, and a fifth failed to share any criteria for evaluation at all. Overall, just a quarter of the sample applied acceptable criteria for evaluation. Therefore, given the current need for thoughtful and effortful critical evaluation of web sources, the deficiencies of many students’ present critical evaluation skills, and the responsibility of schools to teach those skills, this study examined the effects of an instructional 4 unit designed to move students toward a critical stance in their use of web sources. The unit incorporated elements of instruction that prior research suggested may effectively promote critical evaluation, and the study examined the effects of this instruction on students’ critical evaluation of web sources. 5 CHAPTER 2: THEORY AND RESEARCH Theory and past research contributes to our understanding how critical evaluation is integrated in the online reading process, offers insight into why it may be underpracticed by readers, and sheds light on instruction that might be effective to address those deficiencies. In this chapter, I begin with a discussion of the research that informs my understanding of how critical evaluation occurs in the context of online reading, why critical evaluation on the web is problematic, and what these suggest about the substance of effective instruction. I then move to a discussion of the literature informing my choice of specific activity structures to teach critical evaluation skills. Comprehension Theories and Critical Evaluation Theories of comprehension offer insight into how critical evaluation of web sources is embedded in the complex broader context of online reading. An overview of reading theory and multiple text processing theory is essential for generating an understanding of online reading comprehension, as well as how critical evaluation plays out in the online context. Offline Reading Theory It is widely accepted that reading is a complex cognitive process involving several subprocesses, whether it occurs online or offline. Seminal work by Kintsch (1974) established that reading involves the decomposition of written text into propositions that are subsequently organized into a hierarchical structure that together create a mental representation of the textbase (Kintsch & van Dijk, 1978). This textbase grounds the reader’s ability to summarize a text, recall facts, and answer simple questions about it. For deeper comprehension to occur, however, the reader must integrate the textbase with prior knowledge, activated in long-term memory. 6 This requires the reader to make inferences, defined by Rouet (2006) as “any idea that is generated from the reader’s long-term memory, in reaction to incoming textual information” (p. 11). Inferences can be automatic, as when strong links between incoming information and prior knowledge exist, or controlled, as when the reader must deliberately manipulate information in working memory to integrate it with prior knowledge to construct a cohesive situation model (Ericsson & Kintsch, 1995). The latter process occurs when automatic inference generation fails, and its success depends on several individual factors, including the depth of the reader’s prior knowledge, the capacity of short-term memory, and the reader’s motivation to construct a cohesive situation model in the first place (Rouet, 2006). Ideally, the result of this effort is an integrated situation model representing the reader’s conception of the subject matter, updated as new information is encountered and assimilated, and stored in long term memory for later retrieval (Kintsch & van Dijk, 1978). Multiple Text Processing Kintsch’s comprehension theory grounds a subsequent theory of multiple text reading posited by Rouet (2006). The Documents Model of Multiple Text Processing explains what occurs when a reader must comprehend multiple texts. Since reading experiences on the web almost always require the processing of multiple documents, the theory is of central importance to learning from web sources. Rouet asserts that multiple text processing is unique in three ways: First, side by side encounters with different texts draw attention to their differences, which facilitate the updating of situation models. Second, a subsequently encountered document may corroborate or complement a previously encountered document by reiterating ideas or filling in gaps, a process that requires the reader make higher level connections between and among texts. Finally, and most relevant to the present study, source information takes a 7 prominent role in identifying and differentiating between documents so each can contribute to a global representation of the topic. The prominence of source information is, in fact, central to Rouet’s (2006) model. The model posits that the situations model, that is, the global representation an expert reader creates while reading multiple texts (hence the plural form), includes two components or nodes. The content node includes a mental representation (or singular situation model) of the content of each source, integrated with the prior knowledge of the reader. Individual sources’ situation models combine to create the situations model into a kind of content overview. The source node is a mental representation combining information about a source and its author, including identification, affiliations, expertise, bias, and prior knowledge of these. Together the source representations of individual texts combine to create a broader sources model, which would reflect, for example, whether the authors agreed or disagreed. The content and source representations of individual texts are integrated with prior knowledge to form a global understanding of the topic. To successfully construct this global model, the reader must identify the source of each individual text, compare information in one text to that of another, and integrate these into a coherent global model. This requires the reader maintain a connection between source nodes and content nodes, remembering not only what was said in each source, but who said it. Informing Rouet’s theory was a seminal study by Wineburg (1991), which examined how expert and novice historians read multiple texts. Expert readers in the study demonstrated three important strategies that novices did not: They focused on source information, corroborated information by comparing details in the documents, and used their prior knowledge to place new information into broader contexts. On the other hand, Golder and Rouet (as cited in Rouet, 8 2000) found that younger readers—more likely to be novices—demonstrated difficulty in keeping track of connections between source and content nodes. The findings suggest that sourcing is an important requirement for facilitating students’ text comprehension of multiple online texts, but that it is also challenging for nonexpert readers. One would expect younger students to fall into the latter category, and therefore to be challenged by both comprehension and evaluation while reading multiple texts online. In light of these findings, enabling attention to the relationship between the author, purpose, and content of a message may be an effective step toward both comprehension and evaluation. Critical Evaluation in Models of Online Reading. Several frameworks of online multiple-text reading view multiple-text comprehension and critical evaluation as interrelated aspects of the meaning-making process. For instance, Leu, Kinzer, Coiro, and Cammack (2004) outlined online literacies to include processes for (a) identifying questions, (b) locating information, (c) analyzing the usefulness of that information (evaluation), (d) synthesizing information, and (e) communicating information. Afflerbach and Cho (2009) proposed an updated model of constructively responsive reading grounded in Pressley and Afflerbach’s (1995) previous model that included an expanded list of strategies including (a) meaning-making from multiple texts; (b) self-monitoring to troubleshoot comprehension problems and assess reading progress; (c) information evaluation to determine the “validity and reliability of the content, author reputation, or source information” (Cho & Afflerbach, 2015, p. 25); and (d) internet-specific strategies for text location. Similarly, Goldman, Braasch, Wiley, Graesser, and Brodowinska (2012) categorized the strategic processing verbalized by study participants as self-explanation, paraphrasing, monitoring, evaluation, and navigation. Hartman, Morsink, and Zheng (2010) posited that a reader must 9 integrate three types of knowledge in comprehending online text: (a) knowledge of identity— knowing who wrote a text and how authors “construct, represent, and project online identities” (p. 146); (b) knowledge of location—knowing how to “orient oneself in a website” and “in cyberspace” (p. 148); and (c) knowledge of one’s own goal—knowing why one is reading and remaining focused on that goal. The first of these requires a reader attend specifically to source characteristics; however, evaluation presents itself in each type of knowledge: The reader assesses an author’s trustworthiness, a site’s effectiveness in helping to orient the reader, and a site’s match to reading goals. Though one may argue that good readers of print texts have always practiced critical evaluation, it is not woven into offline text processing to the extent it is with online texts. In fact, Hartman and colleagues theorized that a complication of online reading resides in the “multiple plurals” (p. 140) of online texts. The various elements that combine to establish meaning—for example, reader, author, task, context, and so forth—are themselves plural and continually shifting, and therefore confound exponentially the act of meaning construction. The continually shifting nature of these elements makes critical evaluation more difficult, but also more essential to a reader’s global construction of meaning. Empirical studies confirm a strong relationship between comprehension and web source evaluation. Multiple text comprehension skills, which, according to Rouet (as discussed above), require attention to source characteristics, appear to interact with evaluation processes. In several studies, students who evaluated sources for credibility and reliability also comprehended better (Braasch et al., 2009; B.Y. Cho, Woodward, Li, & Barlow, 2017; Goldman et al., 2012; Wiley et al., 2009). Goldman et al. found that “the evaluation processes observed . . . seemed to emerge as a result of meaning-making processes” (p. 376). In another study, students who conducted cross-document linking and source evaluation also constructed better arguments on a 10 scientific topic (Anmarkrud, Bråten, & Strømsø, 2014), suggesting that source evaluation and meaning construction are closely linked. It appears, then, that teaching students to attend to sourcing might be one effective way to improve both evaluation and comprehension of multiple online texts. Influences on Evaluating Multiple Conflicting Texts Given the cognitive challenges that comprehension of multiple conflicting texts requires, however, it is not surprising that students overlook author and source information as they evaluate web sources (Bråten, Strømsø, & Britt, 2009; Coiro & Dobler, 2007; Zawilinski et al., 2007). In this section I discuss several factors known to influence students’ ability to effectively evaluate multiple conflicting texts. Prior Knowledge An important influence on a reader’s ability to critically evaluate multiple conflicting texts is the topic knowledge the reader brings to the task (Bråten, Strømsø, & Salmerón, 2011; Eastin et al., 2006; Fogg et al., 2003; Wineburg, 1998). Prior knowledge has been found to positively influence self-regulated learning (Greene, Costa, Robertson, Pan, & Deekens, 2010) and to improve critical evaluation (Rouet, Favart, Britt, & Perfetti, 2014; Wineburg, 1991). Bråten et al. (2011) hypothesized the limited-resources hypothesis, positing that readers with more background knowledge of a topic could expend fewer cognitive resources in determining the relevance of potential web sources for an information task, and therefore have more resources available for evaluating their trustworthiness; Salmerón, Kammerer, and Garcíacarrión (2013) affirmed that hypothesis with college students. Johnson, Sbaffi, and Rowley (2015) found third year university students based critical evaluations most often on quality of content, whereas first year students most often cited ease of use. Moreover, credibility was the 11 second most cited criteria among third year students, but did not appear as a significant factor among first year students. If the limited-resources hypothesis holds true, it is likely that secondary and middle grade students, who have generally accumulated less topic knowledge than college students, face even greater shortages of cognitive resources for effective evaluation. The theory may help to explain why, as students grow older, they increase attention to critical evaluation when reading online texts (Metzger, Flanagin, Markov, Grossman, & Bulger, 2015). In addition, Mason, Junyent, and Tornatora (2014) found prior topic knowledge to moderate the effectiveness of an intervention to improve critical evaluation skills with ninth graders. Students with more prior knowledge about the topic of genetically modified foods benefitted from the short-term intervention, whereas students with low prior knowledge did not. Prior Opinion A complication to critical evaluation that is relevant specifically to conflicting texts occurs when the reader brings a strong prior opinion to the task. Conflicting texts may present controversial issues, which will necessarily offer opposing perspectives. A reader with at least some prior knowledge of the issue may approach the task with a prior opinion, and this presents a challenging scenario: The executive functions necessary for effective comprehension and evaluation must occur while, at the same time, the reader must bracket off his initial bias to retain a measured, objective stance while evaluating both the content and the source. This presents an added layer of complexity requiring metacognitive skill and cognitive resources (Haria & Midgette, 2014). Indeed, there is considerable evidence that prior opinion influences a reader’s comprehension of multiple conflicting texts. Research of offline readers has long revealed a confirmation bias to exist; that is, readers seek information that aligns with their prior opinion 12 and ignore information that does not (Lord, Ross, & Lepper, 1979; Taber & Lodge, 2006). The tendency has been confirmed in readers of multiple texts; Van Strien, Brand-Gruwel, and Boshuizen (2014) found eleventh graders’ prior attitudes strongly biased how they dealt with conflicting information in an open-ended reading and writing task. Kim and Afflerbach (2014) found that undergraduates reading conflicting texts applied reading patterns differently depending on their topic attitudes, and found evidence of biased assimilation. In a prior study I also found prior opinion to be significantly related to students’ evaluations of web sources presenting opposing sides of a conflicting issue, with sources presenting aligned opinions rated on average more highly than sources presenting misaligned opinions (Johnson & Putnam, 2014). It seems, therefore, that strong prior opinion may impede, or at least influence, effective evaluation of multiple conflicting sources. Metacognitive Skill Metacognitive skill is another influence on students’ critical evaluation (Kiili, Laurinen, & Marttunen, 2009; Lombardi, Sinatra, & Nussbaum, 2013; Mason, Boldrin, & Ariasi, 2009; Stadtler & Bromme, 2007; Zhang & Quintana, 2012), and an important focus in this research. Metacognition involves the ability to monitor and self-regulate one’s thinking and learning, and has been likened to a toolbox: The skilled learner knows which tools to use in specific circumstances, and can alleviate some of the mental workload by efficient selection and application of those tools (Ford & Yore, 2012). In the cognitive workspace, metacognition is an executive function that allows for “planning, monitoring, and regulating actions and command of materials to respect the spatial limitations” of memory to allow greater cognitive resources for message processing (Ford & Yore, 2012, p. 258). 13 In addition, metacognition is generally considered to be a “significant path to critical thinking” (Magno, 2010, p. 137). According to Kuhn and Dean (2004), critical thinking requires “meta-level operations” (p. 270) that consist of both separate and integrated metacognitive skills functioning at the executive level. The executive operations that serve metacognition include declarative, conditional, and procedural knowledge; planning, monitoring, and debugging strategies; information management; and evaluation functions (Magno, 2010). Metacognition in online reading is operationalized in comprehension strategies used by strategic readers. Pressley and Afflerbach (1995) called this constructively responsive reading, delineating strategies such as judging the relevance of a text for the reading goal (evaluating), detecting comprehension difficulties (monitoring) and regulating reading in response to those (monitoring) as necessary for effective comprehension. Afflerbach and Cho (2009) updated the model to include strategies essential in the online environment, including locating potential texts to read, evaluating information based on the validity and reliability of both content and sources, and constructing meaningful intertextual linkages. In their close examination of seven competent high school internet readers, the students habitually self-monitored as they read online, displaying adept and effortful metacognitive strategies that served them in all phases of the search process, from planning and locating to critical evaluation and intertextual synthesis. Afflerbach and Cho (2009) affirmed that “self-monitoring is central to flexibly regulating the transactional processes between the reader and the internet text environment (Coiro & Dobler, 2007) and . . . it is an important executive function to help readers achieve their goals (Stadtler & Bromme, 2007)” (p. 282). Ostensibly, this metacognitive process requires considerable skill on the part of the reader, which may explain, in part, why students who are better offline comprehenders also tend to be better online comprehenders and better evaluators (Coiro, 2008). 14 All of this suggests that critical evaluation can be facilitated by metacognitive scaffolds. Lombardi and colleagues (2013) implemented a successful intervention for supporting the evaluation of arguments that employed a method by which students mapped evidence for opposing models. Recognizing the complexities of online inquiry, Zhang and Quintana (2012) designed a digital support system for scaffolding students’ metacognitive processes, which they found to facilitate a “fewer-but-deeper pattern” of reading (p. 194) in which students read fewer sources but spent more time on each. Although their study did not specifically examine evaluation behaviors, their results suggest that metacognitive supports allow for deeper processing, a likely prerequisite for evaluation. Likewise, Stadtler and Bromme (2007) found that students receiving metacognitive prompts for evaluation more frequently justified their critical evaluations by mentioning information source features and demonstrated more knowledge of sources than did a control group. Teaching Critical Evaluation Although many studies have presented a pessimistic picture regarding students’ critical evaluation skills, considerable research shows a variety of short-term interventions to be effective in improving those skills (Braasch, Bråten, Strømsø, Anmarkrud, & Ferguson, 2013; Mason et al., 2014; Walraven et al., 2009). Here I present specific elements of interventions that informed the design of this study. The Interplay of Contrasting Cases and Source Features There is some evidence that the contrasting cases presented by multiple conflicting texts may inherently encourage readers’ attention to source characteristics, and by extension, to critical evaluation. Schwartz and Bransford (1998) found classroom instruction using contrasting cases to increase a learners’ attention to the similarities and differences between 15 cases, deepening their conceptual knowledge and improving the success of learning transfer to new contexts. Braasch, Rouet, Vibert, and Britt (2012) proposed an explanation for the effectiveness of using contrasting cases to increase attention to source features, the discrepancyinduced source comprehension (D-ISC) assumption. The D-ISC assumption posits that readers who are confronted with conflicting ideas are prompted to pay greater attention to source characteristics to make sense of the discrepancies. That increased attention facilitates the building of source-content links and stimulates critical evaluation based on source characteristics, allowing the reader to construct a more coherent global understanding. Their two-part study used eye-tracking methodology to observe undergraduate students reading sets of consistent and conflicting stories. Readers then wrote summaries of the stories from memory. They more frequently mentioned sources, and remembered the sources better, after reading conflicting stories. Moreover, eye-tracking data showed that readers made more visual fixations and had longer gaze times when reading conflicting stories, signifying that readers attended more closely when texts conflicted. On the other hand, there is evidence that conflicting texts may prompt more attention to content and less attention to source features. Bråten, Braasch, Strømsø, and Ferguson (2014) found that college undergraduates faced with conflicting scientific texts focused their attention primarily on the arguments presented when evaluating rather than to source or author characteristics. Likewise, fourth and fifth graders reading texts presenting different points of view on the same topic had difficulty retaining links between content (what was said) and sources (who said it). When they did link content to sources, students often failed to spontaneously consider the authority of the source in terms of knowledge or expertise (MacedoRouet, Braasch, Britt, & Rouet, 2013). In view of the limited resources hypothesis discussed 16 previously, it may be that, when a text is particularly challenging (as with argumentative or scientific texts) or when the reader possesses limited prior knowledge (as with young readers), greater resources are expended for text comprehension at the expense of attention to source features. In the same study, the source-feature intervention did improve poorer comprehenders’ performance on source knowledge questions, suggesting that attention to source features can improve critical evaluation skills. The link between evaluation and sourcing has been recognized by Anmarkrud and colleagues (2014), who concluded that “attention to source information and the establishment of source–content links are required to construct a global, coherent understanding of an issue when documents present conflicting perspectives” (p. 74). Likewise, Britt, Richter, and Rouet (2014), in their implications for instruction to improve science literacy, concluded that “lack of conflict, opposing perspectives, and uncertainty can all deprive students of the training they need to develop skills and dispositions for dealing with science on the web” (p. 119), a recommendation that could be applied to the pursuit of information on the web in any domain. A premise underpinning this study is that the use of conflicting texts can increase a reader’s attention to source features as long as the reading task does not overtax the reader’s cognitive resources. I also presuppose that attention to source features is an important prerequisite for critical evaluation, and therefore strive to improve students’ critical evaluation competencies by increasing their attention to the source features of multiple conflicting texts. A contrasting cases intervention designed by Braasch and colleagues (2013) is of interest to this study. As noted, contrasting cases have been used effectively to improve learning (Gadgil, Nokes-Malach, & Chi, 2012; Kurtz, Miao, & Gentner, 2001; Rittle-Johnson & Star, 2011; Schwartz & Bransford, 1998). Braasch and colleagues capitalized on the effectiveness of contrasting cases interventions to improve students’ critical evaluation skills. In their 17 intervention, upper secondary students participated in three modules, each of which presented a print document excerpt purportedly found online to answer the question of whether cell phones pose health risks. Directly above one another were think-aloud responses of two students to the texts. Participants were told that one of these students was a “better, more critical” student and the other was a “poorer, less critical” student, but the students were not specifically identified. The think-aloud responses contained strategy protocols used by better and poorer critical evaluators, and while examining each text students completed a worksheet table asking them to list the strategies used by each student, label each as a sound or poor strategy, and explain why. Participants discussed their answers in small groups and in a large group. In a postintervention measure the following day, participants critically evaluated several types of sources (a textbook, a popular science print magazine, a popular science online magazine, an online blog post, a political commentary from a print newspaper, and a newsletter from a personal web site), and wrote essays using the sources. Students who had received the contrasting cases intervention more effectively differentiated between more and less reliable sources, were more likely to rank sources based on trustworthiness, and more likely to justify their rankings with source feature attributes (author, publisher, venue, document type) than the control group. Moreover, the intervention students wrote essays containing more information from reliable sources and more essential core concepts on the topic, suggesting stronger mental representations constructed from multiple documents and more discerning evaluations. The results of the study show promise for using a contrasting cases scenario as an efficient and effective method to “stimulate source feature considerations in multiple-documents comprehension” (p. 192). 18 Authentic Tasks and Cognitive Flexibility Studies examining longer term interventions for encouraging critical evaluation in authentic tasks are few, and at least one of these points to a problematic gap between controlled studies and classroom practice. Colwell, Hunt-Barron, and Reinking (2013) used a formative design to implement critical evaluation instruction in a middle school classroom using the method of internet reciprocal teaching. Although the students showed growth in their ability to evaluate web sites immediately after instruction, researchers found students did not significantly apply those skills in an independent inquiry project; instead they reverted to previous habits of relying on teacher supports to assist them in source location, selection, and evaluation. The authors suggested that results may have been influenced by the teacher’s willingness to assist students when they demonstrated frustration during the information gathering process, which occurred often. Bearing in mind this study and previously discussed results from Forzani and Markel (2015), I postulate that the highly demanding process of critical evaluation may suffer when placed in the context of a less structured open task in which other cognitive processes compete for resources. Especially when increased cognitive demands require greater mental effort, students may understandably seek ready relief from supportive teachers. In any case, teaching students to independently apply critical evaluation skills in authentic open internet tasks is the goal of instruction. This problem of transfer to authentic environments is longstanding, and may be explained in part by Spiro and colleagues’ cognitive flexibility theory (CFT) (Spiro, Coulson, Feltovich, & Anderson, 1998), advanced to explain the difficulties of knowledge application in ill-structured domains. An ill-structured domain contains many interrelated concepts as well as inconsistent or irregular information. CFT posits that learning a skill in a structured environment can make 19 transfer to an ill-structured environment particularly difficult. This is because oversimplification in the structured environment leads to a reductive bias whereby complex ideas are simplified to the extent that they are no longer relevant to dynamic ill-structured environments (Spiro, Feltovich, Jacobson, & Coulson, 1992, p. 61). There is evidence that certain interventions for critical evaluation may, indeed, lead to a reductive bias and actually impede students’ ability to evaluate. A study of over 2,700 11- to 18-year-olds found that students who reported having been exposed to training in critical evaluation were more likely to trust a hoax web site than students reporting less training, even though they used more critical evaluation strategies (Metzger et al., 2015). The authors concluded that checklist style methods of teaching critical evaluation direct students regarding what to do without helping them to understand why they’re doing it. Their conclusions echo the assertions of Meola (2004), who advocated for a more contextual approach to critical evaluation. Contextual approaches, argues Metzger et al., “help users make reasoned judgments of information quality based on a more holistic and situated assessment of the information in question” (p. 340). Their recommendations echo Spiro and colleagues’ assertions that cognitive flexibility requires more than simple declarative knowledge. In fact, Spiro, Feltovich and Coulson characterize learners who effectively transfer knowledge in ill-structured domains as having “expansive and flexible world-view(s)” (p. S55) that afford them the ability to grapple with complexity rather than to accept oversimplifications. They are able to synthesize knowledge from multiple partial representations, to perceive interconnectedness between those representations, to assume disorderliness and complexity, to tolerate ambiguity, to place conceptual knowledge in the context of experiences, to pursue knowledge actively, and to favor independent construction of knowledge over automatic trust of authorities (Spiro, Feltovich, & Coulson, 1996). Spiro et al. identify this epistemic worldview as 20 a prefigurative schema that fosters adaptation to the diverse and changing nature of concepts in an ill-structured domain and encourages the constant restructuring of one’s global understanding, competencies reflecting cognitive flexibility. Furthermore, deeper and more transferrable learning occurs when students forego the oversimplified treatment of a subject for deeper examinations from multiple vantage points, “criss-crossing” the landscape of an issue to build a richer and more flexible mental representation of it (Jacobson & Spiro, 1995). Feltovich, Spiro and Coulson (1993) posited 11 principles for instruction to move students from introductory to advanced knowledge in any complex domain. Those principles address several dimensions of the content itself (e.g., whether content is static or dynamic, separable or interactive, universal or conditional) as well as learner differences that influence understanding (e.g., prefigurative schemas about the world and learning, or credence given to authorities as information sources). Principles 1 and 2 relate to a learner’s prior knowledge, including misconceptions of the content, which should be challenged through instruction. Principles 3 and 4 address the problems of conceptual isolation, which are countered by emphasizing connection and dependency between concepts and examining their variations within related clusters of concepts. Principles 5, 6, 8, 9 and 10 address the problem of reductive bias by advocating for the replacement of simplistic analogies in instruction with multiple complex representations. Instruction should include numerous varied cases and emphasize the relationships between cases, allowing for the assembly of new knowledge from various parts and pieces of others, and students should actively apply conceptual knowledge to varied cases, comparing and contrasting them to ascertain patterns that emerge. Principle 7 advances active learning as essential for engaging students in manipulating knowledge and is therefore a condition for the successful application of other criteria. Finally, Principle 11 extends 21 the importance of conceptual construction and reconstruction by encouraging the revisiting of cases and concepts from different points of view. In keeping with these instructional principles, Boyd and Ikpeze (2007) asserted that knowledge must be must be “learned, represented, tried out, and applied in many ways” (p. 233). Constructing situation models from opposing viewpoints on a topic involves cognitive flexibility because readers examine divergent perspectives; consider factors like author expertise, bias, and purpose in evaluating the sources and their viewpoints; and combine these with textbase models and stylistic elements of a text to build a coherent understanding of the issue. Given this, I assume that effective instruction in the critical evaluation of multiple, conflicting web sources should support the development of cognitive flexibility: The effective teacher of reading “assemble[s] a situation-sensitive approach that draws from different elements of knowledge and different prior-case experiences” (Spiro, 2004, p. 655). Similarly, Kolodner (1997) asserted that Knowledge will be more accessible, flexible, deeply learned, and accurate, if learners have the opportunity to encounter multiple situations in which knowledge is used and multiple ways in which similar situations are addressed and if students have the opportunity to reuse and try out knowledge gained through experiences. (p. 63) Given that Bulger (2009) found a positive relationship between the frequency of internet search assignments in school and the skills essential for online comprehension and evaluation, including “the ability to make connections, to see patterns in arguments, identify significant differences, and engage in deep processing” (p. 118), I presuppose that skilled critical evaluation of multiple conflicting online texts necessitates iterated practice in varied contexts. 22 Dialogic Instruction Whereas monologic classrooms place the teacher at the center of knowing and as a provider of information (Davis, 2013), dialogic classrooms distribute knowledge among the group and emphasize the learners’ construction of knowledge through discussion (Barnes, 1976). The traditional Initiation-Response-Feedback structure (Mehan, 1979; Wells & Arauz, 2006), in which teachers consecutively pose questions that have anticipated correct answers, has been established as a less effective form of monologic instruction (Zwiers, 2008). In contrast, dialogic instruction involves the exchange of diverse ideas as students engage more wholly with one another (Applebee, Langer, Nystrand, & Gamoran, 2003). The use of dialogue to affect cognitive growth has been established by researchers in past decades (Applebee et al., 2003; Nystrand, 2006; Nystrand & Gamoran, 1991a, 1991b). Their work found strong effects on student learning in courses in which teachers devoted longer time to discussion and asked more open-ended and follow-up questions. Students recalled more text details and gained a deeper understanding of texts than did students in more didactic, monologic classrooms. Both Applebee and Nystrand argued that dialogic discussions help to shape classroom epistemology (Nystrand, 2006), influencing what is defined as knowledge by students and teachers. The success of dialogic instruction may be related the student engagement it promotes. Several theorists have argued that open discussions leverage adolescents’ emerging independence (Adler, Rougle, Kaiser, & Caughlan, 2004; Alvermann et al., 1996; Nussbaum & Sinatra, 2003), provide opportunities for students to see the relevance of their learning to personal experiences (Davis, 2013), and to take responsibility for their learning (Alvermann et al., 1996). Because, as discussed earlier, critical evaluation is a complex process requiring the 23 simultaneous activation of both cognitive and metacognitive processes, lack of engagement is likely to impede learning considerably. As discussed previously, critical evaluation requires the consideration of many contextual factors influencing the creation and use of a text. Within dialogic discussion, students are encouraged to assert individual opinions and perspectives, thereby bringing multiple contextual factors (i.e., prior opinion, reading goals, interpretations of author biases and expertise, text qualities, personal experiences) to the fore, while providing students experience in negotiating those. Indeed, Kuhn (2015) reports that an essential aspect of effective collaboration is that students “directly engage one another’s thinking” (p. 47). As participants confront the interpretations and critical evaluations of texts that are offered by individual participants, they gain cognitive flexibility for comprehending and evaluating future texts, in addition to practicing the dispositions of mind that critical evaluation requires (Zwiers, 2008). Dialogic instruction may be particularly effective for poorer comprehenders, who, in at least one study by Macedo-Rouet and colleagues (2013), appeared to benefit specifically from its spoken modality. Their finding aligns with Vygotsky’s view that cognitive growth occurs more readily when learners verbalize, explain, and defend their thoughts to others (1978). Placing poorer comprehenders in a dialogic setting affords them the opportunity to observe how strong comprehenders articulate and justify their thinking. Moreover, dialogue is not inherently dependent on the decoding and comprehension skills of individual participants, and therefore provides an alternative mode through which meaning can be constructed. A poorer comprehender may begin by simply listening and observing. In time, having established a stronger understanding of the text through the contributions of others, he or she may gradually contribute verbally. In addition, as strong comprehenders refer directly to the textbase in their 24 justifications, they may cue poorer comprehenders to re-read, question the text, build inferences, and make predictions that enhance understanding. Therefore, dialogic discussion affords students of varying comprehension levels the scaffolding required for them to experience cognitive growth within their zone of proximal development (Nystrand, 2006; Vygotsky, 1978). 25 CHAPTER 3: AN INSTRUCTIONAL UNIT ON CRITICAL EVALUATION To combine the most effective elements of instruction for teaching critical evaluation, I designed an instructional unit based on several principles grounded in the literature discussed in Chapter 2. These principles served as the groundwork from which the content and pedagogy of the instructional unit was built. I outline the design principles below, and subsequently describe the primary activity structures emerging from those principles. Finally, I lay out an overview of the unit’s sequence. Design Principles Drawn from the research described above are several principles for the content and pedagogy of an instructional unit to improve students’ critical evaluation skills. These were drawn from previous studies demonstrating the most promising interventions for improving critical evaluation skills in a classroom setting with middle school students. They incorporated theory and research about teaching critical evaluation in a multipronged approach that was integrated into the regular school curriculum, that attended to the affordances and constraints of an actual classroom setting, and that induced students to approach web sources with a default critical mindset in learning from the web. 1. Because sourcing is an essential aspect of comprehending multiple conflicting sources on the web (Afflerbach & Cho, 1998; Rouet, 2006), and because critical evaluation has been linked to deeper comprehension (Braasch et al., 2013, 2012; Britt et al., 2014; Strømsø, Bråten, & Britt, 2010), interventions to amplify students’ attention to source features may improve their critical evaluation skills. Therefore, attention to source features was a central aspect of this instructional unit. 26 2. Because presenting students with multiple perspectives and conflicting information is considered beneficial for promoting online literacy skills in general (Britt et al., 2014; Ferguson & Bråten, 2013), the reading materials chosen for this instructional unit presented conflicting or alternate perspectives. Moreover, because an intervention using conflicting cases increased students’ critical evaluation skills (Braasch et al., 2013), the unit introduced students to web evaluation using a contrasting cases activity. 3. Because cognitive flexibility is required for the effective use of multiple texts (Boyd & Ikpeze, 2007) and web sources (Deschryver & Spiro, 2009; Kim & Afflerbach, 2014; Richter, 2011), because the principles for instruction of complex concepts in illstructured environments advocate for the active and engaged examination of concepts in clustered, interdependent, and varied multiple cases in multiple contexts and from multiple perspectives (Feltovich, Spiro & Coulson, 1993), and because previous research supports the development of cognitive flexibility through repeated exposure to multiple representations of problems with varied contextual factors influencing their solutions (Boyd & Ikpeze, 2007), this unit required students to repeatedly practice evaluating sources whose usefulness and reliability is influenced by a variety of contextual features, especially source features such as author expertise, author bias, information sourcing, site affiliations, and site purpose. In addition, because the transfer of expert skills requires practice within the context of authentic problem-solving (Spiro & Deschryver, 2009), students practiced evaluating sources that presented real world issues in their original online formats. 4. Because metacognitive awareness is essential for multiple text comprehension (e.g., Kiili et al., 2009; Pressley & Gaskins, 2006; Rouet, Ros, Goumi, Macedo-Rouet, & Dinet, 27 2011) and for evaluating multiple online sources (Goldman, 2012; Stadtler & Bromme, 2007; Wiley et al., 2009), and because increasing students’ metacognitive awareness has improved both comprehension (e.g., Cubukcu, 2008; Johnson, Archibald, & Tenenbaum, 2010; Palincsar & Brown, 1984) and critical evaluation (e.g., Gerjets, Kammerer, & Werner, 2010; Macedo-Rouet et al., 2013; Mason et al., 2014; Sanchez et al., 2006) this instructional unit scaffolded the acquisition of those skills through the use of annotation, a strategic reading practice (Porter-O’Donnell, 2014). 5. Because dialogic instruction has been shown to increase students’ independence and responsibility for learning (Davis, 2013; Wells & Arauz, 2006) and has been credited for improving student learning in general (Applebee et al., 2003; Reznitskaya et al., 2009); and because student-centered construction of knowledge through dialogue has been shown to improve the application of learning to new situations (Palincsar & Brown, 1984), I chose to incorporate dialogic instruction into the instructional cycle. This would provide support for struggling learners, who might respond more positively to verbal rather than written scaffolds (Macedo-Rouet et al., 2013), and who might benefit from stronger readers’ verbalizations of strategic reading strategies and critical thinking in general (Nystrand, 2006; Vygotsky, 1978). It also provided opportunities for individual students to benefit from the shared prior knowledge of the group, and to entertain opposing viewpoints, both of which can influence comprehension of multiple conflicting sources (Bråten et al., 2011; Eastin et al., 2006; Fogg et al., 2003; Lord et al., 1979; Taber & Lodge, 2006; van Strien et al., 2014; Wineburg, 1998). 6. Because moving students from novice to expert skill requires a gradual increase of task complexity through supported instruction placing students in a Zone of Proximal 28 Development appropriate for their learning (Vygotsky, 1978), this instructional unit scaffolded critical evaluation first through modeling, then through social meaning construction, to application in an independent online research project. In addition, the evaluation practice spiraled from lesser to greater complexity over the course of the unit. Activity Structures Drawing on these design principles, I developed three core activity structures for this unit. The first two structures comprised the two-part practice cycle, and the third functioned within the critical evaluation lesson that began the instructional unit. Structure 1: Inquiry-of-the-week The first activity structure was an adaptation of an article-of-the-week assignment proposed by Gallagher (Gallagher & Allington, 2009) for improving students’ comprehension of nonfiction text, which uses nonfiction texts from a variety of real world sources: newspaper articles, magazine articles, speeches, essays, editorials, and blog posts. Authentic texts afford opportunities for students to grapple with the multiple complexities of texts in context. Each week students independently read, annotate, and respond to an article. Here, individual practice is important because the goal is to motivate students to think critically and independently about the sources they encounter on their own. Prior to the first week’s assignment, Gallagher coaches students through the process of close reading by having the class annotate a nonfiction text together while using discussion to make transparent the strategic reading strategies relevant to the task. These include having the students “Look for patterns in the things (they’ve) noticed about the text—repetitions, contradictions, similarities” and “Ask questions about the patterns (they’ve) noticed—especially ‘how’ and ‘why’” (Gallagher & Allington, 2009, p. 101). Moreover, Gallagher notes that providing historical context, along with an author’s background 29 and purpose, are two methods by which teachers frame texts to make them more accessible. Since the “how and why” of a text is integrally linked to author identity and purpose, these prompts sharpen students’ attention to the purpose and author behind each text, an important goal of the instruction. In addition, modeling the annotation process while drawing students’ attention to these features builds their understanding of metacognitive strategies for critical evaluation. Finally, Gallagher advances the importance of modeling strategy instruction to achieve “the proper balance between underteaching and overteaching those metacognitive skills that good readers employ” (p. 107). Whereas underteaching fails to make transparent the skills needed to tackle challenging texts and may lead to students’ abandoning those texts altogether, overteaching can lead to interruption of flow and emphasis on surface level skills in lieu of deeper comprehension. Since application of critical evaluation skills in authentic online reading tasks is particularly problematic, Gallagher’s attention to scaffolding just the right amount of support for student motivation and success fulfills the need for instruction that places students within Vygotsky’s zone of proximal development. Therefore, in accordance with Gallagher’s bid to incorporate into comprehension instruction close reading practice, annotation, and text framing; increased frequency of practice with a variety of text types; and a balance between overteaching and underteaching useful strategies, I appropriated his article-of-the-week structure for teaching critical evaluation of online texts. There were, however, several key differences in my revised version of the activity. Whereas Gallagher provided paper copies of texts and encouraged students to read “with pencil in hand” (p. 101), I used digital texts to give students an authentic online reading experience. This is important because, as discussed earlier, the critical evaluation of online sources is 30 integrally dependent on contextual factors; to learn effective critical evaluation, students require practice considering those. Therefore, texts presented to students in the activities were online texts in their original format, and annotation was done using an online annotation tool, Diigo (www. Diigo.com), to retain some authenticity of the online reading experience. Finally, I chose to provide two alternative sentence starters (“I notice…” and “I wonder…”) to structure students’ thinking and narrow the focus of their annotations. A second distinction in my use of the article-of-the-week activity structure was in the quantity and type of articles that students read. Although Gallagher asks students to read and respond to a single text at a time, I expanded the structure to include two conflicting texts for each reading assignment. Because critical evaluation of online texts is complicated by the conflicting information and perspectives that they present, it was important for students to confront those challenges in varied and repeated instances. In addition, I renamed this activity the inquiry-of-the-week to encompass the broader goal of critical evaluation in the context of online learning from multiple texts. A third, but important, distinction in my adaptation of Gallagher’s strategy is in the specific criteria by which text framing was emphasized in instruction. Instead of providing students with background information about a text’s author and purpose prior to reading, students were encouraged to actively locate information about author and purpose on their own. In addition, the initial instruction of sourcing occurred within a contrasting cases framework, which differed from Gallagher’s demonstration/discussion format for reading strategy instruction. (See discussion of prior research on contrasting cases above and specifics of that instructional session below.) However, in the weekly inquiry assignments that occurred after the initial critical evaluation lesson, students were directed to examine source features that helped to 31 frame each online text: identifying the author and publication source, questioning an author’s expertise and perspective, and weighing the likelihood of bias. The active search for such information was designed to assist students in building source models of individual online texts, thereby serving both comprehension and evaluation. Finally, whereas Gallagher’s model requires that students annotate and construct responses to texts within the same assignment, I divided the reading/annotation activity from the response-writing activity by inserting a Socratic seminar between the two experiences. My students read and annotated independently to prepare for the group seminar; these annotations served to launch student contributions and provided evidence for their assertions in the discussion. Structure 2: Socratic Seminar A second activity structure in the instructional unit leveraged the affordances of dialogic discussion to teach critical evaluation. The dialogic discussion format I have used most successfully is the Socratic seminar (Hinchman & Sheridan-Thomas, 2008). It encourages the open exchange of ideas through its specific structure: Students sit in a circle or semicircle, with the teacher joining the circle as an equal participant, thereby de-emphasizing the teacher as sole distributor of knowledge. Students do not raise hands, but instead take turns speaking; therefore, the exchange of ideas between students is direct rather than mediated through a teacher authority. Students focus their attention toward the group rather than toward the teacher when they speak, and are encouraged to sustain the conversation by questioning each other, asking for clarification, and extending the ideas of others with minimal teacher direction or orchestration. The combination of these elements fosters student independence and self-directed learning, disperses authority among participants rather than concentrating it in the teacher, and emphasizes 32 the multiple perspectives that shape comprehension and evaluation. These are important aspects of the instruction because critical evaluation is served by reader engagement, by a critical stance toward information in general, and by an epistemic belief that not all knowledge is absolute (Barzilai & Zohar, 2012; Feltovich, Spiro & Couson, 1993; Kienhues, Bromme, & Stahl, 2008; Muis et al., 2015). Finally, students were required to refer to the text(s) being discussed for evidence and support, and at the end of each seminar, the teacher prompted participants to verbally reflect on their learning, usually by sharing their “takeaway of the day.” This debriefing required the collective metacognitive reflection that critical evaluation demands. The latter is important because, as Kuhn asserts, “socially mediated talk about knowing may be a key factor in conferring any benefit the collaborative activity requires” (2015, p. 49). My aim was, therefore, to leverage the metacognitive talk embedded in the Socratic seminar to guide students toward metacognitive thinking about critical evaluation. Structure 3: Contrasting Cases Scenario I adapted Braasch and colleagues’ (2013) contrasting cases scenario as a framework for the initial critical evaluation lesson included in this instructional unit. Recall that in their intervention, participants examined several texts. Some of those were identified as online texts but all were presented on paper. Participants examined the think-aloud responses of two students to the texts, being told that one was a “better, more critical” student and the other was a “poorer, less critical” student, but without identifying each. The think-aloud responses contained strategy protocols of better and poorer critical evaluators, and participants were to identify which strategies were used by each student, label it as a sound or poor strategy, and explain why. Participants discussed their answers in small groups and in a large group. 33 My instructional lesson drew on Braasch and colleagues’ (2013) intervention, with two notable differences. One involves the method by which texts were presented, and the other involves a posttask measure of the unit’s effectiveness. In the Braasch intervention, students read texts originally from web sources, but retyped on paper. Source information was summarized above each text for readers to consider while evaluating. In contrast, I provided links to authentic web sources for students to examine, and did not highlight source information for them. Although this more open method provided less control of the process, it was more authentic. Because, as previously discussed, a central problem of critical evaluation is in the difficulty of its application to authentic tasks, I aligned the activities more closely with actual search tasks in the hopes that evaluation behaviors would transfer more readily. Interestingly, Braasch and colleagues advanced their findings as possible evidence of far transfer, since the intervention topic (cell phone use) differed in domain from the measure topic (weather systems). Given that web sources are digital, and that web texts rarely provide summaries of source information for the reader prior to critical evaluation, I find their claim of far transfer tenuous. In addition, because prior research has found longer-term application of critical evaluation specifically to be problematic (Colwell et al., 2013), it seems important to test the benefits of intervention beyond a day or two. Braasch and colleagues measured their intervention one day after conducting the lesson, and interpreted their results as evidence of longer-term transfer, stating that “it seems unlikely that the source evaluation benefits emerged specifically because application was tested 1 day later” (p. 193). To test the longer effects of instruction, this study measured critical evaluation skills several days after the last cycle of a multiweek unit, as well as in an authentic, student-directed search task some weeks after the instructional cycle concluded. 34 Figure 1. Overview of the study. Pre-unit measures of critical evaluation and source model strength precede instruction. Unit begins with introductory critical evaluation lesson followed by three consecutive weeks of inquiry-of-the-week annotations and Socratic seminars. Post-unit measures of critical evaluation and source model strength follow the unit, and students apply evaluation skills to an independent inquiry task in the weeks following instruction. Overview of the Instructional Unit The unit began with an initial lesson leveraging the effectiveness of contrasting cases for learning critical evaluation skills. Then, within the framework of the first and second activity structures outlined above, students experienced three iterative practice cycles for improving their critical evaluation skills. Figure 1 provides a visual overview of the unit, including the initial contrasting cases lesson and the three practice cycles, each consisting of one inquiry-of-the-week and one Socratic seminar on a single topic. With each new practice cycle, the inquiry topic 35 changed and new evaluation complexities were introduced. Therefore, the tasks that students completed within each cycle were designed to (a) draw their attention to source features through the examination of multiple texts with conflicting perspectives or information; (b) motivate their engagement in critical evaluation through the choice of controversial issues as subject matter; (c) provide learning scaffolds to move students from the simple identification of source information to more complex consideration of author motives, bias, and purpose; and (d) move the students recursively between scaffolded support of critical evaluation skills via group inquiry and greater independence in evaluating sources via individual inquiry. Research Questions This unit provided the setting for students to improve their critical evaluation skills through instruction grounded in the design principles outlined above. The study sought to answer the following research questions: 1. Do students participating in an instructional unit focusing on the critical evaluation of web sites show change in their evaluation of multiple conflicting sources? a. Do they show change in their ability to rate web sites based on source reliability? b. Do they increase their attention to source features in justifying their site ratings? c. Do they build stronger source models after the critical evaluation instruction? 2. After participating in the instructional unit, to what extent and in what ways do students evaluate the source features of web sites when the evaluation process is embedded in an online inquiry task? 3. What do students’ individual annotations and contributions to dialogic discussions reveal about the critical evaluation processes they use to judge the usefulness and reliability of web sites? 36 CHAPTER 4: METHOD This study tested the efficacy of an instructional unit designed to improve critical evaluation skills through the comparison of student performance on a critical evaluation task administered before and after the unit. That task measured students’ ability to evaluate the reliability of multiple conflicting web sites, to provide justifications that reflect attention to source features, and to examine the association between references to source features and the strength of students’ mental source model. Then, to measure the students’ critical evaluation skills as they were embedded in an independent inquiry task, I analyzed students’ source selection notes and annotated bibliographies of the final sources students used for a research project on a self-selected controversial issue, noting both general references to evaluation criteria, as well as specific criteria related to authorship and sourcing. Process data was collected over the course of the instructional unit to provide insight into the critical evaluation processes that students used as they examined sources. These data comprised individual annotations of web sites and transcripts of dialogic discussions about those web sites. Setting The location of this study was the middle school where I taught two sections of eighth grade Language Arts and acted as the middle school library media specialist and technology integrationist for the remaining four periods of the school day. The district is predominantly white and middle-class, with an average family income, according to City-Data.com, of $53,569 in 2015, slightly above the state average. The racial composition of the school at the time of this study was 89% white, 6% black, 3% Hispanic, 1.5% Asian, and less than 1% Native American. 37 Students in the district attend one of three elementary schools, but combine to attend one middle school. Therefore, most eighth graders have attended the school for two full years and will move together to the high school. Consequently, the student makeup in eighth grade is fairly stable, and students are generally comfortable with their peers. Since 2014 all students have been provided one-to-one access to Chromebooks in their core subject area classes: language arts, mathematics, science, and social studies. The students in this study were in the second year of Chromebook use, and most were comfortable with the basic functions of the devices as well as with Google Apps for Education, used schoolwide. Participants The participants were my eighth grade students, including one section of regular education students of varying abilities and one section of advanced language arts students. To be admitted to the advanced section students must earn A’s in language arts in seventh grade and, in the spring of seventh grade, must test at the 1100 Lexile level on the reading comprehension measure of the Northwest Evaluation Association’s (NWEA) Measure of Academic Progress, also known as the MAP test. The computer-administered MAP test is comprised of 54 multiplechoice questions on short reading passages. The test is computer-adaptive, meaning questions increase in difficulty as students answer them correctly, and decrease in difficulty as students answer incorrectly. In addition, a student’s final score is based on an algorithm that considers the standard deviation of the latest test (taken in the spring of seventh grade), as well as two previous scores from sixth grade and one from seventh, to determine the student’s final score. Although no multiple-choice assessment predicts a student’s online comprehension perfectly, there is evidence that online assessments are more likely to align with the challenges that online reading presents (Paek, 2005), and the MAP has been shown to predict students’ ACT scores as 38 early as fifth grade (Brown, 2014). Students also apply for admission to the course by writing a one-page essay explaining why they wish to enroll. Therefore, students admitted to advanced language arts display the desire and the ability to succeed in it. Students in the regular section of language arts were of varying ability but did not include special education students. Although the choice of eighth graders for this study was in part a matter of convenience, research shows that students in the middle grades experience considerable physical, intellectual, social, and emotional transformation, and increase their ability to apply analytical thinking to the critical evaluation process (Kuhn, 2000; Metzger et al., 2015). This makes eighth grade a logical level at which to integrate more complex evaluative skills into the curriculum. In addition, because our high school curriculum makes significantly greater demands on students in the areas of research and information literacy, it makes sense to provide specific instruction in critical evaluation before students are faced with those complexities at the high school level. The Instructional Unit This section includes details of the introductory critical evaluation lesson, the introductory lesson on web page annotation using Diigo, and explanation of both the inquiry-ofthe-week and the Socratic seminar elements of the practice cycle, as well as an overview of subsequent practice cycles. Critical Evaluation Lesson In the initial contrasting cases scenario students examined two preselected web sites on a controversial issue, one more reliable and one less reliable. The students worked with their table partners to collaboratively complete an activity sheet on the sites (see Appendix A). The sheet listed in random order the evaluation statements of two fictional students to the sites, with Student A described as a skilled evaluator and Student B described as an unskilled evaluator. 39 The student pairs worked together to ascribe each evaluation statement to a student by writing S (skilled) or U (unskilled) before each statement. After the student pairs completed the activity chart, a large group discussion took place in which student pairs shared the logic of their answers, and I intervened where needed to ensure students correctly attributed statements to good and poor evaluators. In the context of this discussion I also asked students to summarize the “takeaway” of each statement, which was defined as “what we learn from each of these examples about evaluation web sites.” As pairs of students shared their thinking about different items, I summarized the students’ takeaways in simple statements, listing them on the board while the students copied them to their own papers. Annotation Lesson At the beginning of the annotation lesson, using a projector showing a sample web site, I demonstrated how to create comments on the web page using Diigo, and then demonstrated annotation of a sample web site article. During the demonstration I prompted students to practice evaluation behaviors listed in our previous lesson (searching for author identity, considering the author’s expertise, determining the publication source and type, and questioning the purpose and intent of the article) by beginning their annotations with one of two sentence starters: “I notice…” and “I wonder…” The students practiced annotating a second web site using these two sentence starters. While students worked, I circulated the class to assist them, and students were encouraged to assist each other. By the end of the class period, students had worked through technical questions and, upon inquiry by the teacher, appeared familiar enough with Diigo to use it for the inquiry-of-the-week assignment. The students were encouraged to see the teacher for additional help during the school advisory period or before or after the school 40 day for further assistance as needed. Two students sought additional help after class, and some students sought assistance from fellow classmates over the course of the unit. Inquiry-of-the-week Following the annotation lesson students were assigned to read and annotate two web site articles on a controversial issue. This assignment was disseminated via a Google Classroom document, a sample of which is shown in Appendix B. At the top of the document I provided a one-paragraph introduction to the assignment, which summarized the controversial issue and provided basic background knowledge. This was to provide a minimum of prior knowledge to all students. Controversial issues were chosen to appeal to students’ interest and included the following issues: a) whether high school football is too dangerous for students and should therefore be discontinued as a school sport; b) whether virtual schools should replace brick and mortar schools; and c) whether video games are beneficial for teenage brain development. None of the issues was heavily scientific, which would have disadvantaged students with minimal prior knowledge of the topic, and all were likely to draw out a diversity of opinions. Below the introductory paragraph were links to two online articles. The pairs of sites assigned for each topic are shown in Appendix C. In each pair, one article argued one side of the issue; the other argued the opposing side. The articles varied with regard to reliability and sourcing information. The students annotated each article using Diigo, and linked their finished annotations to the Google Classroom document. Students had three evenings to complete the assignment. This assignment comprised the first part of practice cycle one. Socratic Seminar The Socratic seminars took place the day after the inquiry-of-the-week assignments were due, and students were required to complete the inquiry-of-the-week annotations to participate. 41 On the day of the seminar the desks were arranged in two semicircles; students sitting in the center circle were seated in chairs, and students in the outer circle sat on desktops to facilitate their elevated observation of the inner circle. Students opened their annotations of the web sites using their Chromebooks. Because students were already familiar with the seminar format, a brief review of guidelines was enough to begin the first seminar: (a) no raising hands; instead, take turns speaking; (b) refer directly to the text as much as possible to support your assertions; (c) take responsibility for your own and others’ participation by speaking up respectfully while leaving time for others to do the same; (d) help to move our collective understanding forward by asking questions or clarification of others, and by extending or building on others’ ideas; (e) speak to the group, not to the teacher. I began the seminar with the following question: When considering reliability, what should a good critical reader notice about THIS site? Wording the question to allow for open-ended responses provided space for students to consider various contextual factors that might influence evaluation. Rather than responding to questions about specific evaluation criteria, students generated evaluation criteria organically, thereby encouraging varied perspectives and bringing forth differences of opinion. Since the teacher in a Socratic seminar is expected to help guide but not to drive the discussion, I intervened only to (a) redirect students when they veered from the seminar guidelines; (b) underscore and affirm contributions that showed attention to important source features of the articles; or (c) follow up insightful student contributions with questions that offered opportunities for deeper critical evaluation. Occasionally I also offered more didactic explanations to scaffold the students’ background knowledge or to help them make sense of confusing content. For homework students wrote three short paragraphs summarizing their thoughts about the two web sites using a template provided by the teacher (see Appendix D). 42 Subsequent Practice Cycles The two subsequent practice cycles repeated the latter half of the process above, including the inquiry-of-the-week (IoW) annotation assignment and the Socratic seminar. In both cases the IoW was assigned on a Monday and was due the following Thursday. The Socratic seminars took place the following day (Friday). Therefore, during the span of the unit, students did three sets of article annotations and participated in three Socratic seminars about three controversial issues, writing a template-based response after each seminar. Data Sources Data for examination of the four research questions outlined above was collected from the pre-unit and post-unit critical evaluation task, from the pre-unit and post-unit source model measure, from process data including videos and transcripts of Socratic seminars, and from the independent inquiry assignment. In the following sections I describe data collection and analysis for each of these sources. Critical Evaluation Task A critical evaluation task administered before and after the unit provided data to address Research Question 1, Do students participating in an instructional unit focusing on the critical evaluation of web sites show improvement in their evaluation of multiple conflicting sources? To address 1a, Do students show improvement in their ability to rate web sites based on source reliability?, students rated the reliability of four preselected web sites. To address 1b, Do students increase their attention to source features in justifying their ratings of those sources?, students justified their ratings by listing reasons to support their evaluation of each site. For each critical evaluation task students examined four sites on one of two issues. To ensure that topics did not influence differences from pre-unit to post-unit tasks, the two topics were balanced 43 among students in both classes. Approximately half of the students, therefore, evaluated sites about cell phones in the pre-unit task, and the other half evaluated sites about homework; this order was reversed in the post-unit task. I aligned the topics as much as possible vis-à-vis student interest, complexity, and type. Both topics (Should students be allowed to use cell phones in school? and Should homework be banned?) related to school policies that affect students and were, therefore, relevant to them. Neither topic introduced complex scientific information that would disadvantage lower level readers. Moreover, discrepant levels of prior knowledge were less likely with a school-related topic than with many others. Of the four sites students evaluated in each task, two were more reliable and two were less reliable. Here I defined reliability primarily in terms of author affiliations, expertise and bias, as well as availability, relevance, and quality of information sourcing. Specifically, more reliable sites had authors (or secondary information sources) with strong credentials providing considerable expertise on the topics, whereas less reliable sources had authors whose credentials did not provide expertise, who were potentially biased, or whose credentials were not revealed. Reliable sites also used relevant, high quality information sources and made those available to the reader. Perspectives on both sides of the issues (pro and con) were equally represented, including one more reliable site and one less reliable site for each stance on the issues. Evaluation tasks were administered in a 50-minute class period during the week prior to the instructional unit and again in a 50-minute period in the week following the unit. Students accessed the sites via a Google Form presenting links to the four sites that loaded in random order, and had access to the open internet while completing the evaluation forms, which were collected at the end of the class period. Students worked individually and without discussion. To help ensure students completed the task within a class period, I notified the students when 30 44 and 45 minute intervals had passed. Students who finished before the end of the class period closed their Chromebooks and read silently. Data collected in the pre-unit and post-unit evaluation tasks consisted of site ratings and justifications for those ratings, written on an evaluation form provided for each site (see Appendix G). After examining each site, students first listed favorable and unfavorable aspects of the site in a bulleted list. There was no limit to the number of items allowed in each list, and students were encouraged to be as specific as possible in their explanations. Then, taking into consideration the favorable and unfavorable aspects noted, students rated each site on a 6-point Likert scale ranging from “very reliable” to “very unreliable.” Expert reliability ratings were gathered from a group of six seasoned internet users to serve as a standard against which to judge students’ ratings. Experts included a university professor of educational technology and new literacies; two veteran school media specialists and information literacy instructors; the head educational technologist in a public school district; and two veteran secondary level language arts teachers with longtime experience teaching online research. Those expert ratings were averaged to determine a single reliability rating for each site. Each student’s ratings were compared to the average expert rating for each site by calculating the absolute value of the difference between the two ratings. An absolute value close to zero indicated a student rating close to the experts’ average, with higher absolute values indicating greater differences in rating. The four absolute value differences (one for each rated site) were averaged for each student, resulting in a mean distance from experts’ score for each student. A paired-samples t-test was conducted to compare distance from expert ratings in the pre-unit evaluation task to distance from expert ratings in the post-unit evaluation. A separate 45 paired t-test comparing students’ average site ratings in the pre-unit and post-unit tasks provided an additional perspective on student changes with respect to site ratings. Analysis of the evaluation justifications offered by students in the pre-unit and post-unit, tasks involved a grounded coding process whereby codes emerged from the data organically. Each bulleted item was viewed as a separate segment for coding, unless a statement clearly contained references to two separate evaluation criteria connected by “and.” Coding involved several passes through the data to determine main themes and subcodes appropriate for finer grained distinctions within those themes. The final coding scheme, including main themes and subcategories, is discussed in greater detail in Chapter 4. Once the final coding scheme was finalized, a doctoral student in educational technology was trained in its application, coding 15% of the pre-unit, post-unit, and embedded task justifications for an interrater agreement of 83%. To determine whether students’ attention to specific evaluation criteria changed from preunit to post-unit tasks, the frequency of students’ references to different evaluation criteria were compiled using the software program NVivo. Because my primary interest was to establish whether students’ attention shifted from some categories of evaluation criteria to others, I converted frequencies into percentages of total criteria cited for each task. I also converted frequencies into mean references per student per site to facilitate a finer grained comparison of pre-unit and post-unit results at the category, subcategory, and code levels. A paired t-test compared the frequencies of Key Criteria cited in the pre-unit and post-unit tasks, and a Chisquare test of independence compared the proportions of references to specific subcategories within Key Criteria. 46 Source Model Measure In response to Research Question 1c, Do students build stronger source models subsequent to the instructional unit? I designed a measure to test the students’ recollection of specific source features after completing each evaluation task (See Appendix H). The measure presented students with two open-ended questions about each site the students had evaluated the previous day. The four sites were represented by small color thumbnails large enough to recognize visual elements (banners, sidebars, colors, and larger graphic elements), but with the titles and text blurred to make them illegible. This was to ensure the students’ responses were based on recollections of the sites rather than on rereading titles, which contained clues about site stance and authorship. For each site the students responded to the same two questions: (1) What was this site's OPINION about homework? If you don't remember, just say that; and (2) What do you remember about the AUTHOR, SPONSOR, or CREATOR of this site? List as many things as you can remember. The four sites, with a total of eight questions (two per site), were presented on a single Google Form so that students could examine all four thumbnails simultaneously and could record responses in whatever order they wished. Students completed the source model measure on the day immediately following each evaluation task at the beginning of the class period. Their responses comprised the data for comparison of the source models created by students before and after the instructional unit. Frequencies for student recollections of site stance and source features provided data for comparing the strength of students’ source models in the pre-unit and post-unit measures. I compiled frequencies by student of correct, incorrect, and unable-to-recall instances for both recollection types. Paired t-tests compared the correct recollections of site stance and authorship features. 47 Embedded Evaluation Data addressing Research Question 2, After participating in the instructional unit, to what extent and in what ways do students evaluate the source features of web sites when the evaluation process is embedded in an online inquiry task?, consisted of student work generated in an independent inquiry assignment that began one week after students finished the instructional unit. Students chose a controversial issue and composed an argumentative essay on that issue. They conducted online information searches on their topics, evaluated potential resources on a working source sheet, compiled an annotated bibliography with at least five reliable sources, and composed an argumentative essay. In their annotated bibliographies students accompanied each reference with one paragraph explaining why they chose the source for their project. The working source sheets required students to list at least eight potential sources as they conducted their research, and these were to include at least two rejected sources (see Appendix I). For each potential source the students were to list three reasons justifying their assessment of the site as reliable or unreliable. These working source sheets, along with the final one-paragraph justifications of sources used included on their annotated bibliographies, provided data for determining the extent to which students considered source features in their evaluations of sites in the independent project. Appendix J shows the student directions for creating their final annotated bibliographies. To code evaluation justifications in the independent inquiry assignment, I first coded the students’ annotated bibliographies using the same coding scheme applied to evaluation justifications in the pre-unit and post-unit evaluation tasks. Then, I cross-referenced each student’s justification paragraphs with their source notes; this was (a) to add justifications for rejected sources; and (b) to add justifications that students may have included in the source notes 48 but not in the final annotated bibliographies. This allowed for inclusion of justifications the students made for all potential sources, and revealed the number of sites each student had examined in the inquiry process. Although this data did not determine whether students increased their attention to source features as a consequence of the instruction, it did provide limited evidence for whether students attended to source features after a postinstruction lapse of at least one week, and also revealed the types of source features students attended to when evaluation was embedded in an actual inquiry assignment. A paired samples t-test compared the percentages of Key Criteria cited by each student in the pre-unit and embedded tasks, and a Chisquare test of independence compared the proportions of specific subcategories within Key Criteria in the pre-unit and embedded tasks. Process Data To examine Research Question 3, What do students’ individual annotations and contributions to group discussion reveal about the critical evaluation processes they use to judge the reliability of web sites?, two primary types of process data were collected over the course of the instructional unit—students’ written assignments and transcripts of dialogic discussions. The extent to which these data would reveal evaluation processes was unclear prior to the study; therefore, the following written data were gathered for potential analysis but not, in the end, analyzed: (a) annotations of two sites recorded in Diigo for each inquiry-of-the-week assignment (six sites per student over the course of the unit); and (b) a postseminar response to each pair of sites discussed in three Socratic seminar discussions (three postseminar responses over the course of the unit). This generated a total of six site annotations and three seminar responses per student (see Appendices B and D). The second type of process data collected were video recordings of the Socratic seminar discussions, which reflected the students’ vocalization 49 of evaluative thinking and, especially, attention to source characteristics. Video data was collected for three Socratic seminars with each class. These videos provided insight into the individual and collective thinking of students as they evaluated selected sites. Because the preunit, post-unit, and embedded task data showed marked differences between the evaluation criteria and investigative behaviors of students in the regular class, a group of students for whom instruction is arguably more necessary and more challenging, I chose to analyze the three regular class sessions to examine how those conversations foreshadowed student growth. I began by transcribing each session nearly verbatim. I then perused the transcripts to trace major themes, looking especially for discussion related to the evaluation categories Authorship, Information Sourcing, and Investigation, which had emerged as important foci of students’ evaluation processes in the post-unit and embedded evaluation tasks and as principal topics of discussion in the sessions. I attributed these three categories to exchanges within each seminar discussion where appropriate, and then traced the development of students’ thinking in each category across the three seminars. This analysis comprises Chapter 6. Because time for additional discussion with the advanced group allowed students to reflect on the value and nature of the seminar experience over the unit, I also transcribed the final segment of that session for the insight it provided into the effectiveness of the dialogic discussion from a students’ perspective, which is addressed in Chapter 7. 50 CHAPTER 5: CHANGES IN STUDENTS’ WEB SITE EVALUATION In this chapter I return to Research Questions 1 and 2 to examine student changes that emerged after the instructional unit. This includes (a) changes demonstrated by comparison of pre-unit and post-unit evaluation tasks, including both site ratings and evaluation criteria justifying those ratings; (b) examination of evaluation criteria cited by students in the independent inquiry task, in which evaluation was embedded; and (c) examination of source model measures before and after the instruction. Site Ratings To address Research Question 1a, Do students show improvement in their ability to rate web sites based on reliability?, students rated the reliability of four web sites about a controversial issue before and after the instructional unit using a 6-point Likert scale ranging from very reliable to very unreliable. Of the four sites, two presented the issue from a proleaning perspective and two presented a con-leaning perspective. One of the two sites from each perspective was more reliable; the other was less reliable. Recall that expert reliability ratings by six seasoned internet served as a standard against which to judge students’ ratings. Those expert ratings were averaged to determine a single reliability rating for each site, presented in Table 1 along with students’ average pre-unit and post-unit ratings for each site. The absolute value differences between student ratings and average expert ratings were averaged for each student, resulting in a mean distance from experts’ score for each student. A paired-samples t-test compared distance from expert ratings in the preunit evaluation task to distance from expert ratings in the post-unit evaluation. There was a 51 significant difference in distance from expert ratings pre to post, t=4.131(49), p <.001, d=.8491 with a pre-unit task mean difference of 1.61 (SD=.49) and a post-unit task mean difference of 1.23 (SD=.40). Thus, students’ site ratings aligned more closely to expert ratings after the instructional unit. Table 1. Mean site ratings by experts and students _____________________________________________________________________________ Topic and Site Mean Expert Rating Mean Student Rating Mean Student Rating Pre-unit Post-unit _____________________________________________________________________________ Homework Hi/Con: Stanford 5.17 5.17 4.77 Hi/Pro: CPE 5.50 3.96 3.65 Lo/Pro: Leapfrog 2.33 4.92 3.73 Lo/Con: BDV 1.83 3.25 2.08 Cell Phones Hi/Pro: ASCD 5.83 4.73 3.70 Hi/Con: Guardian 4.50 4.31 4.09 Lo/Con: Care2 2.50 4.23 3.04 Lo/Pro: Nielsen 1.83 3.58 2.00 ______________________________________________________________________________ Note. Sites are ordered from highest to lowest rated by expert group. Pro describes sites in favor of homework and cell phones in school; Con describes sites against homework and cell phones in school. Hi describes sites of higher reliability as assessed by expert group; Lo describes sites of lower reliability. Additional examination of the pre-unit and post-unit site ratings showed that students rated all sites more critically in the post-unit evaluation task than in the pre-unit evaluation task (See Figure 2). A paired samples t-test showed a significant difference between the students’ mean pre-unit site ratings (M=4.27, SD=.68) and their mean post-unit site ratings (M=3.35, SD=.77), t=7.143(49), p<.001, d=1.2661. This suggests that, overall, students increased their skepticism of online sources in the post-unit task. Note that, when evaluating sites recognized by expert raters as higher in quality, such as those from Stanford University or the Association for 1 Effect size calculated as Cohen’s d based on the average SD from two means (Becker 2000). 52 Supervision and Curriculum Development, students’ increased criticality in the post-unit task moved their ratings farther from the experts’ (See Table 1). One explanation for this is that middle school students’ general knowledge of institutions like these is limited, whereas the experts in this study were familiar with such institutions—in part because of their specific background as professional educators, but also because of their broader knowledge base as educated adults. However, if a lack of prior knowledge brought to the evaluation task by young people leads to an overly critical view of sources, this seems preferable to their assuming reliability in the absence of prior knowledge. Figure 2. Mean site ratings by students. Represents mean rating of each site in the pre-unit and post-unit evaluation tasks. Evaluation Criteria Two research questions focused on the features students attended to while evaluating reliability: Research Question1b, Do students increase their attention to source features in justifying their ratings of those sources? and Research Question 2, After participating in the instructional unit, to what extent and in what ways do students evaluate the source features of web sites when the evaluation process is embedded in an online inquiry task? Addressing these questions entailed coding students’ justifications for their site ratings. The initial coding scheme 53 consisted of two general categories that previous studies suggested are often underused by students: Authorship and Information Sourcing (Coiro et al., 2015; Johnson & Putnam, 2012; Johnson & Putnam, 2015). Authorship encompassed author identification, expertise, and bias; and Information Sourcing encompassed references to the sources of the information presented by a site (references lists, links, and so on). In addition, I included categories for Site Qualities, comprising references to site type, publication information, affiliations, and site purpose; and Content Qualities, comprising references to content aspects such as balance, logic, evidencebased support, corroboration, and anecdotal support. I added codes to each of these categories as they emerged from the data. I also adapted two additional codes from a previous study (Johnson & Putnam, 2015) to ascribe when students appeared to be Off Task. These suggested a student was not clearly focused on site evaluation and was evidenced by two distinct tendencies. One applied when the student listed arguments for or against the controversial issue (Lists Argument For or Against Issue), for example, “If teachers don’t give homework, the students won’t learn discipline.” The other applied when students copied information from the site as if taking notes (Restates Info on Site), for example, “half of students now take cell phones to school.” Listing arguments suggests the student misunderstood the task of evaluating the site and instead evaluated the argument or issue. Restating information suggests a misunderstanding of site evaluation as a note-taking process, whereby students list facts they have learned from the site instead of evaluating the site’s reliability. In both cases the student’s response conveys a disconnect between the assigned task of site evaluation and the ‘criteria’ he or she reported using for evaluation. There also emerged, as in previous studies, a need for the code category, Vague/Inarticulate, assigned to student justifications that did not clearly articulate a criterion or were too vague to code reliably. 54 Figure 3. Hierarchy of code categories and subcategories for evaluation justifications. As coding proceeded, a need emerged to divide codes that would be considered key criteria for evaluation from those that would be considered extraneous. Consequently, I created a fifth category, Extraneous Criteria, to contain considerations that would less effectively serve the reader in determining reliability. The category comprised codes that referred to whether the site’s views aligned with the reader’s (Aligns to Prior Opinion); whether the reader viewed the information as relevant to his or her needs (Information Needs); whether the site was user friendly (Functionality/Ease of Use); whether the site was popular, received many “likes” or visits, or received favorable comments by previous users (Popularity/User Comments); and a code that included references to appearance, ads, sidebar content, grammar and proofreading issues (Surface Level Characteristics). Although some site aspects included in the Extraneous Criteria category, particularly surface features, may be useful for determining a site’s reliability in some circumstances, the goal of this instruction was to encourage students’ attention to aspects of the sites that served deeper critical evaluation, particularly authorship and source features, which were present in three of the four Key Criteria (Authorship, Site Qualities, 55 Content Qualities, and Information Sourcing). Figure 3 shows the hierarchy of code categories and subcategories, and Appendix K includes a comprehensive list of codes and their definitions. The pre-unit and post-unit evaluation tasks were designed to determine how students would evaluate web sites when directed and given ample time to attend to evaluation alone. In the embedded task, however, evaluation was contextualized in an online inquiry assignment for which students chose their own topics; students were required to evaluate web sites and justify their evaluations both during the research process and in the annotated bibliographies attached to their final argument essays. In both the pre-unit and post-unit evaluation tasks students were limited to the evaluation of four sites, whereas the embedded task was not constrained by number of sites. However, to make the evaluation task manageable in the context of independent online inquiry, students were required to provide three justifications in their evaluations of each site, considerably fewer than were provided for many of the pre-unit and post-unit evaluations. These factors made comparing code frequencies problematic, and I have chosen, therefore, to display code frequencies as percentages of total criteria cited in each task. Figure 4 shows the percentage of student references attributed to broad categories in the pre-unit, post-unit, and embedded tasks. Comparison showed a considerable post-unit increase in the percentage of Key evaluation criteria and a corresponding decrease in Extraneous, Inarticulate/Vague, and Off Task responses. These changes were significant, as determined with a paired samples t-test comparing references to Key Criteria in the pre-unit task (M=.60, SD=.281) and the post-unit task (M=.93, SD=.089); t=-8.281(49), p<.001, d=-1.5831. The embedded evaluation task showed a similar result; a paired samples t-test showed a significant difference in the percentage of references to Key Criteria in the pre-unit task (M=.60, SD=.281) and the embedded task (M=.91, SD=.095); t=-7.994(49), p<.001, d=-1.3471. 56 Figure 4. Pre-unit, post-unit, and embedded task categories. Represent the percentage of evaluation justifications attributed to each category. When examined by class, the increase in Key Criteria and relative decrease in Extraneous Criteria, Inarticulate/Vague justifications, and Off Task indicators held for both classes in the post-unit task and in the embedded task. Figure 5 shoes the code categories attributed to responses by students in the advanced class in each task, and Figure 6 shows these results for students in the regular class. The increase in references to Key Criteria was less pronounced among advanced students than regular students, but increases were notable for both groups. This increase held in the embedded task among both groups. The increases were balanced by decreases in references to Extraneous Criteria and Vague/Inarticulate justifications. Off Task behaviors that were present among both groups in the pre-unit task did not appear in the post- 57 unit or embedded tasks among either group, suggesting that students better understood the evaluation task after the unit, and this improvement was greater in the regular class than in the advanced class. Figure 5. Advanced class pre-unit, post-unit, and embedded task categories. Represent the percentage of evaluation justifications attributed to each category by advanced students. 58 Figure 6. Regular class pre-unit, post-unit, and embedded task categories. Represent the percentage of evaluation justifications attributed to each category by regular students. Key Criteria Categories Within the Key Criteria category marked shifts occurred from pre-unit to post-unit and embedded evaluation tasks. A Chi-square test of independence showed strong evidence of a different distribution of Key Criteria categories between the pre-unit and post-unit tasks (Chi square=195.444, df=3, p<.001, Cramer’s V=.3982), as well as between the pre-unit and embedded tasks (Chi square=272.713, df=3, p<.001. Cramer’s V=.3412). Figure 7 shows the percentages of Key Criteria subcategories cited by all students in the three tasks. The greatest Cramér’s V is commonly used to determine the strength of the association when there are greater than two categories within a variable (Cramér, 1946); 0 corresponds to no association and 1 to complete association. 2 59 increases were in the subcategories Authorship and Information Sourcing. Conversely, the number of references to Content Qualities decreased, whereas the number of references to Site Qualities increased slightly. However, because the total number of Key Criteria justifications increased in the post-unit evaluation task, this resulted in a small percentage decrease in references to Site Qualities from 17% to 15%. Figure 7. Pre-unit, post-unit, and embedded task Key criteria subcategories. Represent the percentage of Key criteria attributed to each subcategory of Key criteria for all students. 60 Figure 8. Advanced class pre-unit, post-unit, and embedded task Key criteria subcategories. Represent the percentage of Key criteria attributed to each subcategory of Key criteria for advanced students. Key Criteria Categories by Class A closer look at results within the Key Criteria category reveals that trends were consistent across both classes, with minor differences. Figures 8 and 9 show the Key Criteria breakdown for each class in the pre-unit, post-unit, and embedded tasks. In the post-unit task advanced students markedly increased their focus on Authorship, moderately increased their focus on Information Sourcing, and decreased their focus on Content Qualities and Site 61 Qualities. This pattern held for regular students as well, except that, among regular students, references to Site Qualities remained consistent. The instruction appears to have shifted the focus of both groups toward Authorship and Information Sourcing, and away from Content Qualities. In the embedded task the trend away from Content Qualities and toward Authorship and Information Sourcing continued. After the unit students attended more to who created the sites and where the information was acquired, and less to whether content included evidence, convincing arguments, or was balanced (though content remained an important consideration). Figure 9. Regular class pre-unit, post-unit, and embedded task Key criteria subcategories. Represent the percentage of Key criteria attributed to each subcategory of key criteria for regular students. 62 Key Criteria Subcategories and Extraneous Criteria As noted above, references to subcategories of Key Criteria shifted from the pre-unit to post-unit task. Figure 10 shows the comparison of mean references to Key categories per site by each student in the pre-unit and post-unit tasks. In the following section I examine the specific codes within the four Key Criteria subcategories, and in the category Extraneous Criteria. In each section I begin by comparing the mean number of references to codes per site by each student in the pre-unit and post-unit evaluation tasks. Because the embedded task differed from the pre-unit and post-unit tasks as previously discussed, comparison of embedded task means to pre-unit and post-unit means was problematic. Therefore, instead of including a preunit to embedded task comparison by code references, I have converted these to percentages to show the proportion of the subcategory attributed to each code. This allows for visualization of shifts in student focus within subcategories in the pre-unit, post-unit and embedded tasks. Figure 10. Mean references to Key criteria subcategories per site by each student. Represents the likelihood of a student citing criteria in the subcategory for a single site in the pre-unit and post-unit evaluation tasks 63 Authorship. Authorship, the first category within Key Criteria, comprised three codes: Author Identification, Author Expertise, and Author Conflict of Interest/Bias. Figure 11 shows the mean references to Authorship subcategories for a single source by a single student in the pre-unit and post-unit tasks. Rates of each code increased, with the greatest increase in increase in references to Author Expertise. Mean References to Authorship Criteria 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00 author id_ info conflict of interest_ bias PRE-unit Task expertise_ authority POST-unit Task Figure 11. Mean references to Authorship criteria subcategories per site by each student. Represents the likelihood of a student citing specific criteria within Authorship for a single site. Figure 12 shows the percentage of all Authorship criteria cited for each code within Authorship in the pre-unit, post-unit, and embedded tasks. The code Author Expertise increased as a percentage of Authorship criteria from the pre-unit task to both the post-unit and embedded tasks, whereas the percentage of references to Author Identification decreased. This suggests that, after the instructional unit, students shifted their attention from searching solely for a site author’s identification to considering the author’s level of expertise or authority. The percentage of references to the code Conflict of Interest/Bias increased slightly after the instructional unit, but decreased as a percentage of total criteria cited in the embedded task. 64 Figure 12. Percentage of references attributed to each code within the subcategory Authorship in the pre-unit, post-unit, and embedded evaluation tasks. Figure 13. Mean references to Site Qualities criteria subcategories per site by each student. Represents the likelihood of a student citing specific criteria within Site Qualities for a single site. 65 Site qualities. The second category within Key Criteria, Site Qualities, comprised seven codes: Contact/Publication Information Available; Date/Age of Article/Site; Familiarity with Site; Site Type; Published/Vetted; Purpose/Intent/Target Audience; and Affiliations/ Sponsorship. Figure 13 shows the mean references per site by each student to specific codes within this category. Within the subcategory Site Qualities the greatest increase was in references to Date/Age of Article or Site, with a fairly large proportional increase in references to Purpose/Intent/Target Audience. Figure 14. Percentage of references attributed to each code within the subcategory Site Qualities in the pre-unit, post-unit, and embedded evaluation tasks. Figure 14 shows the percentage of references to each code within the Site Quality category. The codes Date/Age of Article/Site, Purpose/Intent/Target Audience, and Affiliations/Sponsorship increased in percentage from pre-unit to post-unit tasks and from preunit to embedded tasks. Likewise, the percentage of references to the code Contact/Publication 66 Info Available decreased markedly from both pre-unit to post-unit and pre-unit to embedded tasks. Information sourcing. The subcategory of Key Criteria, Information Sourcing, comprised of the codes Identifies/Links Source Information; Age of Source Information; and Nature/Quality of Source/Source Information. All criteria in this category related to where the information provided on the site was found or acquired, not to qualities of the site itself; the word source therefore refers to the source(s) of the site’s information, not to the evaluated site. Figure 15 shows the mean references to Information Sourcing per site by each student in the pre-unit and post-unit evaluation tasks. References to all codes increased after the unit, with the greatest increase in references to the Nature/Quality of Source Information. The frequency of the code Identifies/Links Source Info increased moderately, and the code Age of Source Information increased modestly. Figure 15. Mean references to Information Sourcing criteria subcategories per site by each student. Represents the likelihood of a student citing specific criteria within Information Sourcing for a single site. 67 Figure 16 shows the percentage of references to each code within the category Information Sourcing in each task. The percentage of references to the Nature/Quality of Source Information increased from the pre-unit to the post-unit task and from the pre-unit to the embedded task, whereas percentages of Identifies/Links Source Information and Age of Source Information showed relative decreases. This suggests that students shifted their attention from examination of whether information on the sites was linked and current to deeper examination of whether the site’s information sources were high quality, diverse, and relevant. Figure 16. Percentage of references attributed to each code within the subcategory Information Sourcing in the pre-unit, post-unit, and embedded evaluation tasks. Content qualities. The final category within Key Criteria, Content Qualities, was comprised of the codes Balance/Objectivity; Logic/Strength of Argument; Information Corroborated; Anecdotal Support; and Evidence/Research-Based Support. Figure 17 compares the mean references to content qualities per site by each student for each task. References to these criteria decreased from pre-unit to post-unit tasks for all codes except Balance/Objectivity, 68 which increased negligibly. The greatest decreases were in the frequency of the codes Logic/Strength of Argument and Research/Evidence Based. Figure 17. Mean references to Content Qualities criteria subcategories per site by each student. Represents the likelihood of a student citing specific criteria within Content Qualiities for a single site in the pre-unit and post-unit evaluation tasks. Figure 18 compares the percentage of references to each code within Content Qualities in each task. Within this category, the percentage of references to the codes Research/Evidence Based and Anecdotal Support remained consistent, whereas the percentage of references to the code Logic/Strength of Argument decreased. Balance/Objectivity grew as a percentage of this category from the pre-unit to the post-unit and from the pre-unit to the embedded tasks. Therefore, although the students’ overall attention to Content Qualities decreased in relation to other subcategories, their attention to whether a site provided evidence or presented a balanced view remained an important aspect of their attention to Content Qualities. Their shift away from considering the logic and strength of a site’s argument may be explained by a limited capacity for attention to specific criteria—as their attention to other issues (i.e., Authorship and 69 Information Sourcing) increased, their cognitive capacity for evaluating the logic and quality of a site’s argument decreased. Figure 18. Percentage of references attributed to each code within the subcategory Content Qualities in the pre-unit, post-unit, and embedded evaluation tasks. Extraneous criteria. The category Extraneous Criteria was created for justification criteria that could not be coded as one of the four Key Criteria. These criteria included the codes Aligns with Prior Opinion, Information Needs, Functionality/Ease of Use, Popularity/User Comments, and Surface Characteristics. The mean references to Extraneous Criteria cited in pre-unit and post-unit tasks are shown in Figure 19. Within the category Extraneous Criteria, the greatest difference in per-student means was in Information Needs, assigned to criteria that related to meeting the information needs of the reader. This change suggests that, in the pre-unit task, students were more likely to evaluate reliability based on whether a site was relevant their information needs—not, in fact, a measure of reliability but of relevance. There were also decreases from pre- to post-unit tasks in the references to Surface Level Characteristics and 70 Functionality/Ease of Use. The former was assigned to criteria referring to advertisements, sidebar content, appearance, attractiveness, pictures, grammar and proofreading issues, whereas the latter was assigned to criteria describing features that made a site easier to use or more accessible to the reader. Decreases in the use of the codes Information Needs and Functionality/Ease of Use suggest that students focused in the pre-unit task on whether a site was going to help them achieve their task rather than on site reliability, whereas they focused more w holly on site evaluation in the post-unit task. Figure 19. Mean references to Extraneous criteria subcategories per site by each student. Represents the likelihood of a student citing specific criteria within Extraneous criteria for a single site in the pre-unit and post-unit evaluation tasks. Attention to Information Needs, however, appears to have re-emerged in the embedded task (see Figure 20). The percentage of Extraneous Criteria attributed to Information Needs increased considerably in the embedded task. This may be because the embedded task required evaluation in the context of an authentic information need; therefore, the students’ attention to their information needs were more relevant in that context. However, numbers of total 71 references to Extraneous Criteria decreased considerably from the pre-unit task to the post-unit task and remained fairly low in the embedded task—recall that Extraneous Criteria comprised 23% of total criteria in the pre-unit task, but 5% of total criteria in the post-unit task and 7% in the embedded task. In short, the students still decreased their overall attention to both Extraneous Criteria and most codes within it. Figure 20. Percentage of references attributed to each code within the category Extraneous criteria in the pre-unit, post-unit, and embedded evaluation tasks. Approach to Task Although I did not initially intend to compare the behaviors of students as they approached the task of web evaluation in the pre-unit and post-unit evaluation tasks, trends emerged from the data that revealed interesting patterns in their approach to the task. Students’ justifications in the post-unit evaluation task appeared more complete and often implied investigative behaviors. Such justifications began with phrases such as “When I clicked on the 72 author I found . . .”, “When I Googled the author I learned . . .” or “The source links go to . . .” suggesting the students were actively pursuing links beyond the page to investigate certain aspects of the site. The frequency of such phrasing in students’ justifications necessitated an additional code category, Investigation. In this section I provide explanations of codes that emerged within that category, which were applied to the pre-unit and post-unit evaluation tasks in a separate coding pass. Due to the nature of the embedded task, in which students evaluated sites of their own choosing in a broad and unstructured time frame, it was not possible to attribute Investigation codes to that task. The category Investigation included two codes ascribed when the student clearly exerted additional effort in their investigation of the sites: Clicks Off Page for Info; and Searches Off Site. I assigned the first code only when it was evident that the student had clicked off the site, either because he or she explicitly reported it (“When you click on About the Author nothing comes up”) or when the student’s comment included information that was not available without clicking off the initial page (“The sources they list are all from the same web site”), determined by review of the preselected sites. The second code required a deliberate description of an independent off-site search to inquire about an author, source, or piece of information. In most cases this was a Google search (“When I Googled the author I learned he was not an expert on this topic”). Although there were few of these, they revealed significant effort to investigate a site and were therefore worth noting. A second theme related to the students’ Approach to Task was present almost entirely in the pre-unit evaluation task and included cases in which the student appeared to be Off Task. Though I presented these in the initial code category comparisons, I revisit them here because they reflect an Approach to Task that contrasts sharply with the Investigative behaviors present 73 almost entirely in the post-unit evaluation task. Recall that the code Off Task suggested a student was not cognitively engaged in site evaluation and instead focused on either listing arguments for or against the issue, or restating information learned on the site. Figure 21. Approach to Task by category. Mean instances per student of Investigative and Off-Task behaviors in the pre-unit and post-unit evaluation tasks. An examination of the two approaches to the task (Investigation and Off Task) in the preunit and post-unit tasks showed marked differences in the students’ approach to evaluation after the instructional unit (Figure 21). The per-student mean of Off Task behaviors decreased from 2.84 in the pre-unit task to 0.02 in the post-unit task, whereas the per student mean of Investigative behaviors increased from 0.02 in the pre-unit to 2.2 in the post-unit task. Code breakdowns for Investigation and Off Task behaviors are shown in Figure 22. Within Investigation, Clicks Off Page for Info increased pre-unit to post-unit from .02 to 2.14 mean references per student, and Searches Off Site increased slightly from 0 to .06 mean references per student. In the category Off Task behaviors the code Lists Argument For or 74 Against Issue decreased pre-unit to post-unit from 2.06 to 0 mean references per student, whereas the code Restates Info on Page decreased from .78 to .02 mean references per student. Figure 22. Approach to Task by code. Mean frequencies per student of specific Investigative and Off-task Behaviors in the in the pre-unit and post-unit evaluation tasks. Approach to Task differed to some extent between students in the advanced class and students in the regular class, although trends remained consistent. Figures 22 and 23 show the Approach to Task by class. In both groups the mean rates of investigative behaviors increased from the pre-unit task to the post-unit task, and off-task behaviors nearly disappeared. However, students in the regular class displayed off-task behaviors more frequently in the pre-unit task than did the advanced students, and showed a greater reduction of off-task behaviors as well. Interestingly, advanced students showed fewer investigative behaviors in the post-unit task than did students in the regular class. One explanation for this difference may be that advanced readers better comprehend site content, read more quickly, and are therefore less motivated by a sense of confusion to investigate. Nevertheless, although the impact of the instructional unit was greater on regular students, it appears to have been effective for both classes, with regular and 75 advanced students showing a reduction of off-task behaviors and an increase of investigative behaviors. Figure 23. Advanced class Approach to Task by code. Mean frequencies per advanced level student of specific Investigative and Off-task Behaviors in the in the pre-unit and postunit evaluation tasks. Figure 24. Regular class Approach to Task by code. Mean frequencies per regular level student of specific Investigative and Off-task Behaviors in the in the pre-unit and post-unit evaluation tasks. 76 Source Model Construction In response to Research Questions 1c, Do students build stronger source models after the instructional unit? students responded to a measure testing their recollection of specific source features after completing each evaluation task. It presented two open-ended questions about each site with small color thumbnails to prompt students’ recollections. Students completed the source model measure the day following each evaluation task at the beginning of the class period. Their responses comprised the data for comparison of the source models created by students before and after the instructional unit. Appendix H includes a screen capture image showing one source model measure used. Instances of stance recollection in the pre-unit and post-unit source model measure are summarized in Figure 25 and show little change. A paired samples t-test comparing the frequencies per site of pre-unit to post-unit recollections of site stance showed no significant difference in the students’ ability to recall site stance in the pre-unit (M=2.87, SD=1.154) and post-unit measures (M=2.91, SD=.855); -.202(46), p = .841. Figure 25. Stance recollection frequencies per student in source model measures. Mean instances per student for which site stance recollections were correct, incorrect, and not recollected in the pre-unit and post-unit source model measures. 77 However, as shown in Figure 26, correct recollection of authorship features did increase from pre-unit to post-unit Source Model Measures, whereas false recollections remained about the same. In the case of responses including more than one element of authorship recalled, each fact was considered a separate instance. A paired t-test comparing the number of students’ correct recollections of authorship features showed a significant difference between the number of correct authorship recollections in the pre-unit Source Model Measure (M=2.49, SD=1.864) and in the post-unit measure (M=4.96, SD=2.726); -7.177(46), p <.001, d=-1.0581. These results suggest that students did build stronger source models after the unit, as indicated by a significant increase in correct recollections of authorship features. Figure 26. Author recollection frequencies per student in source model measures. Mean instances per student for which authorship recollections were correct, incorrect, and not recollected in the pre-unit and post-unit source model measures Examination of the students’ recollections of authorship features by class indicate that both regular and advanced classes increased their recollections. Figure 27 shows the mean number of authorship features recalled by each student in the advanced and regular level classes. The average per-student increase in the number of separate recollections of site features related 78 to authorship was 2.45 in the advanced class and 2.48 in the regular class, indicating that students in the two classes increased their average recollections by approximately the same amounts. The increase in the number of author features recalled by students may indicate that they built stronger source models during the post-unit evaluation task than they built in the pre-unit evaluation task. Because, as previously discussed, authorship is an important element of source model building, it is possible that the students’ increased attention to authorship after the instructional unit led them to focus on authorship more closely in the post-unit source model measure, which in turn led to their building stronger source models for the sites, as evidenced by their ability to better recall those features. Figure 27. Mean correct author recollection frequencies per student by class level. Mean instances per student for which site stance recollections by advanced and regular students were correct in the pre-unit and post-unit source model measures. 79 CHAPTER 6: EVALUATION THEMES IN DIALOGIC DISCUSSIONS Research Question 3 asked What do students’ individual annotations and contributions to dialogic discussions reveal about the critical evaluation processes they use to judge the usefulness and reliability of web sites? However, due to student changes observed in the preunit, post-unit, and embedded tasks, the focus of this question shifted to address the processes by which those changes may have occurred over the course of the unit. I have chosen, specifically, to examine how the students’ attention to authorship, information sourcing, and investigation developed. Student changes in these areas were the greatest among students in the regular class, and were most evident in discussion that emerged in the Socratic seminars. Consequently, I analyzed the three Socratic seminars conducted with regular students, focusing on the themes of Authorship, Information Sourcing, and Investigation, to illuminate the processes by which these themes emerged and developed. I begin by presenting a series of examples showing the collective examination by students of the themes Authorship and Information Sourcing. My goal is to explore how the students’ conceptual understanding of these themes developed from more reductive and simplistic to more expansive, complex, and flexible over the course of the unit. Because the nature of the seminar format allows considerable freedom for students to explore their responses to the question, the themes were interwoven throughout the seminars and, due to the interdependence of the concepts, flowed from theme to theme freely. I have chosen to discuss each theme separately to more readily draw parallels between the content of these discussions and the quantitative results gathered in the pre-unit, post-unit, and embedded evaluation tasks, in which the themes Authorship and Information Sourcing were also prominent. 80 The third theme discussed in this section is not an evaluation criteria but a behavioral tendency. Just as investigation emerged as a common theme in the post-unit evaluation task, it also emerged as a primary theme in the seminars. An examination of investigative behaviors in the seminars was therefore warranted to determine how those behaviors were practiced and developed within the discussions. I discuss these three themes using the third person to improve readability, but contributions attributed to the teacher in all cases are my own. Links to the web sites discussed in each of the three seminars are listed in Appendix C. The Socratic Seminars The students in the regular eighth grade language arts class had become accustomed to the Socratic seminar format. This unit took place in January and February; students were introduced to the seminar format in October, and had had several opportunities to practice and improve. Specifically, the students understood that their role was to participate but not dominate the discussion, that they were to explore the essential question collaboratively by speaking with their classmates rather than to the teacher, and that their contributions would not be arbitrated by the teacher selecting speakers through raised hands. The students recognized that the conversation was theirs and that the goal was to come to collective understanding. Each of the three Socratic seminars followed a familiar structure. Students self-identified as either “talkers” (those comfortable and eager to speak up in a group discussion) or “nontalkers” (quieter personalities whose tendency was to listen and refrain from contributing to a group discussion). The seminar was divided into two parts, with one group (either talkers or non-talkers) sitting in a center circle of tables to discuss one site for half of the seminar, while the other group sat in an outer circle to observe, listen, and chat digitally with each other on the topic being discussed in the center circle. Half way through the seminar the groups exchanged 81 places, and a conversation about a second site took place. This division ensured that students identified as talkers would practice listening and taking turns, and the students identified as nontalkers would have ample opportunity to practice speaking up and to encourage other quiet types to do the same. Each seminar began with an essential question posed by the teacher. That question was open-ended to encourage a variety of possible responses, and in all three sessions was a variant of the same idea: “When considering reliability, what are the most important things for a good critical reader notice about THIS site?” The teacher’s verbal emphasis on the word ‘this’ in the question was intentional and designed to emphasize the variability of reliability criteria for different sites on the web. Students were encouraged to refer frequently to the text to support their assertions. In this case, because the texts consisted of two web sites, the students had their computers open to the sites as they participated in the seminar, and often directed the group to specific points of interest on the sites. Complexities of Authorship Attention to authorship is an evaluation theme that emerged from the three Socratic seminars as an important criterion to consider when evaluating internet web sites. Students centered on authorship from the beginning of the first seminar, during which their focus sharpened as they became more deeply analytical about authorship issues. In doing so, they often revealed movement from reductive to flexible epistemic views of knowledge as their verbal exchanges about specific ideas developed. Thus, evidence of movement from reductive prefigurative schemas to cognitively flexible prefigurative schemas emerged. 82 Authorship in Seminar 1 The first seminar contained the most extensive and revealing exchange related directly to authorship. It focused on the question of whether high school football programs should be discontinued due to the rate of injuries sustained by teens who play. The students examined two web sites: one defending football programs and written by a medical doctor, and one criticizing football programs and written by a journalist. In response to the opening question When considering reliability, what are the most important things for a good critical reader to notice about THIS site?, Rick responded with the suggestion to “look up the author and who wrote it” to “see if there’s anything else he’s written in the past.” There wasn’t an immediate response to Rick’s observation, but a few minutes later Jason asserted, “It talks about how he’s a dad and his dad was a coach and stuff.” The teacher took this opportunity to prompt further exploration: T: “Do you think that’s important?” Jason: Yeah because he’s gonna know a lot about football and the effects on my head.” Dustin: “I agree ‘cause he’s a physician but he’s a medical researcher at Stanford, so you can believe it because of his background.” In the exchange above, Jason, an enthusiastic football player, identified his consideration of authorship more specifically with a concern about author expertise, and Dustin then articulated the idea in greater detail. Here the students progressed from simply identifying the author, as Rick initially suggested, to considering what the author knows and how he knows it as an indication of his expertise. However, they applied their knowledge of a physician’s expertise in a somewhat reductive way by assuming that a physician working at Stanford warrants the reader’s trust. These assessments were based on the information that was readily available on the page, but did not require students to investigate beyond it. 83 The conversation drifted from authorship for several minutes, until Dean made a pivotal contribution: “Another thing about the author is, you notice that at the bottom it says Ed Riley works as a professor of anesthesia, and to me, what does that have to do with any injuries other than when he debilitates patients—puts them under?” Here Dean questioned the students’ previous assumption that, as a physician, the writer was an authority on football injuries. Recognizing that Dean had taken the students in an important direction, the teacher encouraged the line of inquiry: T: “Did anybody actually go look him up, to find out what he did?” Charles: “I clicked on his name… but it didn’t really give much information about him.” T: “Did anyone go a step beyond that to learn more about him?” [chorus of no] T: “That was an interesting point you made, Dean, about his medical experience. I had the same thought. And I did go a step further and Googled him.” At this point the teacher shared the search terms she used and students followed her lead to access the Stanford University site where the author’s biography was located. There they learned that the author is a doctor of obstetrics and gynecology and specializes in anesthesia for women giving birth by Cesarean section. This new knowledge encouraged the students to revisit their previous conclusions about the author’s expertise; their conceptual change was evident in this exchange: Rick: “So he has exactly nothing to do with brains.” Dean: “It’s kind of ridiculous, though, because he’s acting like he knows all this stuff when all he does is … [trails off]” 84 Rick, whose interest was piqued, began reading the Stanford bio aloud to the group. After listening for a minute, Dean summarized with an assessment of the author’s expertise: “So he has everything to do with childbirth, but not with football in any way.” However, just as a subset of the group expressed agreement with Dean’s summary, Jason offered an alternative perspective, reminding the class that the author did have “two sons, (one) a senior quarterback.” Dean and Charles both challenged Jason: Dean: “That doesn’t mean he focuses at all on brain injuries…” Charles: “He’s not gonna have the same information…” At this point Landon jumped in to defend Jason’s perspective: “He mighta got it (the information) from his brother…” and the teacher probed for further clarification: “What is his brother? What about his brother?” Both Damian and Landon responded by identifying the author’s brother as the Oregon State University football coach and a former quarterback. When the teacher asked for more explanation, Landon continued, “He’s in a college and he knows about the injuries, what can happen to his players…” Rick defended his original view: “But his brother could have told him one side, because his brother probably likes football because he’s a coach, so he’s telling the positives more than the negatives, like people do.” At this point several students appeared to reassess the author’s expertise, and a series of comments supported and extended that assessment to a more general consideration of how expertise might be misrepresented on the internet, as well as the implications of doing so: Dustin: “Just because you’re a coach doesn’t mean you know all about the injuries in football.” 85 Cole: “That could be one of the reasons why there isn’t any places where you can look for sources, because maybe he got all this stuff from his brother and didn’t have the facts that could support it.” Zach: “He also said in his paper, ‘I’m a physician and a medical researcher at Stanford’ and we find out that he’s a gynecologist who does C sections, so that’s kinda untrue. He said medical researcher at Stanford, but didn’t really state what he actually did.” Rick: “So he’s leaving stuff out.” Julia: “To make it sound better.” Charles: “If he leaves that part out, he might leave more of the story out.” It is worthwhile noting that the previous discussion on authorship began with a rather surface-level glance at a web page to see if the author was identified and what he had written. Key contributions by individuals and timely intervention by the teacher moved the discussion beyond the surface level, from identifying the author as a physician to recognizing him as an anesthesiologist, to learning his specialty in anesthesiology research. Consequently, some members of the group changed their initial assessment of the author’s reliability, judging their initial reductive assessment of physician-as-authority to be faulty, and concluding that, in this specific case, the author’s expertise was less relevant than initially assumed. The group experienced critical evaluation of authorship from multiple perspectives as individuals shared their observations and rationale. The students constructed a collective understanding of an important lesson in this exchange: web authors are not always what they appear to be. Deeper inquiry is requisite to confirm first impressions, and it demands active investigation by the reader. In this case, 86 investigation through a Google search of the author illuminated his expertise and authority on the subject. Authorship in Seminar 2 In the second seminar, the topic of authorship arose briefly, but students investigated it more proactively. The controversial issue was whether online schools should replace traditional public schools. The first site was a promotional article published by a national online school corporation and the second was a large city newspaper article reporting on a national study of online schools. The students intentionally searched for information about the authors from the beginning of this discussion, a response they seem to have learned from their experience in the first seminar. Damian responded immediately to the opening question that “there is no ‘about’ page about the author; that makes me question whether it is reliable or not.” When Julia noted that the hyperlink on the author’s name took her to “articles she’s made,” Damian reasserted, “Yeah, but there’s nothing about her.” Presumably applying a lesson from the previous seminar, Rick promptly Googled the author and shared what he learned: “I agree with Damian, because I Googled her name, and nothing really comes up with anything about her writing, it’s more her Twitter and stuff…” At this point the students determined that there wasn’t enough information available to ascertain the author’s expertise, and the discussion shifted to information sourcing as the dominant theme of the seminar. That discussion is analyzed at length in the following section on information sourcing. In fact, the second seminar contained several efforts by individual students to investigate site elements by conducting searches outside the site being evaluated, the application of a lesson related to authorship that was reapplied to questions of information sourcing. 87 Authorship in Seminar 3 In the third seminar the students’ active investigation of authorship reemerged. The controversial issue of this session was whether video games have a positive or negative affect on teenagers’ brains. The first site, from Intel Developer Zone and written by an Intel software developer, presented “Six Fascinating Facts About How Video Games Are Good for You.” It was created by an author whose affiliations presented a possible bias and was intentionally included to extend the students’ thinking about author affiliation and bias. The second site was a weblog article from Discover Magazine titled “Video Games May Have Negative Effects on the Brain.” About five minutes into the third seminar, Davis affirmed that the links on the first site led to some confusing pages: “Yeah, and when I clicked on the author’s name, Eliana P., it didn’t tell me anything about her background at all, it just told where she worked or what else, things she published or articles.” Hope added, “Yeah, just stuff she had, like a brown belt, or whatever…” Here Hope was referring to the description of the author as an Intel software designer with a brown belt in software design, presumably indicating her level of expertise as a programmer. The teacher clarified while reading from the linked page, “Software developer … that’s what it says: ‘Intel brown belt software developer…’” Janet tried to make sense of this aloud: “It, like ranks it by workplace?” Here the teacher explained the brown belt level in martial arts and the implied use of the term for Software developer skill levels. When it became clear that the author of the site was a software developer at Intel corporation who programmed video games, Hope concluded, “This article is self-promoting.” In the above exchange, which occurred near the beginning in the seminar, the students made an early effort to investigate the author’s expertise, but what they found confused them. 88 Within the context of the students’ collective attempt to understand, the teacher offered a more didactic explanation of the author’s professional certification by clarifying the use of the term brown belt in this context. This “just in time” commentary allowed Hope to ascertain that the software developer had a vested interest in video game promotion, and with that realization called into question the author’s reliability. This exchange reinforced the need for close investigation to delve more deeply, not only to determine an author’s credentials, but also to determine whether the author’s conflict of interest makes bias more likely. Hope’s determination that the article was “self-promoting” implied her lack of trust in the author, and by extension, the article. Later in the seminar, when asked about the overall reliability of the site, Davis articulated his ambivalence and the group revisited the notion of author reliability and bias: “I really have mixed feelings about it . . . because some of the information seems pretty good and reliable, but the author of the thing, you can’t find information about that, so that brings reliability down on it . . . I just don’t know about it.” When pressed to articulate more clearly what was “good about” the site, Davis continued, “How they put the references down at the bottom of the page, and how they work at a . . . probably doesn’t matter too much . . . but works at a pretty well known company, and that’s good, the company is good.” At this point the teacher, recognizing that Davis had not yet come to the same conclusion that Hope had come to earlier in the discussion, revisited the author’s affiliations: “Do you think the fact that it comes from Intel makes it more or less reliable, the rest of you?” Here the students disagreed: Che: “It actually makes it more reliable, because they make technology things.” Hope: “Not much though . . . I don’t really, I don’t know . . . “ 89 T: “How does making technology things make one an expert on the effects of video games?” Andrea: “It seems like they would know a little bit more about it than just like, someone who just cares about this topic.” T: “Because they care a lot about it?” A: “Well, yeah, it’s like their job.” The exchange above highlighted a tension between author expertise and author bias: Although experience can lead to expertise, it may also lead to a deep personal investment in an issue on the part of the author, which may in turn lead to bias. At this point in the conversation Che made an important pivot that moved the group beyond the reductive assumption that expertise making video games leads to knowledge about video games’ effects on the brain. Che: “They want people to like the stuff they make, so it’d probably be a little one sided. If just tells you the positive only.” Andrea: “They want to make it sound like these things are good for you.” T: “If they said on here that video games were bad for you, what would be the effect on them?” Hope: “They’d probably get less money.” Che: “Their business would be hurt.” The excerpt above reveals how the students’ dialog led to conceptual change and construction of meaning. They began with a reductive prefigurative schema that assumed a logical link between expertise and reliability—that greater expertise leads to greater reliability. Hope had rejected this assumption early on, but the rest of the students had not. Later, Che’s active questioning of the assumption prompted the group to revise their evaluation of the site. As they grappled with 90 the question of whether the author’s affiliation to Intel was problematic, they wove their way back to the conclusion that Hope had previously made: expertise does not always lead to reliability, and bias is possible when an author is personally invested. It is also worth noting that this evaluation concern was revisited several times over the course of the seminars in the context of different authors and conflicts of interest. Some students, like Hope and Rick, were quick to recognize potential conflicts of interest, whereas others required longer processing time and more interaction to recognize them. However, all the students experienced repeated iterations of this specific tension in variable contexts. Although the students attended to authorship in their initial assessments of the sites by looking for author identification and credentials, the three Socratic seminars exposed them to situations in which further exploration was necessary to fully evaluate an author’s expertise, and illustrated an important precept of internet evaluation: Appearances can deceive. Online authorship is problematic because the driving motivations of authors can misalign with a reader’s pursuit of knowledge on a topic; information provided by an author may be selected to fulfill a promotional or profit motive. Therefore, although a reader may never know the absolute truth regarding an author’s authority, investigation beyond an author’s initial presentation of himself is essential for moving closer to that truth. The series of open-ended Socratic seminars facilitated shared inquiry that illuminated the complexities of authorship, and then provided opportunities to practice grappling with those complexities in varied situations. Complexities of Information Sourcing Attention to information sourcing is a second theme that emerged as an important criterion of internet site evaluation. Over the course of the three Socratic seminars, the students examined to varying degrees the information sources used by each site, with depth of inquiry 91 ranging from simple determination of whether sources were cited to a deeper investigation into what those sources were, where they came from, why they might have been chosen by the author, and finally, how effectively they supported the claims made by the site. Expressly, students moved from a more reductive prefigurative stance to a more expansive and flexible one vis-à-vis information sourcing. Information Sourcing in Seminar 1 In the first Socratic seminar attention to information sourcing emerged within the first 10 minutes of the discussion. Damian suggested, “if they give a list of web sites, click on them to see if they’re not just junk, make sure they actually have the information on those web sites . . .” The teacher inquired, “Did they give a list of web sites?” and two students determined there were few sources other than the writer’s son: Dustin: “There’s no sources, really,” Charles: “Just the son.” After a few minutes, Zach offered, “I think what’s important to notice is where the author got his information, cause as you look on the site, I don’t find any sources, numbers, because he mentioned something about Oregon State football and all these other coaches and stuff.” Here Zach implied that the information required citations that were missing, and both Dustin and Rick affirmed his stance: Dustin: “It’s good probably to have some opinion but not all opinion.” Rick: “Yeah he should back up his facts but like, he did back it up with 2% of people who play football actually get brain injuries and stuff, but there’s nothing to prove that.” Here Dustin made an interesting distinction between opinion and fact, implying that sourcing is one method of providing factual evidence. Rick affirmed the link between information sourcing 92 and facts, observing that a fact might be presented as evidence, but if its source is not cited, there is no way of knowing whether the fact can be trusted. In this exchange the students moved from the simpler proposition that a reference list or links to sources is desirable to a more complex consideration of why, justifying their interest in information sourcing by articulating that sourcing is the method through which a writer offers verification for claims. At this point Landon presented an alternative proposition, challenging Rick’s assumption that an unattributed fact can’t be trusted: “What if he knows the facts?” Here Landon suggested that an author may present facts he knows to be true simply because he is an expert, and such an author can be trusted. His assertion indicated that he took the authority of the author at face value, whereas his classmates did not. In response, Dustin and Charles asserted that, although an author may indeed know the facts, citing sources provides assurance that is necessary: Dustin: “I think he should still give sources though, because we need to know he didn’t just make it up.” Charles: “Yeah, all this information could be false. [several others respond ‘yeah’] There’s not sources that allow us to go back and find where the information came from.” This exchange showed students deconstructing the importance of information sourcing and juxtaposed two prefigurative world views about adherence to authority: The reductive stance evinced by Jason’s willingness to trust an authority without question was set against a more flexible prefigurative schema espoused by Dustin and Charles that assumed a more critical, skeptical stance. Even when the author is someone we are inclined to trust and who may have considerable expertise on a topic, references or information source links provide reassurance that the information provided is accurate and reliable. Except for Jason, the students seemed to conclude that site evaluation should involve a search for such references or links. After several 93 conversational turns that moved the discussion away from this issue, Quinn later revisited it. When the teacher asked students to compare the reliability of the two sites, he restated the point: “I think this one is more reliable because . . . it has links to stories that happened, to back up the evidence and stuff.” Katelyn expanded Quinn’s contribution: “I feel like it’s important to look at the links the site is linked to, because if you click on it, and the tab opens up to a total different topic, then that would be totally unreliable, too.” Her comment affirmed the importance of source citation, but also suggested that the existence of links is not enough; it is essential that a link be relevant, and the only way to know that is to click on it. Information Sourcing in Seminar 2 In the second seminar about online schools the discussion of information sourcing presented evidence of greater nuance in the students’ evaluative thinking. They began grappling with issues of information source selection related to bias and conflict of interest. In discussing the first site produced by an online school corporation, Sebastian noticed the evidence consisted entirely of anecdotal quotes from one family attending an online school: “I noticed that most of the people that talked are all, like, from the same family, and no one else, like to compare it, only that one family . . . I thought it was just weird.” No one picked up on Sebastian’s comment at the time, and the conversation moved away from the topic of sourcing until Rick spoke again. Rick: “I think what makes it a little unreliable, is if you click on the hyperlinks she put up, if you click on it, all it does is goes to her . . . thing . . . other pages she has written in the past . . . I don’t think that really helps her situation.” Caleb: “That’s kind of like self-promotion.” Rick: “Yeah. On citing it.” 94 T: “Did anyone else click on the links in the first couple of paragraphs?” [many respond “yeah”] Caleb: [reading] “’The Highs and Lows’ written by Sara Bernard.” Dustin: “Yeah, they’re all written by Sara Bernard.” In this exchange the students moved beyond the site page to examine where source links led, and determined that all sources were written by the author of the original article. Caleb articulated one problem with this: it suggested the author used the source links to promote her own work. The students drew no further conclusions at this point. However, the discussion revealed Caleb’s awareness of how selection of sourcing might be driven by a writer’s ulterior motive or a site’s promotional goals. Interestingly, Caleb revealed his ability to consider the same site from multiple vantage points just a few minutes later, when he suggested that the sources might be reliable after all. Caleb: Something that makes this site pretty reliable is that, what Dustin said earlier, the science teacher, a lot of their quotes actually come from these teachers and people that have degrees in learning.” Caleb was referring to the quotes that were included in the text of the article, some of which were from teachers and some were from students. In fact, these were the student comments that Sebastian had said were “weird” a few minutes earlier. None of them provided citations or links to information about the people interviewed. Zach recognized this problem, and immediately challenged Caleb’s assertion that the expertise of the quoted sources was enough to make the site reliable, reiterating the idea that, even if an author seems to have expertise, citation is critical: Zach: “I disagree, because it’s just as easy to talk to the teacher; they can lie about talking to these people. They could just put it down there, because they don’t have any sources, 95 like contact numbers or officials they talked to, they just said that they talked to them. So they could easily be lying.” This exchange exemplified how the issues of authorship, information, site affiliations, and site purpose can be intertwined. Caleb viewed the anecdotal evidence in the article positively because it came from people he viewed as having expertise, but Zach suggested that additional sourcing is still essential to provide reassurance that information can be trusted. Rick reiterated that conflicts of interest can influence both a message and the sources a writer uses to defend his message, and can be related to affiliations, author motives, or site purpose. He revisited that important lesson, observing, “There are a lot of quotes from people, like teachers and parents and students from Florida Virtual School.” He then offered a method for addressing such challenges when he continued, “I searched it and it’s an online school for students in grades k-12 for all over the world.” Here Rick made an important intellectual move to identify who the sources were and their affiliations. His inquiry led him to investigate the Florida Virtual School and revealed a potential conflict of interest that any critical reader should have observed. Here the teacher pursued Rick’s line of inquiry to reiterate the point. T: “So what does that mean? So what? So we have all these quotes from people from Florida Virtual School, Florida Virtual School instructors, Florida Virtual School students, we have a CEO of Apex Learning who runs an online school, we’ve got . . . who else? More students . . . what do you make of that? Does that matter?” Caleb: “It’s not that diverse. It’s one standard . . . it’s like we need more evidence.” Rick: “Yeah, more different sources.” Zach: We only get surveys and opinions from one school. They should expand their . . . surveys on schools and what they think about learning and all their students.” 96 The above exchange revealed that, although the students struggled to some extent to articulate clearly the information sourcing problem presented by this site, they recognized the importance of source diversity. A site advocating for online schools that quotes students and teachers from only one online school without providing broader data-based research, and linking only to the same author’s previous work warrants reader skepticism. Later in the discussion, when the teacher asked students to rate the site’s reliability, Andrea gave it a low rating because “the information isn’t diverse,” and the teacher summarized a point relevant to site evaluation—that sources referenced on a site may be selectively chosen to support a bias: T: Yeah, so imagine this. Imagine I wanted to write an article about how great Mrs. Johnson’s class was, and how Mrs. Johnson should be the teacher of the year . . . and I got a bunch of students to write quotes about me . . . so how many people are quoted in this article? Four or five? Do you think I could find four or five students to say nice things about me? Does that mean that all (my) students . . . would say nice things about me? It could be that I just cherry pick a handful of students who happen to love me, or who I gave As to or I brought ‘em some brownies . . . could I write an article that sounded exactly like this? [lots of agreement] And how reliable would that article be?” The class vehemently agreed that the article would be biased. In the second seminar, then, the students explored information sourcing beyond a simple list of citations to an examination of source diversity, exploring the selectivity biases that can affect the quality and diversity of information sources and the nature of the evidence provided by them. This was another shift in the student’s prefigurative schemas from a reductive inclination (e.g., simply looking for a reference list or sources) to a multifaceted, integrated analysis of those sources (e.g., where the links lead, what the sources contain, where that information was acquired and by whom). 97 Information Sourcing in Seminar 3 The third seminar further challenged the students’ thinking about information sourcing and pressed their ability to grapple with disorderly and ambiguous content. It focused on the question of whether video games were good for teenagers’ brains, with the first site by a video game developer and the second from a popular science magazine. Both were published shortly after the release of a study by a British neuroscience researcher, Daphne Bevelier, and both referenced her work in their source links. However, the first source from the Intel Developer Zone also referenced another study, whereas the Discover Magazine blog reported primarily on the Bevelier study. Although the sources on both sites included peer-reviewed, scientific research on video games and brain, the studies were not well aligned with the video game promoted on the first site by the Intel Developer Zone. In fact, a careful look at the referenced studies made it clear that the Intel programmer had misapplied the research for her own purpose. This, however, was a challenging conclusion to ascertain by eighth grade students, and the difficulty they faced in grappling with it was evident. When the teacher asked students to consider what was most important to notice about the Intel site when considering its reliability, Landon noted that “she put the source number after her facts, so you could go and find the facts she wrote.” Landon referred to the footnote style citation used by the author and the teacher explained how footnotes were traditionally used to cite sources. Katelyn had also noticed the footnotes: Katelyn: “I notice that she put reference on here. I clicked on the first one, it had a video instead of an article, so that you could listen to that.” T: “Did you?” K: “No, I didn’t but . . . “ 98 T: “Did anyone?” Kamil was the only student to answer, “the first minutes, but then I quit it . . .” The teacher pursued the issue of investigating sources, asking the group, “Are you familiar with that site? Have you seen it before? . . . I was talking about the site that Katelyn just sent us to, which was the first link on the source list. . . it takes you to TED, the TED site. Is anyone familiar with TED? Have you ever heard of TED talks?” Some of the students had heard of them, and the teacher explained TED talks for the others. Then the teacher addressed Katelyn: T: “So did you see that as a good source or not a good source, Katelyn?” Katelyn: “Well, I mean, I didn’t watch the video, but I believe that it could be a good source, because not many people know it, but some people know it.” Katelyn’s noncommittal answer suggested she was conscious that her lack of familiarity with TED talks was problematic, and paired with the fact that she did not view the video, was unable to determine its reliability as an information source. The teacher invited others to weigh in. April: “Well, I looked up the . . . I don’t remember . . . the name . . .” T: “Daphne?” (the first name of the author) A: “Yeah, I looked her up, she was, she came from some pretty popular college, and she seemed pretty reliable from some of the things I looked at . . .” T: “Daphne Bevelier . . . and then if you click on the full bio on the TED page . . .” At this point the students began to follow the link as modeled by the teacher, and the teacher read from Bevelier’s biography about her research in learning, video games, neuroscience, and brain plasticity. The teacher summarized Bevelier’s job and location of work, and the students clicked on the link to her lab to investigate. At this point the students seemed satisfied with the 99 background of the researcher, and they did not comment further. The teacher returned to the original article to pursue inquiry into the source links. T: “Did anyone look at any of the other references at the bottom of the page, of the original article? Kamil: “The third evidence, I click on, and it was kind of different from others . . .” Hope: “Yeah, she used that one to promote Happy Jump (a video game).” T: “Why? What do you mean?” H: “Well, because it, she talks about how games are like sweets, and the article talked about that.” Hope noticed that the author was promoting a video game called Happy Jump, which involved the player jumping to “catch” pieces of candy appearing on the screen, but seemed unsure about how she deduced it. She responded only, “Well . . .” and trailed off. At this point, the teacher asked about additional source links: “What about the second one? Did anyone look at the second one?” Here a pause in the discussion suggested that the students had not, and the teacher intervened to emphasize the essentiality of pursuing source links: T: “Here’s the thing. When I look at the second article, it comes from Science Daily, ‘your resource for the latest research news,’ okay, that seems all right . . . and then this is the title of the article: ‘Video Games Can Be Highly Effective Training Tools,’ study shows . . . (reading from text) ‘Employees learn more, forget less, and master more skills.’ It was done in 2010 at the University of Colorado . . . Is it a study about teens playing video games about candy?” 100 When the students agreed aloud that it was not, the teacher continued, “It says video games can be effective training tools . . . so what was the subject of the study? What did they study?” Quinn immediately responded, “Employees,” and the teacher pursued the line of logic further: T: “Does this study suggest ‘Video games are good for you and therefore you should play Happy Jump?” Hope: “No, because this is about employees . . . “ In the exchange above the teacher took a central role in the discussion to help the students approach the important but difficult notion of generalizability in research: In this case, a study about using video games to train employees for specific tasks cannot be assumed to show anything at all about teens playing video games for fun. In the exchanges that follow, the students continued to grapple with this abstract concept, and the teacher scaffolded their learning with questioning, summarizing, and explanation. Andrea then raised a relevant point made in the article, that employees at Coldstone Creamery improved their ability to scoop ice cream by playing a video game in which they scooped ice cream. “So, does that prove that video games are good for you in general?” the teacher asked. Katelyn responded, “I think no, because you don’t really need to learn how to scoop ice cream . . .” Katelyn’s comment suggested that she understood there must be a connection between the specific skill taught by the game and the skill the user wishes to gain, and the teacher extended Katelyn’s contribution to better articulate that idea: “The study is about how people on a job play video games targeting special skills to get better at those skills . . .” Che offered an example from his prior knowledge: “Like if there’s a game about flight, like flying airplanes, and you’re a pilot . . .” T: “Which there are, actually, for training . . . is that good?” 101 Che: “I think so.” Janet: “Not if that was his only training . . . I feel like this second source, I don’t see anything about how video games are bad, because it does focus on one thing, how video games contribute.” T: “Does it, and maybe it does . . . does it prove what the Intel developer is trying to convince us of?” Hope: “It gives us one kind of . . . I guess . . . but doesn’t really help . . . “ The above exchange demonstrated several challenges the students faced in determining the reliability of the information sources linked to this web site. On the one hand, the TED talk appeared to be a reliable source by an expert in the field of neuroscience whose research seriously examined the effects of video games on the brain. Likewise, the second source linked to a reliable-looking research study on the benefits of using video games to train employees. The next logical leap, however, was difficult for them. The students struggled to tease out an important issue with the site’s information sources: although reliable in themselves, the sources were not completely relevant to the site’s argument. The research study and the TED talk highlighted the benefits of using video games for very specific outcomes. The application of that research to support the assertion that video games are, in general, good for the teenage brain, was problematic. When asked whether they would, in the end, use this site as a source for research on the effects of video games, the students struggled to come to consensus. Che’s answer revealed his indecision: Che: “I would use some of it, not all of it . . .” T: “What would you use?” C: “When it says, 5 to 10 hours a week . . . better vision . . . and nongamers . . . “ 102 Here Che picked out specific facts that he thought would be useful in an argument, and the teacher, scaffolding the students’ practice in critically evaluating information sources, refocused their attention on the original sources: “But that’s cited as number one, that comes from number one at the bottom.” At this point Hope quickly picked up on the logic suggested by the teacher: “I’d probably use the resources more than the actual site.” Katelyn confirmed Hope’s suggestion by explaining that she would use the sources of the site, but not the site itself, going directly to the TED talk and the research study without citing the Intel study at all. Before moving on to the text site, the teacher summarized the point: “Why would you want to use a questionably biased site for good information, when you could go to the original sources of the good information and get it there instead?” In the third seminar, then, the students addressed issues of information sourcing that confused and challenged them, but, with scaffolding provided by the teacher, came to greater clarity regarding those complexities. When, in the second half of the seminar, discussion shifted to basic differences between a site that is convincing and one that is reliable, Davis offered the intellectually complex observation that a site “could be convincing and lie at the same time” and therefore, information sourcing is essential to know the difference. Daniel revisited the importance of examining information sources to ensure they are themselves reliable and objective: Daniel: “Well, one thing that wasn’t very reliable was that they did it (the study) on 26 people, and that’s a really, really small amount . . . plus they had more people that didn’t play video games than did play video games.” Dustin: “Yeah, they could just go get people who are anti.” Julia: “Yeah, they didn’t have, like, a wide variety of people, they only had adults, too.” 103 In the final minutes of the third seminar, then, the students applied previously constructed schemas to new observations. They looked not only for the existence of information sources, but examined them to determine whether they were reliable (in this case, whether the conclusions drawn from the study really supported the site’s assertions, and whether the study itself was conducted without bias.) Their contributions suggest a shift toward thinking about information sourcing in complex and critical ways. Investigation In Seminar 1, as recounted above, the students did not initially investigate the identity of the Stanford physician who argued against the banning of high school football programs. The discussion of this author’s expertise emerged early in the first seminar, but the teacher found it necessary to model the investigation process before the students followed suit. When she asked the students whether they had gone “a step beyond that (the biography on the page) to learn more about him,” the response was a chorus of “no.” The teacher intervened with a demonstration of what investigation might look like and what it might reveal, and the students followed her example to investigate the site themselves. In doing so, they learned that the physician’s specialty was not related to sports injuries, brain injuries, or athletics of any kind, and that knowledge changed their evaluation of the site considerably. Once the students experienced the cognitive change possible from information gleaned through independent investigation, they began to approach the sites with greater skepticism and felt compelled to investigate beyond the surface of most sites. For example, in the remainder of the first seminar students shared information gained by clicking beyond the first page of a site to examine authorship, affiliations, or sources six more times. Some of this investigation occurred within the seminar itself, 104 exemplified most obviously when Caleb said, “I just Googled Steve Almond . . .” and then shared what he learned. In the second seminar the trend toward investigation was even stronger. In total, the students shared information they had gained by clicking off the initial page of a site (either by checking the author’s biography, the site’s affiliations, or the information sources) 14 times. Six instances of investigation beyond the initial site page occurred within the first five minutes of the seminar. Investigation was prompted when Damian asserted that there was no author “About” page; his classmates clicked on her name to see the other articles she had written and Rick “Googled her name . . .” to investigate the sources she used in the article. Likewise, Dean “looked up Cheryl Bedou, the CEO of Apex Learning” and Daniel “looked up Maureen Cotrell . . . a science teacher at an academy . . .”. Rick investigated the Florida Virtual School discussed in the article, noting, “I searched it and it’s an online school . . .” Likewise, Dean “did look up Apex Learning and they, do, actually run a school.” The information gathered from the students’ collective investigative efforts was synthesized with scaffolding when the teacher prompted students to construct new understanding from it, asking them to step back and consider, “So. What does that mean? So what?” The question was enough to assist their synthesis, and they finally concluded the site lacked source diversity. Later in the same seminar the students applied this investigative mindset again when examining the sources of the second site. When a student mentioned a source from CREDO, the students clicked on it. It opened to a PDF of a research study conducted by the organization. The teacher briefly clarified the title of the CREDO study, its date and publishing information, and noted its affiliation with Stanford University. Several minutes later Cole, looking up from his computer, offered the following: 105 “So I researched that CREDO web site, and it says it stands for the Center for Research on Education Outcomes, and it says it wants to improve education in America, which could make it more reliable . . . it gives many ways to learn about the site, which is a good thing to include . . . it says she interviewed a person who works at CREDO, that person could know a lot of facts and evidence that could contribute to this article.” Prompted by the teacher’s modeling, Cole exhibited an in-the-moment curiosity about the CREDO site which resulted in a richer understanding of the information source and led the group to revise their final evaluation of the site. Dean articulated this conceptual change at the end of the seminar when the group was asked, “What’s the biggest takeaway of the day for you?” He responded, “Personally, I think all the evidence I had was washed down the drain, because I learned CREDO was something different than I thought . . .” In fact, the students’ responses to this question suggested a new appreciation for investigation. Sebastian noted, after clicking on sources to examine them, that “They’ve conducted all this research on only two schools . . .” and it completely changed his initial opinion of the site. Zach’s takeaway was to “look and see who’s made the site, nonprofit group or university like Stanford,” and Landon’s was to “look more at the authors to learn more about them and see if they actually know what they’re doing and stuff.” Charles’s takeaway suggested a shift to a more inquisitive stance: “Before today I just kind of looked to see what the sponsors were and the UFL and stuff outside, and I think I need to look more inside . . .” Adele articulated her shift with an example: “When we did the citations and stuff I was in a hurry and didn’t take the time to look at the hyperlinks but then when you asked if we looked at them, I was like, no, I didn’t, and when I looked they were all by the same person.” Here the teacher took a moment to reiterate a primary lesson of the day: “You have GOT to click on those links.” 106 Investigative behaviors continued in the third seminar, where students’ comments indicated they had clicked off the initial site pages several times to examine aspects of authorship, affiliations, and sourcing. Janet had “pressed the partner sites” to learn more about the first article’s affiliations to the software industry; Davis “clicked on the author’s name, Iliana P.,” and Katelyn “clicked on the first (reference); it had a video instead of an article.” Later, Kamil added, “The third evidence, I click on, and it was kind of different from the others . . .” At one point in the conversation, when no student admitted to clicking on the second source link, the teacher modeled investigation of it. That investigation led, in turn, to the recognition that the site’s information sources were not relevant to the article’s claims. In the second half of that seminar Caleb investigated a study referenced, adding to our understanding of its reliability: “Daphne Bevelier, I looked her up, and she is a brain doctor, she’s a neurosurgeon, and what she’s saying could possibly be true . . .” Adele investigated a source as well: “. . . when you click on the site, it was published in 2014 . . .” and Julia added, “It’s a Pubmed web site, and I looked it up, and it shows that it’s a U.S. National Library of Medicine, National Institutes of Health.” Cole and Rick discussed what they learned by investigating the author. Cole: “. . . when I clicked on the author’s name, I found out that he writes many articles that have to do with the things that affect your brain, which adds to credibility, but maybe not, because how do we know that he IS a specialist on the topic instead of just writing what he thinks about it?” Rick: “I think it brings credibility because he has ten plus articles on things about the brain, different subjects and stuff related to the brain, and that means he has a lot of knowledge, because he looks things up and writes this, and it’s not all bias and has facts in there, and the facts add up . . . a very smart man.” 107 It’s interesting to note that the above exchange shows Cole and Rick negotiating and collectively synthesizing meaning from the information gleaned in their shared investigation. In accord with the demands of problem solving in ill-structured settings, they considered multiple possibilities and integrated different aspects of the writer’s background (quantity and diversity of experience, factual evidence, absence of conflicts of interest) which led Rick to ascertain that the author was reliable. However, despite the students’ inclination to investigate many aspects of the sites, it was still necessary for the teacher to scaffold their investigation with prompting. In an important section of the article the authors referenced a study about how video gamers and nongamers solved problems differently, which was a pivotal source of evidence for the article’s claim. Within the text of the article the word report was hyperlinked, but none of the students had noticed it. When the teacher directed students to click on that link, the students were taken to a recently published, peer-reviewed study by the British Royal Society. When the teacher asked students whether it appeared reliable, they agreed that it did. The following exchange ensued: T: (The research study) is cited, it certainly gives information and statistics, and certainly describes where the information came from, which is not what you said when we started.” Dustin: “We didn’t know then . . . “ T: “Why didn’t you know?” D: “Because we didn’t click on it.” T: “Because you didn’t click on it. The whole first part of the article, all of that information comes from one source that is cited, but you didn’t notice it. That’s really important.” 108 D: “Is that okay, for a web site to, instead of give links, to give like a red word?” T: “That’s exactly what a lot of sites do these days.” D: “Yeah, I remember you saying that. “ Here Dustin revealed that he had forgotten an explanation provided by the teacher in an earlier seminar about how source referencing on the web is often done through hyperlinks instead of reference lists. The entire class was reminded, and the importance of thorough investigation was reinforced. At the end of the seminar, the teacher asked students how their thinking about web evaluation had changed over the course of the unit. Their responses reflected conceptual change about the necessity for independent inquiry and an investigative mindset: Daniel: “I realized that you should click on everything and question everything.” Caleb: “I agree . . . you kind gotta look into what you’re gonna read and see how it’s displayed . . .” Charles: “Yeah, I think some of the most important stuff to determine a site’s reliability is actually within the text, so you kind have to look thoroughly . . . “ Jason: “Another important thing is who the article’s written by, and what they do, and how they know, and how they got their information.” Dean: “I agree with all four of them, because you have to look very deeply into the text to find out information.” Julia: “Yeah, I was literally going to say, look deeper into the text.” Cole echoed the sentiment when he admitted that, before the unit “I would normally just not even click on the links, just skim through quickly.” Most striking about the students’ selfevaluative statements was their emphasis on active, self-reliant, effortful reading on the web. 109 Whether their investigation focused on authorship, bias, affiliations, or information sourcing, the students clearly realized that examining a web site beyond surface characteristics, beyond a cursory skim, and beyond a single page is essential for an informed assessment of its reliability. 110 CHAPTER 7: DISCUSSION This study fills a gap in the literature on internet web site evaluation by examining the results of an instructional unit designed to increase eighth grade students’ attention to Key Criteria for evaluation. The unit included an introductory one-day introduction to web evaluation followed by three Socratic seminars, each focusing on two preselected web sites about a controversial issue of interest to middle school students. Prior to each seminar students read and annotated the two web sites to prepare for the discussion, and the seminars took place over the course of three weeks, with one discussion per week. A pre-unit web site evaluation task and a pre-unit source model measure were given over the two days prior to the unit; a post-unit web evaluation task and a post-unit source model measure were given over two days at the beginning of the week following the evaluation unit. In the 4th and 5th weeks students applied the web evaluation process in an independent project in which they chose a controversial issue to research. That project required students to justify their web site choices during the information gathering process and again in an annotated bibliography attached to their final essays. The pre-unit and post-unit evaluation tasks showed that students improved their ability to rate internet sites and became more critical of internet sites after the unit. Likewise, their attention to Key Criteria for evaluation, particularly to authorship and information sourcing, increased. Additionally, their approach to the post-unit evaluation task reflected an increase in investigative behaviors after the unit. Source model measures showed that students’ recollections of authorship features were stronger after the unit than before it, though their recollections of site stance did not change. In the embedded task, the students’ focus on Key 111 Criteria generally remained, and students retained their attention to authorship and information sourcing features. Analysis of the three Socratic seminars conducted with the regular-level language arts class provided insight into the students’ evaluation processes and revealed interesting shifts in their thinking, especially regarding authorship, information sourcing, and investigation. Contributions revealed multiple instances in which students offered initial justifications for an evaluation of a site, then extended, deepened, or changed their justifications over the course of the discussions. Students were frequently exposed to differing perspectives on a single web site, and as those perspectives were articulated, challenged, compared, and revisited, the students revised their evaluations, often becoming more critical. The Socratic seminar format afforded students the ability to grapple with the complexities of web site evaluation, providing a space for both practice and growth. Developing Students’ Evaluation Skills In the section that follows I discuss how the present study reinforces theories about teaching complex critical thinking in ill-structured environments. I also consider implications for practice and for future research in the teaching of critical evaluation of web sources. Attention to Authorship and Sourcing Results of the pre-unit and post-unit evaluation tasks suggest that instruction increasing students’ attention to features such as authorship and information sourcing can positively affect their ability to effectively evaluate the reliability of web sites. When authorship and sourcing issues arose organically in the seminar discussions, the teacher reinforced attention to them by posing follow-up questions, modeling investigation of authorship and sourcing, and summarizing important lessons about these issues. The students’ increased attention to these two Key Criteria 112 for evaluation in the post-unit and independent embedded tasks suggests that it is indeed possible to move students away from dependence on surface characteristics toward more meaningful criteria for evaluating web sites. The present study joins the ranks of several others that suggest meaningful evaluation can effectively be taught (Braasch et al., 2013; Mason et al., 2014; Walraven et al., 2009). In addition, as theorized by Rouet (2006), the source models students build may be strengthened when their attention to authorship is increased, at least concerning aspects of the source model that depend on attention to authorship features. The failure of students to better recollect site stance even when their recollection of authorship increased is worth noting; it suggests that the content and source nodes of Rouet’s (2006) situations model may function independently. In this study the author nodes of students’ source models seem to have been strengthened, whereas content nodes were not—that is, students did not better recall the stance of the site, but did better recall aspects of authorship. This is not surprising, given that the instruction steered student thinking toward authorship and information sourcing features and away from content features. It should also be noted that students’ attention to content features in the pre-unit evaluation task focused largely on whether a source provided factual evidence for its stance. Students’ justifications referring to factual evidence were often simplistic statements about whether facts or evidence existed on the web site. The students’ increased attention to the quality and relevance of information sources in the post-unit evaluation task suggests a shift from a simplistic hunt for facts and statistics toward a more critical consideration of evidence. Likewise, the students’ increased attention to authorship in both recalling and evaluating the sites reinforces Rouet’s theoretical link between source evaluation and source model building. 113 Behaviors Related to Evaluation An examination of the students’ approach to the evaluation task before and after the instructional unit, as well as over the course of the three seminars, illuminates the improvement shown in their ability to thoughtfully and critically evaluate web sites. In the pre-unit task students were more likely to show off-task behaviors in which they confused listing arguments for or against a controversial issue or simple note taking with justifications for site evaluation. They were also more likely to write inarticulate or vague justifications. Alternately, students in the post-unit evaluation task were more likely to investigate aspects of a site by clicking on links, examining those links, or searching off site for author and source information, and also articulated their evaluation justifications more clearly. Interestingly, students in the regular-level class showed greater movement towards investigation in the post-unit task than did students in the advanced level class. One explanation for this may be that advanced students, who are competent readers, more often—and more quickly—make evaluative inferences from textual and source information available on the initial web page, whereas regular level students, who may be less competent readers, make fewer inferences and therefore require greater investigative effort to draw conclusions about reliability. This would align with research finding struggling readers to have greater difficulty evaluating online sources than competent readers (Bråten et al., 2011; Eastin et al., 2006; Fogg et al., 2003). Average readers may come to the task with gaps in prior knowledge that advanced readers do not have, requiring greater investigation to fill those gaps, a possibility that would align with Bråten et al. (2011), whose limited-resources hypothesis posited that less able readers require greater cognitive resources to determine the relevance of potential web sources for an information task. Nevertheless, the instruction seems to have prompted both groups to evaluate with greater effort and engagement; both classes increased their investigation 114 of the sites beyond a surface level scan, and the decrease in both groups’ references to surface features strengthens that claim. The results align with prior research suggesting that deeper engagement with a text leads to stronger source model construction (Rouet 2006), better comprehension, and more critical evaluation (Braasch, Rouet, Vibert, & Britt, 2012; Zhang & Quintana, 2012). It also supports the notion that, not only is it possible to improve evaluation by engaging students more deeply with the text, but it is also possible to encourage the shift toward effortful engagement by providing experiences for students in which such engagement is fruitful and empowering. Effective Instruction in Critical Evaluation Because web site evaluation is a complex task that requires cognitively flexible knowledge (DeSchryver & Spiro, 2009), this unit was designed to adhere to principles of instruction that develop such knowledge (Feltovich, Spiro, & Coulson, 1993). One framework for instruction included in the unit was the Socratic seminar, chosen because it aligned closely with the design principles outlined in Chapter 3. The positive results of the instruction suggest that those design principles were appropriate for instruction to improve critical evaluation skills, and that the Socratic seminar was an effective instructional framework for doing so. Here I revisit some of those design principles to show both how and why the Socratic seminars served the unit’s objectives. Conceptual interdependence. The seminar discussions about web site evaluation showed the students engaging with the complexities of authorship and information sourcing criteria in ways that closely aligned with the principles Feltovich, Spiro and Coulson (1993) posited for effective instruction toward advanced knowledge in any complex domain. One aspect of such instruction is to present concepts interdependently rather than in isolation to 115 emphasize the connection and combination of ideas, conceptual dependency, and conceptual variation in different contexts. Because the Socratic seminar format was essentially a freeform discussion, and because the essential question was broad and inclusive of any evaluation criteria the students brought to light, neither did it organize the web evaluation process into a series of clear steps, nor did it force students to consider evaluation criteria in isolation, the way an evaluation checklist might. Students were free to share any criteria they viewed as useful at any time. The conversation consequently moved from one topic to another and back again, revealing the connectedness of different evaluation criteria. This occurred, for example, when the students were asked in the first seminar which was more important to consider when evaluating the reliability of that site, authorship or information sourcing. The class was unable to come to consensus on the answer, determining ultimately that both mattered equally and that they were closely related. As discussed in the previous chapter, the identity of an author often led to questions about expertise, and those in turn led to questions about the source of the information an author provided. The seminar format provided space for the students to examine how the interdependence and connectedness of web site elements (authorship, information sourcing, purpose, and so on) can influence reliability in multiple, shifting ways. Avoiding reductive bias. Because the Socratic seminar format provided space for attending to the interconnectedness and contextuality of the subject matter, it was also appropriate for avoiding reductive bias. As previously discussed, reductive bias occurs when complex concepts are reduced to simplistic organizational structures or analogies that fail to adequately convey the complexity of the content (Spiro, Feltovich, & Coulson, 1996). For example, a misconception among some of my students was that any site URL ending in the .org domain is reliable because it was created by a nonprofit organization, whereas any site URL 116 ending in the .com domain is unreliable because it was created by a for-profit organization. The second seminar juxtaposed two such sites. One student presented the reductive viewpoint, asserting that “...this site is a .org, so that’s a nonprofit organization, so they’re doing it for the cause, not just making money. I think that adds credibility.” This assessment was based on the reductive dichotomy he had learned previously, but over the course of the discussion he concluded that the rule did not apply; in this case, the .com was determined to be more reliable than the .org. When asked to share the “biggest takeaway of the day” the student admitted, “Before today, I just kind of looked to see what the sponsors were and the URL and stuff outside, and I think I need to look more inside.” Ultimately, then, the seminar presented an opportunity for students to organically voice prior misconceptions and for the group to address them, thereby revising their approach to site evaluation from a simple, reductive judgment to a more thoughtful consideration of the case in context. Multiple variations and contexts. The six different sites students examined in the seminars were chosen for their unique variations on the evaluation process to provide opportunities for students to “reuse and try out knowledge” in “multiple situations in which knowledge is used” (Kolodner, 1997, p. 63). Examination of each case began with the same question: “What’s important to notice about this site when considering its reliability?” Strong emphasis on the word this suggested that each site was a unique case, and that contextuality would be a primary concern each time evaluation was practiced. Although many of the same issues arose with each of the six cases examined, each required students to re-examine the same issues of authorship, expertise, bias, sourcing, and affiliations in different contexts to apply concepts recently learned to new situations. This emphasis on contextuality is an important 117 principle of instruction for cognitive flexibility advocated by Feltovich et al. (1993), and the seminar format successfully integrated this principle. Multiple perspectives. The affordance of examining sites from multiple perspectives was also a principle of design built into the essential structure of the seminar, and it encouraged students to examine and revisit cases and concepts from multiple viewpoints. This frequently led students to revise their evaluations of a site. Such instances usually began with a preliminary observation by a student, followed by interactions extending that observation, (usually) another student’s alternative proposition, and further interactions leading to conceptual change. Additionally, the natural flow of discussion and the requirement that students build on what other students were saying encouraged an organic revisiting of concepts or cases. This became apparent in the number of comments beginning with phrases such as, “Going back to what A said . . .”; “I agree with what B said earlier . . .”; “I think C may be right but I also think. . .” Conceptual change was often revealed in these recursive exchanges, such as, “I was thinking that . . . but now I think . . .” or “At first I thought that . . . but now I think . . .” The advanced students were particularly articulate in expressing both the importance and complications of examining multiple perspectives. When asked, at the end of the final seminar, how “talking in this way about web evaluation is beneficial or not beneficial?” Sylvia responded, “I think it makes us think deeper about stuff that we notice, because some things I noticed . . . I noticed the same things other people did, but they noticed it in a different way, like, they viewed it in a different perspective than I did, and I felt like that helped me a lot.” Katherine agreed that “. . . we get opened up to a lot more ideas and you notice a lot of things you wouldn’t have noticed if you were just by yourself.” This exposure to various perspectives sometimes led the students to revise their opinion of a site. Peter articulated this when he added, 118 “Yeah, same way with Sylvia, like, every time we come into this, I have a perspective whether it’s reliable or not, and almost every time my opinion has been changed, at least a little bit, because of hearing what other people have thought.” Wes affirmed the assertion: “Yeah, I agree . . . I look into the stuff I missed or something, and I think a little bit more, and it’s like, yeah, that’s probably relative.” This occurred in both the regular and advanced classes. Tolerance for complexity and ambiguity. The diversity of perspectives emerging from the seminar did not necessarily make the evaluation process easier; however the format does seem to have encouraged a shift in students’ prefigurative schemas toward an expectation of complexity and ambiguity (Feltovich, Spiro, & Coulson, 1996). Students experienced the complex and sometimes confusing reality of web evaluation, which demands a reader delve beyond surface judgments to determine whether those judgments hold true under closer scrutiny. Michael asserted that the seminar “kind of hinders what you’re thinking because everyone does have a different opinion, and they notice things, and what you might have thought was someone’s experience, someone else thinks is bias . . .” Sahar agreed that “sometimes after hearing everyone’s opinion, I get more confused.” Her classmates nodded in agreement, and the teacher admitted that the discussions did “muddy the water sometimes.” Rachel noted that the process caused her to question her own judgment when examining web sites, and Lucy added, “I think it really brought to light how unreliable the internet can be . . . so we’re more cautious about it.” There was disagreement among students regarding whether they felt more confident evaluating sites after the unit than they did before—a testament that the seminar did not oversimplify evaluation, reduce it to simplistic processes, or provide for students a false sense of confidence. 119 At the same time, however, the seminar seems to have afforded students the ability to share a variety of solutions and strategies to cope with the complexity they encountered. This may have scaffolded the learning process for students needing more support. Abigail remarked that the seminars “help because you can see the point of view of different people, and you can see what they noticed and what you missed. Because some people will get things in different ways.” Later, Evan explained, . . . you can kind of find out different methods when you’re talking with people about it, because you hear what they found, and often they’ll tell how they found it, and so you learn these different methods, like clicking on the links . . . or looking into what the author has done, so it really opens you up to all these new ideas. Sandra was one of the students who did, in the end, “feel more confident, because after practicing annotating more and after each seminar I learned more and knew more things to look for while annotating the next time . . .” These comments suggest that the seminar format emphasized the complexity of web evaluation while simultaneously providing a forum in which students could collectively confront that complexity and negotiate methods for managing it. The seminar provided opportunities to experience a diversity of approaches to evaluation and to learn from fellow classmates’ misconceptions and understandings. Rosemary articulated this well: . . . people think in different ways, and you don’t necessarily know how other people are going to take something, and I think that’s why seminars are helpful for us, because not all of us think the same way, and we all notice different things about the web site, and when we put it all together, it starts to make sense. Rosemary’s comments highlight a collective construction of understanding present in the seminar discussions, and suggest these were deeper than the individual understandings students 120 might have built alone. If so, it reinforces past research showing the benefits of dialogic discussion for deeper understanding (Applebee et al., 2003; Barnes, 1976; Nystrand, 2006; Nystrand & Gamoran, 1991a, 1991b), and may explain why growth between the pre-unit and post-unit tasks in the regular level class was greater than the growth shown in the advanced class. The advanced class began with stronger evaluation skills at hand, and therefore had less to learn from each other. Conversely, the regular level students began the unit with weaker evaluation skills in general, and therefore may have benefitted more from the diversity of skills and perspectives brought forward in the discussions. Student agency and independence. Walton (2017) contends that an essential element of critical literacy in the information age is motivated curiosity: If we can address the very real problem of motivated reasoning by fostering the ability to act from a position of curiosity, where people can recognise that information is multifaceted and that science generates multiple and often contradictory viewpoints, then and only then might we shift their epistemic beliefs to enable them to reach this cognitive questioning state – which sits outside the bounds of their deeply engrained worldviews. (pp. 151-152). The seminar format placed the students in control of the discussion and thereby centered the responsibility for learning on them. It granted students greater authority and agency in evaluating each web site, and, in keeping with prior research that found dialogic discussion to increase student independence and responsibility (Adler et al., 2004; Alvermann et al., 1996; Nussbaum & Sinatra, 2003), encouraged independent investigation. In the advanced class’s final reflections on the seminar Tim asserted, “I think these are also important because it allows us to share our own opinion,” suggesting that he valued the opportunity to assert his personal views. 121 This sense of agency seems to have empowered students to investigate the sites of their own will, demonstrating the “Active learning, self-reliance, and intrinsic motivation,” suggested as an important element of the prefigurative schema for flexible learning outlined by Spiro et al., 1996 (p. S55). Active learning exists when a learner “Actively constructs knowledge in learning; does not depend on authorities (and may even mistrust them); is concerned more with meeting personal learning criteria than with evaluation by others” (p. S56). In the case of site evaluation, students must rely, at least in part, on their own investigation to determine an author’s expertise and trustworthiness or a source’s quality and relevance. The seminars afforded students opportunities to experience the fruitful results of independent inquiry, thereby nudging them toward this more curious, skeptical, and investigative stance over the course of the unit. This mindset toward investigation was articulated clearly by the advanced-level students when reflecting on how the seminars either served or did not served their learning about web evaluation. It should be noted that this question did not arise in the regular class because the regular students required more time to investigate the sites and draw conclusions from those investigations, and consequently ran out of time for the debriefing question. However, the coherent responses of advanced students to the question provide insight into the effectiveness of the seminar format in general. Evan admitted, “. . . before I would just look at a web site and assume it was good and use it, when I was researching things, but this has really just taught me to do these things to make sure it is good.” Annemarie agreed: “I will never be able to just look at a web site and use it again, because I’m going to have to, like, click everywhere and search everywhere . . .” Marissa went a step further in clarifying her shift in mindset over the course of the unit: 122 You kind of learn to think in a different way from annotating these articles, to think more into what the author does for a living, what the sponsors support, what they’re talking about, what their evidence is saying, what their citations are saying, rather than just saying, okay, it’s a big company or it’s something well known so it’s trustworthy. Marissa’s contribution suggests that she realized not only the importance of independent agency, but also gained the confidence to assert her personal authority to determine what is reliable. Rather than simply trusting a source because it’s made by a large and well-known entity, she realized that there’s much more to consider, and her comment suggests that she feels personally capable of such consideration. This is especially important because, as Walton argues, and information literate reader must consider “the actual production of knowledge, where it comes from, what interests are involved in producing this knowledge, how authority is defined and the notion that all knowledge is provisional; it is under a continual process of change and development.” These students’ reflections on their growth over the unit show a powerful shift in their prefigurative schemas regarding where knowledge comes from, how it is gained, and their own roles in constructing it. In sum, the open interaction and freedom to explore and investigate afforded by the seminar format appears to have highlighted the complexity of the evaluation process, exposed the processes of critical inquiry in the internet context, and accommodated multiple perspectives on a single site. This corresponds with a recent study by Litman and Greenleaf (2017), who examined the teaching of an argumentation in secondary language arts, history, and social studies classes. They theorized that instructional design allowing space for students to explore multiple approaches and perspectives may “explain a wide range of findings related to student engagement and emergent learning behaviors, including the high levels of cognitive engagement 123 that we often observed . . . when argumentation tasks gave students significant interpretive authority” (p. 16). They also found that “scaffolding designed to support student sensemaking within a broad inquiry space promoted engagement and learning” (p. 16). The seminars provided a “broad inquiry space” for organically risen intellectual inquiry within the context of unique and varied cases, drawing attention to their specific qualities and to the application of existing knowledge in each case. The transparency of the inquiry process within the seminar format made its application more accessible and apparent to the students who, in the end, assimilated and applied their learning in the post-unit evaluation task and in the independent inquiry project. Limitations Despite the significant difference between student performance on the pre-unit and post-unit tasks in this study, conclusions regarding the direct impact of the instructional unit should be moderated by certain limitations inherent in the study. Education research in natural settings is complicated by multiple factors, both known and unknown, that may affect results. These include sample composition and size, student motivation, teacher qualities, and unforeseen factors influencing the activities of any school day. Because the student sample is not necessarily representative of the population of eighth grade American students at large, and because the sample size is small, generalization is problematic. In this study an additional limitation exists in that the researcher was also the teacher, and, although every attempt has been made to separate activities inherent in the two roles, complete separation is not possible. However, research in actual classrooms, conducted by practicing teachers, remains valuable for the practical perspectives it can provide and the pragmatic solutions it may offer. Likewise, the purpose of the study was to provide a window into an instructional unit implemented in a real 124 school setting to improve students’ web evaluation skills, and to examine how this unique sample of students might respond to it. A second limitation of the study is in the artificial nature of the pre-unit and post-unit evaluation measures. Students evaluated preselected sites and evaluation was the only objective of the pre-unit and post-unit tasks. This makes findings in these tasks less generalizable to authentic information seeking tasks in which students choose their own topics, search for sites in unstructured time frames, then recursively read for comprehension, evaluate, and take notes for their own purposes. The embedded task was designed, in part, to address this limitation; however, it still required students to provide evaluation criteria for each site using structured materials designed to focus their attention on evaluation within the complex context of the information seeking process. These results may not reflect, therefore, the evaluation behaviors students may exhibit when the requirement to evaluate is not present or when such scaffolds are not provided. The embedded task does, however, provide insight into whether students can sustain attention to Key Criteria in the context of an actual assignment in which they must divide their attention among many subtasks, of which evaluation is one. Additionally, the pre-unit and post-unit tasks distilled students’ evaluative thinking in summary ratings justified by single statements justifying those ratings. Determining whether a justifying statement reflected particular aspects of evaluative thinking required simplifying complex, interconnected processes into separate codes, a necessarily artificial categorization which cannot truly reflect its complexity. However, the dialogic discussions served to encourage a more integrated, multifaceted approach to the process; that students approached the post-unit task with increased investigation and rated sites less favorably suggests that, despite the 125 measure’s limited ability to capture the complexity of students’ thinking, it did reveal movement toward more effortful and critical evaluation. A second limitation resulting from the necessary simplification of the coding process is in the coding of justifications as Off Task when they appeared to justify a stance for or against a controversial issue rather than to justify reliability. It is possible that such justifications emerged from the consideration of an argument’s strength, which could subsequently elucidate reliability. However, the statements did not indicate a closing of the logic process that is requisite to the evaluation of reliability and could not, therefore, be presumed to show it. That such responses all but disappeared in the post-unit evaluation task indicates that, even if students’ critical thinking revealed an approach toward reliability in the pre-unit task, they evaluated more deliberately in the post-unit task and articulated justifications more effectively. Implications and Future Research The positive results of this study suggest that cognitive flexibility theory can effectively be operationalized through teaching structures like the Socratic seminar that follow the principles of effective instruction for complex knowledge in ill-structured domains. The study suggests that such instruction does lead to cognitively flexible knowledge, as evidenced by the students’ continued emphasis on authorship and sourcing criteria even in the embedded evaluation task, which attempted to place the evaluation process in a more authentically applied context. However, whether the student changes evidenced in this study would be retained beyond a few weeks is a topic for future research. Studies should examine whether students’ increased concern about authorship and information sourcing, as well as their increased demonstration of investigative behaviors, would persist several months or years later, with different instructors, with different classes, and in different domains. The prevalence of the internet as an information 126 source in all aspects of life also calls into question whether the evaluation skills gained by students in this type of instruction are transferred to settings outside academia, where evaluation is completely voluntary and the goals for information gathering are far more diverse. The demonstration of critical thinking in the Socratic seminars, combined with the students’ own assertions of the seminars’ effectiveness for challenging and developing their evaluation skills, suggest that dialogic discussion may be particularly well aligned with the goal of developing flexible, adaptive knowledge. Considering the ill-structured nature of the web environment and the complexity of learning in the information age, educators and researchers may be advised to examine whether student-centered dialogic discussion might serve their instruction for other purposes and in other domains. If we are to prepare our students for learning in an ill-structured, complex world, and if dialogic discussion effectively serves such learning, its increased practice in classrooms is warranted. The current movement toward individualized learning programs that move students toward objectives based on individual assessments and at differing paces certainly has a place in education. However, if it supplants socially centered instructional methods, of which dialogic discussion is one example, educators may neglect methods that would otherwise help students to achieve important learning goals for the modern world. If dialogic discussion is particularly appropriate for exposing students to multiple perspectives and developing deeper understandings through collaborative inquiry, it would follow that teachers should be prepared to integrate such discussion into modern classrooms. This requires preservice teacher education and continuing professional development to cultivate teachers’ capacity for facilitating it. Work remains to move educators away from the traditional Initiate-Response-Feedback structure in which teachers distribute knowledge (Mehan, 1979; 127 Wells & Arauz, 2006) to student-centered discussions in which students collaborate to build collective knowledge. Finally, the importance of teaching students to think critically, to practice independent investigation, and to construct meaning for themselves cannot be understated. Walton (2017) advanced a revised theory of information literacy in which the reader “challenges received notions of the construction, communication and exchange of information and knowledge” (p. 137). Likewise, Cooke (2017) proposed a critical literacy in which “information consumers . . . consider the underlying power structures that shape information and consider the acquisition of agency that can come with the acquisition of quality information (Elmborg 2006; Tisdell 2008; Accardi, Drabinski, and Kumbier 2010)” (p. 218). Fundamentally, critical literacy empowers. Our era is one in which the independent, objective press is vilified, access to advanced education eludes greater numbers of Americans each year, fake news emerges from every corner of the web, and social mobility has decreased substantially. The divides in American society are great—between classes, ethnicities, cultures, geographic settings, religions, and political affiliations. I contend that such divisions make reductive, oversimplified assumptions more common and more dangerous. The goal of education is to prepare individuals not just for survival, but for meaningful participation in a democracy; likewise, the survival of a democracy and the wellbeing of its citizens depends critically on informed participation—participation that, by definition, involves the social construction of meaning through civic discourse. This study was designed toward that end, and its results suggest such instruction is both possible and meaningful. 128 APPENDICES 129 Appendix A: Contrasting Cases Activity Sheet Each person in your pair has an assigned role: Computer Navigator’s Name_______________________________ Writer’s Name________________________________ To do this activity, you must have ONE computer and ONE handout. Situate yourselves so that you can both see the computer screen comfortably. The computer operator will navigate the computer; the writer will read aloud and complete this handout. Scenario: Two students did research on the issue of whether teachers should be allowed to carry concealed weapons in schools. Student A was skilled at evaluating the reliability of web sites. Student B was unskilled at evaluating the reliability of web sites. Both students independently evaluated these two web sites: Site 1: http://www.johnstonfamilyministry.com/article/stop-school-shootings-letting-teachersand-principals-carry-guns Site 2: http://ideas.time.com/2012/12/21/viewpoint-arming-teachers-isnt-the-answer/ After rating the web sites, the students wrote reasons to explain their ratings. In the table below is a randomly ordered list of their reasons. Follow these steps: 1. The Computer Navigator should open each of the sites above in a separate tab to move between them easily. 2. You will have 10 minutes to examine the three sites together. Talk with your partner about what you notice and what you wonder. You may navigate wherever you would like as you examine the sites, but do so together, and share all your thoughts about the sites with your partner. 3. At the end of the 10 minutes, complete the back of this sheet. Follow the directions carefully! 130 Directions: • • Write an A in the box before the statement if you believe the reason was listed by Student A (skilled evaluator). Write a B if in the box before the statement if you believe the reason was listed by Student B (unskilled evaluator.) The Reader should read each item aloud, and as a pair you should come to consensus on each answer. You may go back to look at the sites if you wish. Student A (skilled)? Student B (unskilled)? Reason the student wrote for his/her rating. 1. “I completely agree that the only thing that can stop a bad guy with a gun is a good guy with a gun.” 2. “Lots of people read Time Magazine, so I think it’s probably reliable.” 3. “The people who wrote this article work at a university and study public safety issues.” 4. “The photo of the people who made this site look like people I can trust.” 5. “I notice this article is on the publication’s opinion page.” 6. “This page has a list of sources telling where the information in the article came from.” 7. “This article presents both sides of the issue with facts from actual research studies to back it up.” 8. “This page is pleasant to look at, with colorful and attractive photos.” 9. “This article is way too long.” 10. “This site was created by a well-known news magazine that is also published in print.” 11. “The creators of this site run a non-profit company whose goal is to present the truth about issues.” 12. “The writer of this article is very passionate about his opinion, which leads me to believe he’s reliable.” 13. “The creator of this site is a minister, and I trust ministers’ opinions on this issue.” 14. “I think this writer’s opinion is strongly influenced by his political views.” 15. “I don’t agree with this writer, but she does present some convincing facts to support her side of the issue.” 131 Appendix B: Inquiry-of-the-week Assignment IoW 1: MY NAME is OBJECTIVE: To evaluate the quality and reliability of web sites. TOPIC: Should high school football programs be discontinued? BACKGROUND: There is a controversy about the safety of high school football. Some people have argued that football is too dangerous a game for teens to play. They argue that long term physical harm can result from football injuries, especially for teens, whose bodies are still growing and developing in important ways. They also argue that football is considerably more dangerous than other sports. Proponents of high school football programs disagree that football is more dangerous than other sports. They also believe that the benefits of football (motivating kids to stay in school and teaching sportsmanship, teamwork, and self-discipline) outweigh the risks of physical injury. THE SITES: You are to examine and annotate these two web sites: http://portlandtribune.com/lor/49-opinion/242681-109622-high-school-footballs-benefits-outweigh-therisks http://cognoscenti.wbur.org/2014/10/15/ban-high-school-football-steve-almond WHAT TO DO: After examining each site for a few minutes, make 5 annotations on each of the sites using Diigo. Include at least 5 annotations per site, revealing important things you NOTICE or WONDER that might affect its reliability. There are two types of annotations you may make. • General annotations using the sticky note icon in the tool bar at the upper right corner of the page: Do this if you want to make an annotation about the page in general, about something you are unable to highlight, or about something you found through a link you followed. 132 • Specific annotations about highlighted text. In this case, first highlight the text by clicking the highlighter in the tool bar (shown above). Then, run your cursor over the highlighted text to add a sticky note to that specific place on the page. TIP: I recommend you begin your annotations with one of these sentence starters: I notice … I wonder … SUBMIT YOUR ANNOTATIONS IN THE BOXES BELOW. Please make sure they are NOT in bold print. Source 1: “High school football's benefits outweigh the risks” Link to my annotations: Paste annotations for Source 1 here: Source 2: “Sack those quarterbacks! The case for banning high school football” Link to my annotations: Paste annotations for Source 2 here: 133 Appendix C: Web Sites for IoWs and Socratic Seminar Discussions Seminar 1 Topic: Should football programs be discontinued in secondary schools? Site 1 Lake Oswego Review: High school football’s benefits outweigh the risks http://portlandtribune.com/lor/49-opinion/242681-109622-high-school-footballs-benefitsoutweigh-the-risks Site 2 Cognoscenti: Sack Those Quarterbacks! The Case for Banning High School Football http://cognoscenti.wbur.org/2014/10/15/ban-high-school-football-steve-almond 134 Seminar 2 Topic: Should virtual schools replace traditional schools? Site 1 MindShift: 5 Surprising Perspectives About Online Schools http://ww2.kqed.org/mindshift/2011/05/25/5-surprising-perspectives-online-schools/ Site 2 Detroit Free Press: Online charter school students falling behind peers http://www.freep.com/story/news/local/michigan/2015/10/27/online-charter-schoolstudents/74679050/ 135 Seminar 3 Topic: Are video games good for the teenage brain? Site 1 Intel Developer Zone: 6 Fascinating Facts About How Video Games are Good For You https://software.intel.com/en-us/articles/success-story-happy-jump Site 2 Discover: Video Games May Have Negative Effects on the Brain http://blogs.discovermagazine.com/d-brief/2015/05/20/video-games-brain/#.VsHtrfkrKUn 136 Appendix D: Template for Socratic Seminar Response IoW 1: Summary Response Name: Use the template below to construct a 3 paragraph response to the IoW 1 sites we discussed. You may add additional sentences and explanation as needed, and you may alter the wording if you wish. But please follow the construction of the template, including all elements it contains. To help you recall our conversation, you may review the transcript of our digital chat, which is posted on Google Classroom (see posting for Feb. 4, 2016). The online articles ____ by ____, and ____ by ____ both address the question/issue/topic of ____, but differ in opinion. ____ believes/asserts/argues that ____. He/she/they reason(s) that ____. Conversely, ____ believes/asserts/argues that ____. He/she/they reason(s) that ____. Personally, I agree with ____ that ____ (OR I am undecided about…). In considering the reliability of these two sites, I have come to the conclusion that ____. When examining the first site, the most important thing I noticed/wondered/questioned is _____. I think this is important because ____. (Insert explanation as needed.) Therefore, I found the first site to be _____. When examining the second site, the most important thing I noticed/wondered/questioned is _____. This is important because _____. (Insert explanation as needed.) Therefore, I found the second site to be _____. If I were researching this topic, I would _____ (explain what you would do with these sites, including if and how you would use them). 137 Appendix E: Sites for Pre-unit and Post-unit Evaluation Tasks Topic A: Should cell phones be banned in schools? Sites to support banning cell phones Sites to oppose banning cell phones http://www.theguardian.com/education/2015/ma y/16/schools-mobile-phones-academic-results The Guardian: Schools that ban mobile phones see better academic results (576 words) Qualities of this site: • site of respected news publication • references research study by the London School of Economics • study is directly linked to article http://www.ascd.org/publications/educationalleadership/oct10/vol68/num02/Cell-Phones-asTeaching-Tools.aspx ASCD: Education Today: Digitally speaking: Cell phones as teaching tools (881 words) Qualities of this site: • site of respected education publication • references Pew Internet and Life Project at bottom of article • author is veteran teacher and writer of book about Technology Integration methods for teachers http://www.care2.com/causes/5-reasons-to-bansmartphones-in-school.html Care2: 5 reasons to ban smartphones in school (618 words) Qualities of this site: • popular site for online petitioning of various causes • no sourcing • no information about author’s background • non-expert author http://theinnovativeeducator.blogspot.com/2013/ 02/finally-research-based-proof-that.html Lisa Nielsen: The Innovative Educator: Finally! Research-based proof that students use cell phones for LEARNING (941 words) Qualities of this site: • personal blog site of a woman who identifies herself as “the innovative educator,” who “found school boring” which “ticked her off.” • refers to one study on cell phone use by students, which was sponsored by Verizon • veteran teacher, but also sells books online and does paid appearances Higher reliability Lower reliability 138 Topic B: Should school homework be banned? Sites to support homework bans Sites to oppose homework bans http://news.stanford.edu/news/2014/march/toomuch-homework-031014.html Stanford news: Stanford research shows pitfalls of homework (763) Qualities of this site: • report of a research study by a wellregarded university • study published in a peer-reviewed journal • links directly to the study • links directly to email of the researcher for those wishing to contact her http://www.centerforpubliceducation.org/MainMenu/Instruction/What-research-says-about-thevalue-of-homework-At-a-glance Center for Public Education: What research says about homework (900) Qualities of this site: • CPE published by National School Boards Association, a non-profit group supporting public education • references multiple research studies directly • author of article not specifically identified, but About page lists organization’s objectives, leaders, and provides contact information http://www.leapfrog.com/en-us/learningpath/articles/homework-no-more Leapfrog: Homework no more (877) Qualities of this site: • author is site’s web editor and mother; no other credentials in education • company that sells devices and apps for educational games to parents • limited research provided; references other biased sources http://www.bdvmanagement.com/whyhomework-is-so-useful-list-of-strong-arguments/ bvdmanagement: Why homework is good (433) Qualities of this site: • author is unidentified • source is difficult to identify: no information about bvd management • links to sites that sell homework assignments, other homework help sites Higher reliability Lower reliability 139 Appendix F: Directions for Pre-unit and Post-unit Evaluation Tasks Topic A: Cell phones Imagine that you must write an argumentative essay taking a side on the issue of cell phones in school. You may argue one of the following… Cell phone use should be banned in the classroom. OR Cell phone use should be allowed in the classroom. Pre-evaluation Opinion: To begin, put a checkmark in front of the statement below that best characterizes your opinion about this topic RIGHT NOW: _____I strongly agree that cell phones should be banned in the classroom. _____I somewhat agree that cell phones should be banned in the classroom. _____I’m undecided about this issue. _____I somewhat agree that cell phones should be allowed in the classroom. _____I strongly agree that cell phones should be allowed in the classroom. 140 Imagine that you must write an argumentative essay taking a side on the issue of cell phones in school. You may argue one of the following… Cell phone use should be banned in the classroom. OR Cell phone use should be allowed in the classroom. • Now, imagine that you go to the Internet to learn about this topic and to gather resources for your essay. a. You have been required by your teacher to use RELIABLE resources. Reliable sources are trustworthy, high quality resources. b. You have been required by your teacher to consider and discuss the OPPOSING SIDE of the issue in your essay. • You have located four sources and must evaluate them for RELIABILITY. • Go to the link posted on Google Classroom. • Examine each of the sites and CAREFULLY and THOUGHTFULLY complete the evaluation form for EACH site. • Answer the final question on the back page of this packet. • NOTE: You have ONE class period to complete this task. To help you keep track of the time, you will be notified when 15, 30, and 45 minutes have passed. • NOTE: When you have finished the task, complete the last page of this packet. 141 When you have finished the task, put a checkmark before the statement that best characterizes your opinion AFTER you have evaluated all the sites: _____I strongly agree that cell phones should be banned in the classroom. _____I somewhat agree that cell phones should be banned in the classroom. _____I’m undecided about this issue. _____I somewhat agree that cell phones should be allowed in the classroom. _____I strongly agree that cell phones should be allowed in the classroom. 142 Topic B: Homework Imagine that you must write an argumentative essay taking a side on the issue of homework in school. You may argue one of the following… School homework assignments should be banned. OR School homework assignments should be allowed. Pre-evaluation Opinion: To begin, put a checkmark in front of the statement below that best characterizes your opinion about this topic BEFORE reading the sites: _____I strongly agree that homework should be banned. _____I somewhat agree that homework should be banned. _____I’m undecided about this issue. _____I somewhat agree that homework should be allowed. _____I strongly agree that homework should be allowed. 143 Imagine that you must write an argumentative essay taking a side on the issue of homework in school. You may argue one of the following… School homework assignments should be banned. OR School homework assignments should be allowed. • Now, imagine that you go to the Internet to learn about this topic and to gather resources for your essay. b. You have been required by your teacher to use RELIABLE resources. Reliable sources are trustworthy, high quality resources. c. You have been required by your teacher to consider and discuss the OPPOSING SIDE of the issue in your essay. • You have located four sources and must evaluate them for RELIABILITY. • Go to the link posted on Google Classroom. • Examine each of the sites and CAREFULLY and THOUGHTFULLY complete the evaluation form for EACH site. • Answer the final question on the back page of this packet. • NOTE: You have ONE class period to complete this task. To help you keep track of the time, you will be notified when 15, 30, and 45 minutes have passed. • NOTE: When you have finished the task, complete the last page of this packet. 144 When you have finished the task, put a checkmark before the statement that best characterizes your opinion AFTER you have evaluated all the sites: _____I strongly agree that homework should be banned. _____I somewhat agree that homework should be banned. _____I’m undecided about this issue. _____I somewhat agree that homework should be allowed. _____I strongly agree that homework should be allowed. 145 Appendix G: Evaluation Form for Pre-unit and Post-unit Evaluation Tasks Site name: _________________________________________ (You may abbreviate.) What qualities make this site more or less reliable? List as many reasons as you can in both columns. However, it is NOT necessary to have an equal number of reasons in each column. Write your reasons clearly enough that another person could easily understand your thinking. Write your reasons in a bulleted list with ONE reason per bullet. Full sentences are NOT required. What qualities make this site MORE reliable? List as many as you can. What qualities make this site LESS reliable? List as many as you can. Now, after reviewing the reasons listed above, CIRCLE ONE description that best matches your rating of this site’s overall reliability: After considering the qualities above, I find this site to be . . . Very reliable. Fairly reliable. Slightly more reliable than unreliable. Slightly more unreliable than reliable. 146 Fairly unreliable. Very unreliable. Appendix H: Source Model Measure Source Model Sample Directions: Answer each of the following questions in as much detail as you can remember. First item: Try to remember this web site: (miniature screen shot of site with title and source information obscured). a. What was this site’s OPINION about cell phones? If you don’t remember, just say that. b. What do you remember about the AUTHOR, SPONSOR, or CREATOR of this site? List as many things as you can remember. Remaining sites continued on the same page with identical questions for each (four sites per student). 147 Appendix I: Source Sheet for Independent Inquiry Task Possible Sources and Notes (PS = possible source) NAME: You will FIND AT LEAST 6 possible sources. (You may find more.) You will REJECT AT LEAST 2 unreliable sources. (You may reject more.) You will USE AT LEAST 4 reliable sources. (You may use more.) PS1 Site/source title: URL address: Evaluate: IN A BULLETED LIST below, write the 3 MOST IMPORTANT things affecting the reliability of this source: Will you use it? Yes Maybe No (highlight one) • • • IF YOU REJECT THE SITE, DO NOT COMPLETE THE SECTION BELOW! Go on to another source. 1. 2. 3. 4. USE Diigo to do the following: Read the source closely. When you come to a piece of EVIDENCE that supports one of your reasons, HIGHLIGHT IT. Then, add a COMMENT to that highlighted section. In the comment, PARAPHRASE the evidence IN YOUR OWN WORDS. When you are finished, COPY all your annotations from Diigo and PASTE them in the box below: (Students reproduced the table on this source sheet for each potential source accessed.) 148 Appendix J: Annotated Bibliography Instructions for Independent Inquiry Task The Final Expanded Works Cited 1. You will create your Works Cited using the Easybib add-on. (See the video for instructions!) 2. Then, below each source on your list, you will include a one-paragraph justification for your choosing that source. You must justify why it is reliable. You may use the template below. Several things contributed/affected/added to this site’s reliability. First, _______. This is important because ______. In addition, _______. This is important because _______. Finally, I noticed that ______. This matters/adds to/contributes to/affects the site’s reliability because _______. This is an example of how your final Expanded Works Cited will be formatted. Of course, your paragraphs will be complete! :-) 149 Appendix K: List of Codes * Grayscale indicates broad category **Green indicates subcategories within broad category Key Criteria ***Yellow indicates codes developed for second pass describing students’ Approach to Task CATEGORY, SUBCATEGORY, OR CODE Key Criteria for Evaluating Reliability* Authorship** author id_info expertise_authority Description Examples Evaluation criteria considered more desirable for evaluating the reliability of web sites Criteria considering the author or creator of the site considers whether identity of author is revealed, made clear or available, or if information about the author is available considers whether expertise, education, qualifications or experience warrants author(s) or creator(s) to be viewed as expert or authority on the topic conflict of interest_bias recognizes a potential conflict of interest that could lead to an information bias Site Qualities Criteria considering site qualities or identifiers considers whether the site is familiar, recognized, reputable familiarity with site contact_pub info available date_age of article or site site type affiliations_sponsorship They just have an author name, but nothing else about him. They tell about where the author works and who he is. The author is a researcher who probably learned about this topic in college. The author wrote a book about this so she must know something about it. They’re trying to sell homework help, so they probably like it. Lots of ads: they just want your money. The author got her info from Verizon. The Guardian is a well-known news site. I’ve never heard of this site before. site provides an "about" page; a way to You can contact them if you have contact site author, administrator, more questions. editor, publisher, affiliates; and/or There’s an “about” page that tells provides legal information about the site about the site. (copyright, etc.) considers the date of publication or age This was posted five years ago, so of the site or article the info may be old. identifies site type as an indication of It’s a non-profit (.org), so more reliability: blog, news, .org, .com reliable. It’s a personal blog. notes the site's connection to, The site is made by the U.S. sponsorship by, or affiliation with a government. reputed or recognized entity (including It’s a page from Stanford, where the government) as evidence for or there’s a lot of smart people. against reliability 150 purpose, intent, target audience considers the purpose or intent of the site: informational, promotional, etc. or the audience targeted by the site They want to improve education, which is a good cause. This site is made for parents, not teachers. published_vetted considers whether the source is formally The site is by a real magazine. published or info vetted Anyone can add what they want to this. Information Criteria considering the sources of Sourcing information provided on a site (not the site itself) identifies_links considers whether information sources, They have a list of sources at the source info references have been adequately linked, bottom. listed, or identified They have links in the text to go to the places where they got their info. age of source info considers the age or date of the sources The research they use was done ten providing information years ago so it could have changed. nature_quality of considers relevance, variety, type, or Their sources are not really about source info quality of sources referenced school, just about cell phones in general. All their info came from one source. All the sources they use link back to the same site and author. They have references to actual research studies. Content Qualities Criteria considering the informational content of the site, page, or article balance_objectivity considers extent to which source/author They only present one side of the seeks balance on the issue or addresses issue. only one side of the issue, presenting The site is really opinionated. his/her opinion only, or notes a bias He seems pretty biased to his own without explanation views. logic_strength of considers logic, strength, or It’s a pretty convincing article. argument reasonableness of argument, or degree He says cell phones can be used for to which it is compelling; considers good, but he doesn’t explain how. effectiveness of evidence in support of She presents a good case for claim homework bans. info corroborated information on the site is corroborated Their info lines up with what the by another site, or correlates with info other site said. on another source research_evidence considers whether information is They use tons of facts to back it up. based research or evidence based, including They give lots of statistics. references to facts, statistics, data, etc. It’s based on research, not just on what one person thinks. Extraneous Evaluation criteria considered less Criteria desirable for evaluating the reliability of web sites. aligns_prior_opinion references to whether the site aligns I agree that cell phones can be used with the reader's prior opinion for learning. information_needs judgment of reliability based on It has lots of information about cell relevance of information to user needs, phones. 151 functionality_ease of use popularity_user comments surface characteristics Vague_Inarticulate Off Task lists argument for or against issue restates info on page Investigative Behaviors*** clicks off page conducts off site search including usefulness, scope, quantity, length, relevance, or availability of information on the topic considers readability, site functionality, organization, or clarity of information for ease of use considers whether the source is popular, well-liked, highly frequented, shared with others, or whether users comment favorably on the site references to ads, sidebar content, appearance, attractiveness, color, font, pictures, distractibility, grammar or proofreading issues too vague to be coded or general positive/negative statement about a site or a quality of the site without justification Criteria suggests reader is not actually evaluating site instead of listing site strengths and weaknesses, lists arguments for or against the issue simply records information or facts learned on the site, without any indication of site evaluation based on those facts Comments by students indicating they investigated beyond the initial page of a site. Student literally refers to clicking on links that take him or her to a separate page. Student literally refers to conducting an off site search to learn more about an aspect of the site. 152 It doesn’t talk about how homework helps. It has subheadings to help you find things. It takes too long to load. It has a ton of likes, so it’s a popular site. The comments say positive things about it. There are too many pop-ups. The ads are distracting. It’s not very professional looking. Statements are listed. Can state whether or not it works for many types of people. Great site. If we all have cell phones, it will be harder for students to pay attention to the teacher. Cell phone use in schools went up 50% over the past three years. When I clicked on the author it didn’t show me anything about him. The links all take you to articles by the same person. I Googled the author and he works at Stanford. I searched the company and they do make video games. REFERENCES 153 REFERENCES Adler, M., Rougle, E., Kaiser, E., & Caughlan, S. (2004). Closing the gap between concept and practice: Toward more dialogic discussion in the language arts classroom. Journal of Adolescent & Adult Literacy, 47(4), 312–322. Retrieved from http://libproxy.uwyo.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true &db=eric&AN=EJ782185&site=ehost-live&scope=cite Afflerbach, P., & Cho, B. (2009). Identifying and describing constructively responsive comprehension strategies in new and traditional forms of reading. In S. E. Israel & G. G. Duffy (Eds.), Handbook of research on reading comprehension (pp. 69–90). New York: Routledge. Almond, S. (2014, October 15). Sack those quarterbacks! The case for banning high school football. Cognoscenti. Retrieved from http://www.wbur.org/cognoscenti/2014/10/15/ban-high-school-football-steve-almond Alvermann, D. E., Young, J. P., Weaver, D., Hinchman, K., Moore, D. W. M., Phelps, S. F., Zalewski, P. (1996). Middle and high school students’ perceptions of how they experience text-based discussions: A multicase study. Reading Research Quarterly, 31(3), 244–267. http://doi.org/10.1007/s11033-011-1068-8 Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. http://doi.org/10.1016/j.lindif.2013.01.007 Applebee, A. N., Langer, J. A., Nystrand, M., & Gamoran, A. (2003). Discussion-based approaches to developing understanding: Classroom instruction and student performance in middle and high school English. American Educational Research Journal, 40(3), 685–730. http://doi.org/10.3102/00028312040003685 Barnes, D. (1976). From communication to curriculum. Harmondsworth, UK: Penguin. Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30(1), 39–85. http://doi.org/10.1080/07370008.2011.636495 Becker, L. (2000). Effect sizes (Lecture notes). Retrieved from https://www.uccs.edu/lbecker/effect-size.html Bernard, S. (2011, May 25). 5 surprising perspectives about online schools. [Web log post]. Mind/Shift. Retrieved from http://ww2.kqed.org/mindshift/2011/05/25/5-surprisingperspectives-online-schools/ 154 Boyd, F. B., & Ikpeze, C. H. (2007). Navigating a literacy landscape: Teaching conceptual understanding with multiple text types. Learning and Instruction, 39(2), 217–248. Retrieved from http://www.informaworld.com/openurl?genre=article&id=doi:10.1080/108629607013 31951 Braasch, J. L. G., Bråten, I., Strømsø, H. I., Anmarkrud, Ø., & Ferguson, L. E. (2013). Promoting secondary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38(3), 180–195. http://doi.org/10.1016/j.cedpsych.2013.03.003 Braasch, J. L. G., Lawless, K. A., Goldman, S. R., Manning, F. H., Gomez, K. W., & MacLeod, S. M. (2009). Evaluating search results: An empirical analysis of middle school students’ use of source attributes to select useful sources. Journal of Educational Computing Research, 41(1), 63–82. http://doi.org/10.2190/EC.41.1.c Braasch, J. L. G., Rouet, J.-F., Vibert, N., & Britt, M. A. (2012). Readers’ use of source information in text comprehension. Memory & Cognition, 40(3), 450–465. http://doi.org/10.3758/s13421-011-0160-6 Bråten, I., Braasch, J. L. G., Strømsø, H. I., & Ferguson, L. E. (2014). Establishing trustworthiness when students read multiple documents containing conflicting scientific evidence. Reading Psychology, 36(4), 315–349. http://doi.org/10.1080/02702711.2013.864362 Bråten, I., Strømsø, H. I., & Britt, M. A. (2009). Trust matters: Examining the role of source evaluation in students’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44(1), 6–28. http://doi.org/10.1598/RRQ.44.1.1 Bråten, I., Strømsø, H. I., & Salmerón, L. (2011). Trust and mistrust when students read multiple information sources about climate change. Learning and Instruction, 21(2), 180–192. http://doi.org/10.1016/j.learninstruc.2010.02.002 Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20(4), 485–522. http://doi.org/10.1207/S1532690XCI2004_2 Britt, M. A., Richter, T., & Rouet, J.-F. (2014). Scientific literacy: The role of goal-directed reading and evaluation in understanding scientific information. Educational Psychologist, 49(2), 104–122. http://doi.org/10.1080/00461520.2014.916217 Brown, J. E. (2014). Investigating the validity of using NWEA’s MAP results to predict PSAT and ACT results. (Doctoral Dissertation, University of Illinois at Chicago). Retrieved from https://indigo.uic.edu/handle/10027/18964 Bulger, M. E. (2009). Online literacy and the trouble with information. (Doctoral Dissertation, University of California Santa Barbara.) Retrieved from https://searchproquest-com.proxy2.cl.msu.edu/docview/304856547?pq-origsite=summon 155 Burbules, N. C. (1997). Rhetorics of the Web : Hyperreading and critical literacy. In Ilana Snyder (Ed.), Page to screen: Taking literacy into the electronic era (pp. 102–122). New South Wales: Allen and Unwin. http://doi.org/10.4324/9780203201220 CCSSI. (2010). Common core state standards for English language arts & literacy in history/social studies, science, and technical subjects. Retrieved from http://www.corestandards.org/wp-content/uploads/ELA_Standards.pdf Center for Public Education. (2007, February 5). What research says about the value of homework: At a glance. Retrieved from http://www.centerforpubliceducation.org/Main-Menu/Instruction/What-research-saysabout-the-value-of-homework-At-a-glance Cho, B.-Y. (2014). Competent adolescent readers’ use of internet reading strategies: A thinkaloud study. Cognition and Instruction, 32(3), 253–289. http://doi.org/10.1080/07370008.2014.918133 Cho, B.-Y., Woodward, L., Li, D., & Barlow, W. (2017). Examining adolescents' strategic processing during online reading with a question-generating task. American Educational Research Journal, 54(4), 1-34. http://doi.org/10.3102/0002831217701694 Cho, B., & Afflerbach, P. (2015). Reading on the Internet: Realizing and constructing potential texts. Journal of Adolescent & Adult Literacy, 58(6), 504–517. Christensen, C. M., Johnson, C. W., & Horn, M. B. (2008). Disrupting class: How disruptive innovation will change the way the world learns. McGraw-Hill. Retrieved from http://www.amazon.com/dp/0071592067 Citydata.com. (2017). Stevensville, Michigan. Retrieved from http://www.citydata.com/city/Stevensville-Michigan.html Coiro, J. (2008). A beginning understanding of the interplay between offline and online reading comprehension when adolescents read for information on the Internet. University of Rhode Island. Retrieved from http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:No+Title#0 Coiro, J., Coscarelli, C., Maykel, C., & Forzani, E. (2015). Investigating criteria that seventh graders use to evaluate the quality of online information. Journal of Adolescent & Adult Literacy, 59(3). http://doi.org/10.1002/jaal.448 Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by sixth-grade skilled readers to search for and locate information on the Internet. Reading Research Quarterly, 42(2), 214–257. http://doi.org/10.1598/RRQ.42.2.2 Colwell, J., Hunt-Barron, S., & Reinking, D. (2013). Obstacles to developing digital literacy on the Internet in middle school science instruction. Journal of Literacy Research, 45(3), 295–324. http://doi.org/10.1177/1086296X13493273 156 Cooke, N. A. (2017). Posttruth, truthiness, and alternative facts: Information behavior and critical information consumption for a new age. Library Quarterly: Information, Community, Policy, 87(3), 211–221. Coombes, B. (2008). Generation Y: Are they really digital natives or more like digital refugees ? Voices, 7(1), 31–40. Cramer, H. (1946). The two-dimensional case. In Mathematical methods of statistics (pp. 260301). Princeton, NJ: Princeton University Press. Cubukcu, F. (2008). Enhancing vocabulary development and reading comprehension through metacognitive strategies. Issues in Educational Research, 18(1), 1–11. Davis, H. S. (2013). Discussion as a bridge: Strategies that engage adolescent and adult learning styles in the postsecondary classroom. Journal of the Scholarship of Teaching and Learning, 13(1), 68–76. Retrieved from http://josotl.indiana.edu/article/view/2157/3066 Deschryver, M., & Spiro, R. J. (2009). New forms of deep learning on the Web: Meeting the challenge of cognitive load in conditions of unfettered exploration in online multimedia environments. In R. Zheng (Ed.), Cognitive effects of multimedia learning (pp. 134–152). Portland, OR: Ringgold, Inc. Diigo. [online annotation tool]. Retrieved from https://www.diigo.com/ Doward, J. (2015, May 16). Schools that ban mobile phones see better academic results. The Guardian. Retrieved from https://www.theguardian.com/education/2015/may/16/schools-mobile-phonesacademic-results Duke, N. K., & Pearson, P. D. (2002). Effective practices for developing reading comprehension. In A. E. Farstrup & S. J. Samuels (Eds.), What research has to say about reading instruction (3rd ed.). International Reading Association. http://doi.org/10.1598/0872071774.10 Duke, N. K., Schmar-Dobler, E., & Zhang, S. (2006). Comprehension and technology. In M. C. McKenna, L. D. Labbo, R. D. Kieffer, & D. Reinking (Eds.), International handbook of literacy and technology. (pp. 317–326). Mahwah, NJ: Lawrence Erlbaum Associates. Eastin, M. S., Yang, M.-S., & Nathanson, A. I. (2006). Children of the net: An empirical exploration into the evaluation of internet content. Journal of Broadcasting & Electronic Media, 50(June), 211–230. http://doi.org/10.1207/s15506878jobem5002_3 Eliana P. (2015, June 12). 6 fascinating facts about how video games are good for you. Retrieved from https://software.intel.com/en-us/articles/success-story-happy-jump 157 Ericsson, K. A., & Kintsch, W. (1995). Long-term working memory. Psychological Review, 102(2), 211–245. http://doi.org/10.1037/0033-295X.102.2.211 Fabos, B. (2009). The price of information. In J. Coiro, M. Knobel, C. Lankshear, & D. Leu (Eds.), Handbook of research on new literacies (pp. 839–870). New York: Routledge. Feltovich, P. J.; Spiro, R.; Couson, R. (1993). Learning, teaching, and testing for complex conceptual understanding. In N. Frederiksen, R. Mislevy & I. Bejar (Eds.) Test theory for a new generation (pp. 187–217). Retrieved from https://www.academia.edu/2018562/Learning_teaching_and_testing_for_complex_co nceptual_understanding Ferguson, L. E., & Bråten, I. (2013). Student profiles of knowledge and epistemic beliefs: Changes and relations to multiple-text comprehension. Learning and Instruction, 25, 49–61. http://doi.org/10.1016/j.learninstruc.2012.11.003 Ferriter, W. M. (2010). Digitally speaking: Cell phones as teaching tools. Educational Leadership, 68(2), 85-86. Retrieved from http://www.ascd.org/publications/educational-leadership/oct10/vol68/num02/CellPhones-as-Teaching-Tools.aspx Flanagin, A. and Metzger, M. (2010). Kids and credibility: An empirical examination of youth, digital media use, and information credibility. Cambridge, MA: MIT Press. Retrieved from https://dmlcentral.net/wpcontent/uploads/files/Kids_and_Credibility.pdf Fogg, B. J., Soohoo, C., Danielson, D. R., Marable, L., Stanford, J., & Tauber, E. R. (2003). How do users evaluate the credibility of Web sites?: A study with over 2,500 participants. Paper presented at the 2003 Conference on Designing for User Experiences. San Francisco, CA. http://doi.org/10.1145/997078.997097 Ford, C. L., & Yore, L. D. (2012). Toward convergence of critical thinking, metacognition, and reflection: Illustration from natural and social sciences, teacher education, and classroom practice. In A. Zohar & Y. J. Dori (Eds.), Metacognition in science education (Vol. 40, pp. 251–271). Dordrecht: Springer Netherlands. http://doi.org/10.1007/978-94-007-2132-6 Forzani, E., & Maykel, C. (2013). Evaluating a representative state sample of Connecticut seventh-grade students' ability to critically evaluate online information. Storrs, CT: University of Connecticut. Gadgil, S., Nokes-Malach, T. J., & Chi, M. T. H. (2012). Effectiveness of holistic mental model confrontation in driving conceptual change. Learning and Instruction, 22(1), 47–61. http://doi.org/10.1016/j.learninstruc.2011.06.002 Gainer, J. S. (2010). Critical media literacy in middle school: Exploring the politics of representation. Journal of Adolescent & Adult Literacy, 53(5), 364–373. http://doi.org/10.1598/JA 158 Gallagher, K., & Allington, R. L. (2009). Readicide: How schools are killing reading and what you can do about it. Portland, ME: Stenhouse Publishers. Gerjets, P., Kammerer, Y., & Werner, B. (2010). Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking-aloud protocols and eye-tracking data. Learning and Instruction, 21, 220–231. http://doi.org/10.1016/j.learninstruc.2010.02.005 Giroux, H. A. (2010). Rethinking education as the practice of freedom: Paulo Freire and the promise of critical pedagogy. Policy Futures in Education, 8(6), 715. http://doi.org/10.2304/pfie.2010.8.6.715 Goldman, S. R. (2012). Adolescent literacy: Learning and understanding content. The Future of Children. 22(2), 89–116. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/23057133 Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47(4), 356–381. http://doi.org/10.1002/RRQ.027 Gorman-Smith, D. & McLaughlin, M. (2012, December 21). Viewpoint: Arming teachers isn’t the answer. Time. Retrieved from http://ideas.time.com/2012/12/21/viewpointarming-teachers-isnt-the-answer/ Greene, J. A., Costa, L.-J., Robertson, J., Pan, Y., & Deekens, V. M. (2010). Exploring relations among college students’ prior knowledge, implicit theories of intelligence, and self-regulated learning in a hypermedia environment. Computers & Education, 55(3), 1027–1043. http://doi.org/10.1016/j.compedu.2010.04.013 Gregory, A., & Cahill, M. A. (2009). Constructing critical literacy: Self-reflexive ways for curriculum and pedagogy. Critical Literacy: Theories and Practices, 3(2), 6–16. Retrieved from http://criticalliteracy.freehostia.com/index.php?journal=criticalliteracy&page=article& op=viewArticle&path[]=35 Haria, P. D., & Midgette, E. (2014). A genre-specific reading comprehension strategy to enhance struggling fifth-grade readers’ ability to critically analyze argumentative text. Reading & Writing Quarterly, 30(4), 297–327. http://doi.org/10.1080/10573569.2013.818908 Hartman, D. K. (1995). Eight readers reading: The intertextual links of proficient readers reading multiple passages. Reading Research Quarterly, 30(3), 520–561. Retrieved from http://www.jstor.org/stable/747631 Hartman, D. K., Morsink, P. M., & Zheng, J. (2010). From print to pixels: The evolution of cognitive conceptions of reading comprehension. In E. A. Baker (Ed.), The new 159 literacies: Multiple perspectives on research and practice (pp. 131–164). New York: Guilford Press. Higgins, L. (2015, October 27). Online charter school students falling behind peers. Detroit Free Press. Retrieved from http://www.freep.com/story/news/local/michigan/2015/10/27/online-charter-schoolstudents/74679050/ Hirsh, S. G. (1999). Children’s relevance criteria and information seeking on electronic resources. Journal of the American Society for Information Science, 50(14), 1265– 1283. http://doi.org/10.1002/(SICI)1097-4571(1999)50:14<1265::AIDASI2>3.3.CO;2-5 Jacobson, M. J., & Spiro, R. J. (1995). Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: An empirical investigation. Journal of Educational Computing Research, 12(4). Retrieved from http://doi.org/10.2190/4T1BHBP0-3F7E-J4PN Johnson, A. K., & Putnam, R. T. (2012). Sixth graders’ critical evaluation of Internet sites. Poster presented at the 62nd Annual Conference of the Literacy Research Association. San Daniel, CA. Johnson, F., Sbaffi, L., & Rowley, J. (2015). Students’ approaches to the evaluation of digital information: Insights from their trust judgments. British Journal of Educational Technology, 47(6). http://doi.org/10.1111/bjet.12306 Johnson, T. E., Archibald, T. N., & Tenenbaum, G. (2010). Individual and team annotation effects on students’ reading comprehension, critical thinking, and meta-cognitive skills. Computers in Human Behavior, 26(6), 1496–1507. http://doi.org/10.1016/j.chb.2010.05.014 Johnston, P. (2006, October 8). Stop school shootings: Let teachers carry guns [Web log post]. Retrieved from http://www.johnstonfamilyministry.com/article/stop-schoolshootings-letting-teachers-and-principals-carry-guns Julien, H., & Barker, S. (2009). How high-school students find and evaluate scientific information: A basis for information literacy skills development. Library & Information Science Research, 31(1), 12–17. http://doi.org/10.1016/j.lisr.2008.10.008 Kellner, D., & Share, J. (2007). Critical media literacy, democracy, and the reconstruction of education. In D. Macedo & S.R.Steinberg (Eds.) Media literacy: A reader (pp. 3–23). Retrieved from http://gseis.ucla.edu/sudikoff/archive/pdfs/philosophy/Summary_Kellner_CritLitDem ocracy.pdf Kienhues, D., Bromme, R., & Stahl, E. (2008). Changing epistemological beliefs: The unexpected impact of a short-term intervention. British Journal of Educational 160 Psychology, 78(Pt 4), 545–565. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/18166142 Kiili, C., Laurinen, L., & Marttunen, M. (2007). How students evaluate credibility and relevance of information on the Internet. IADIS International Conference on Cognition and Exploratory Learning in the Digital Age (CELDA 2007), 155–162. Retrieved from https://www.researchgate.net/publication/268343928_How_students_evaluate_credibi lity_and_relevance_of_information_on_the_internet Kiili, C., Laurinen, L., & Marttunen, M. (2009). Skillful reader is metacognitively competent. In L. T. W. Hin & R. Subramaniam (Eds.), Handbook of research on new media literacy at the k-12 level (pp. 654–668). Hershey, NY: Information Science Reference. Kim, J. Y. (2014). Strategy and bias in comprehension of multiple texts: How do readers with topic beliefs use strategies? (Doctoral Dissertation, Universi of Maryland). Retrieved from http://drum.lib.umd.edu/handle/1903/15746 Kim, K. S., & Sin, S. C. J. (2011). Selecting quality sources: Bridging the gap between the perception and use of information sources. Journal of Information Science, 37(2), 178–188. http://doi.org/10.1177/0165551511400958 Kintsch, W. (1974). The representation of meaning in memory. Hillsdale, NJ: Lawrence Erlbaum Associates. Kintsch, W., & van Dijk, T. A. (1978). Toward a model of text comprehension and production. Psychological Review, 85(5), 363–394. Kolodner, J. L. (1997). Educational implications of analogy. A view from case-based reasoning. The American Psychologist, 52(1), 57–66. http://doi.org/10.1037/0003066X.52.1.57 Kuhn, D. (2000). Theory of mind, metacognition, and reasoning: A life-span perspective. In P. Mitchell & K. J. Riggs (Eds.), Childrens’ reasoning and the mind (pp. 301–326). UK: Psychology Press/Taylor & Francis. Kuhn, D. (2015). Thinking together and alone. Educational Researcher, 44(1), 46–53. Kuhn, D., & Dean, D. (2004). Metacognition: A bridge between cognitive psychology and educational practice. Theory Into Practice, 43(4), 268–274. Kuiper, E., Volman, M., & Terwel, J. (2005). The Web as an information resource in k-12 education: Strategies for supporting students in searching and processing information. Review of Educational Research, 75(3), 285–328. Kurtz, K. J., Miao, C.-H., & Gentner, D. (2001). Learning by analogical bootstrapping. Journal of the Learning Sciences, 10(4), 417–446. http://doi.org/10.1207/S15327809JLS1004new_2 161 Leu, D. J., Kinzer, C. K., Coiro, J. L., & Cammack, D. W. (2004). Toward a theory of new literacies emerging from the internet and other information and communication technologies. In Ruddell, R. B., & Unrau, N. (Eds.) Theoretical models and processes of reading. Newark, DE: International Reading Association. Litman, C., & Greenleaf, C. (2017). Argumentation tasks in secondary English language arts, history, and science: Variations in instructional focus and inquiry space. Reading Research Quarterly, 52(2), 1–20. http://doi.org/10.1002/rrq.187 Lombardi, D., Sinatra, G. M., & Nussbaum, E. M. (2013). Plausibility reappraisals and shifts in middle school students’ climate change conceptions. Learning and Instruction, 27, 50–62. http://doi.org/10.1016/j.learninstruc.2013.03.001 Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37(11), 2098–2109. http://doi.org/10.1037//00223514.37.11.2098 Macedo-Rouet, M., Braasch, J. L. G., Britt, M. A., & Rouet, J.-F. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31(2), 204–226. http://doi.org/10.1080/07370008.2013.769995 Magno, C. (2010). The role of metacognitive skills in developing critical thinking. Metacognition and Learning, 5(2), 137–156. http://doi.org/10.1007/s11409-010-90544 Makinster, J. G., Beghetto, R. a, & Plucker, J. a. (2002). Why can’t I find Newton’s third law? Case studies of students’ use of the Web as a science resource. Journal of Science Education and Technology, 11(2), 155–172. Mason, L., Boldrin, A., & Ariasi, N. (2009). Epistemic metacognition in context: Evaluating and learning online information. Metacognition and Learning, 5(1), 67–90. http://doi.org/10.1007/s11409-009-9048-2 Mason, L., Junyent, A. A., & Tornatora, M. C. (2014). Epistemic evaluation and comprehension of web-source information on controversial science-related topics: Effects of a short-term instructional intervention. Computers & Education, 76, 143– 157. http://doi.org/10.1016/j.compedu.2014.03.016 Mathews, K. (2015, May 23). 5 reasons to ban cell phones in school. Retrieved from http://www.care2.com/causes/5-reasons-to-ban-smartphones-in-school.html Mehan, H. (1979). ‘What time is it, Denise?”: Asking known information questions in classroom discourse. Theory Into Practice, 18(4), 285–294. http://doi.org/10.1080/00405847909542846 162 Meola, M. (2004). Chucking the checklist: A contextual approach to teaching undergraduates Web-site evaluation. Portal: Libraries and the Academy, 4(3), 331–344. http://doi.org/10.1353/pla.2004.0055 Metzger, M. J., Flanagin, A. J., & Nekmat, E. (2015). Comparative optimism in online credibility evaluation among parents and children. Journal of Broadcasting & Electronic Media, 59(3), 509–529. http://doi.org/10.1080/08838151.2015.1054995 Metzger, M. J., Flanagin, A. J., Markov, A., Grossman, R., & Bulger, M. (2015). Believing the Unbelievable: Understanding young people’s information literacy beliefs and practices in the United States. Journal of Children and Media, 9(3), 325–348. http://doi.org/10.1080/17482798.2015.1056817 Muis, K. R., Pekrun, R., Sinatra, G. M., Azevedo, R., Trevors, G., Meier, E., & Heddy, B. C. (2015). The curious case of climate change: Testing a theoretical model of epistemic beliefs, epistemic emotions, and complex learning. Learning and Instruction, 39, 168– 183. http://doi.org/10.1016/j.learninstruc.2015.06.003 Nielsen, L. (2013, February 16). Finally! Research-based proof that students use cell phones for learning [Web log post]. Retrieved from https://theinnovativeeducator.blogspot.com/2013/02/finally-research-based-proofthat.html Nussbaum, E. M., & Sinatra, G. M. (2003). Argument and conceptual engagement. Contemporary Educational Psychology, 28, 384–395. http://doi.org/10.1016/S0361476X(02)00038-3 Nystrand, M. (2006). Research on the role of classroom discourse as it affects reading comprehension. Research in the Teaching of English, 40(4), 392–412. Retrieved from http://www.jstor.org/stable/40171709 Nystrand, M., & Gamoran, A. (1991a). Student engagement: When recitation becomes conversation. In Effective teaching: Current research (pp. 257–276). Retrieved from http://www.eric.ed.gov/ERICWebPortal/recordDetail?accno=ED323581 Nystrand, M., & Gamoran, A. (1991b). Instructional discourse, student engagement, and literature achievement. Research in the Teaching of English, 25(3), 261–290. http://doi.org/10.2307/40171413 Paek, P. (2005). Recent trends in comparability studies. Pearson Educational Measurement. Retrieved from http://www.pearsonassessments.com/NR/rdonlyres/5FC04F5A-E79D45FE-8484-07AACAE2DA75/0/TrendsCompStudies_rr0505.pdf Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 1(2), 117–175. 163 Parker, C. B. (2014, March 10). Stanford research shows pitfalls of homework. Stanford News. Retrieved from http://news.stanford.edu/2014/03/10/too-much-homework031014/ Paul, J., Macedo-Rouet, M., Rouet, J. F., & Stadtler, M. (2017). Why attend to source information when reading online? The perspective of ninth grade students from two different countries. Computers and Education, 113, 339–354. http://doi.org/10.1016/j.compedu.2017.05.020 Porter-O’Donnell, C. (2014). Beyond the yellow highlighter: Teaching annotation skills to improve comprehension. The English Journal, 93(5), 82–89. http://doi.org/10.2307/4128941 Pressley, M., & Gaskins, I. W. (2006). Metacognitively competent reading comprehension is constructively responsive reading: How can such reading be developed in students? Metacognition and Learning, 1(1), 99–113. http://doi.org/10.1007/s11409-006-7263-7 Reynolds, R. (2011). Digital textbooks reading the tipping point in the U.S. higher education: A revised five-year projection. Retrieved June 8, 2011, from http://blog.xplana.com/reports/digital-textbooks-reach-the-tipping-point-in-the-u-shigher-education-a-revised-5-year-projection/ Reznitskaya, A., Kuo, L., Clark, A., Miller, B., Jadallah, M., Anderson, R. C., & Nguyen‐ Jahiel, K. (2009). Collaborative reasoning: A dialogic approach to group discussions. Cambridge Journal of Education, 39(1), 29–48. http://doi.org/10.1080/03057640802701952 Rheingold, H. (2010). Attention, and other 21st-century social media literacies. Educause Review, 45(5), 14–24. Retrieved from http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolum e45/AttentionandOther21stCenturySo/213922 Richter, T. (2011). Cognitive flexibility and epistemic validation in learning from multiple texts. In J. Elen, E. Stahl, R. Bromme, & G. Clarebout (Eds.), Links between beliefs and cognitive flexibility. Springer: Netherlands. http://doi.org/10.1007/978-94-0071793-0 Riley, E. (2014, December 4). High school football’s benefits outweigh the risks. Lake Otswego Review. Retrieved from http://portlandtribune.com/lor/49-opinion/242681109622-high-school-footballs-benefits-outweigh-the-risks Rittle-Johnson, B., & Star, J. R. (2011). The power of comparison in learning and instruction: Learning outcomes supported by different types of comparisons. Psychology of Learning and Motivation, 55, 199-225. http://doi.org/10.1016/B978-0-12-3876911.00007-7 164 Rouet, A. J., Favart, M., Britt, M. A., & Perfetti, C. A. (2014). Studying and using multiple documents in history: Effects of discipline expertise. Cognition and Instruction, 15(1), 85–106. http://doi.org: 10.1207/s1532690xci1501_3 Rouet, J.-F. (2006). The skills of document use: From text comprehension to Web-based learning. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers. Rouet, J.-F., Ros, C., Goumi, A., Macedo-Rouet, M., & Dinet, J. (2011). The influence of surface and deep cues on primary and secondary school students’ assessment of relevance in Web menus. Learning and Instruction, 21(2), 205–219. http://doi.org/10.1016/j.learninstruc.2010.02.007 Salmerón, L., Kammerer, Y., & García-carrión, P. (2013). Searching the Web for conflicting topics: Page and user factors. Computers in Human Behavior, 29, 2161–2171. Sanchez, C. A., Wiley, J., & Goldman, S. R. (2006). Teaching students to evaluate source reliability during Internet research tasks. Paper presented at the 7th International Conference on Learning Sciences, Bloomington, IN. Scholz-Crane, A. (1998). Evaluating the future: A preliminary study of how undergraduate students evaluate web sources. Reference Services Review, 26(3/4), 53–60. Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4), 475–5223. http://doi.org/10.1207/s1532690xci1604_4 Smith, A., McGeeney, K., Duggan, M., Rainie, L., & Keeter, S. (2015). US smartphone use in 2015. Retrieved from http://www.pewinternet.org/files/2015/03/PI_Smartphones_0401151.pdf Spiro, R. J., Feltovich, P. J., Jacobson, M. J., & Coulson, R. L. (1992). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. In T. M. Duffy & D. H. Jonassen (Eds.), Constructivism and the technology of instruction: A conversation (pp. 57–75). Hillsdale, NJ. Retrieved from http://74.125.155.132/scholar?q=cache:At4p7m5PiwJ:scholar.google.com/&hl=en&as_sdt=0,23 Spiro, R. J. (2004). Principled pluralism for adaptive flexibility in teaching and learning to read. In Ruddel, R. B. & Unrau, N.(Eds.), Theoretical models and processes of reading (pp. 654–659). Newark, DE: International Reading Association. Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson, D. K. (1998). Cognitive flexibility theory: Advanced knowledge acquisition in ill-structured domains. Tenth Annual Conference of the Cognitive Science Society, 375–383. Retrieved from https://www.academia.edu/2552778/Cognitive_Flexibility_Theory_Advanced_Knowl edge_Acquisition_in_Ill-Structured_Domains 165 Spiro, R. J., & Deschryver, M. (2009). Constructivism: When it’s the wrong idea and when it’s the only idea. In T. Richter (Ed.), Constructivist instruction: Success or failure? (pp. 106–120). Florence, KY, USA: Routledge. Spiro, R. J., Feltovich, P. J., & Coulson, R. L. (1996). Two epistemic world views: Prefigurative schemas and learning in complex domains. Applied Cognitive Psychology, 10(7), S51–S61. Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the WWW: The role of metacognition in the formation of documents models. The International Journal of Computer-Supported Collaborative Learning, 2(2–3), 191–210. http://doi.org/10.1007/s11412-007-9015-3 Strømsø, H. I., Bråten, I., & Britt, M. A. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 20(3), 192–204. http://doi.org/10.1016/j.learninstruc.2009.02.001 Sullivan-Cross, A. Homework no more. Retrieved from http://www.leapfrog.com/enus/learning-path/articles/homework-no-more Taber, C. S., & Lodge, M. (2006). Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science, 50(3), 755–769. Thomas, B. (2015, May 20). Video games may have negative affects on the brain. Discover. Retrieved from http://blogs.discovermagazine.com/d-brief/2015/05/20/video-gamesbrain/#.WaCR3lGGOUn van Strien, J. L. H., Brand-Gruwel, S., & Boshuizen, H. (2014). Dealing with conflicting information from multiple nonlinear texts: Effects of prior attitudes. Computers in Human Behavior, 32, 101–111. http://doi.org/10.1016/j.chb.2013.11.021 Vygotsky, L. S. & Cole, M. (1978). Mind in society: The development of higher psychological processes. Cambridge: Harvard University Press. Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2009). How students evaluate information and sources when searching the World Wide Web for information. Computers & Education, 52(1), 234–246. http://doi.org/10.1016/j.compedu.2008.08.003 Walton, G. (2017). Information literacy is a subversive activity: Developing a research-based theory of information discernment. Journal of Information Literacy, 11(1), 137–155. http://doi.org/10.11645/11.1.2188 Wells, G., & Arauz, R. M. (2006). Dialogue in the classroom. Journal of the Learning Sciences, 15(3), 379–428. http://doi.org/10.1207/s15327809jls1503_3 Why homework is good: A list of arguments. Bdv management. Retrieved from http://www.bdvmanagement.com/why-homework-is-so-useful-list-of-strongarguments/ 166 Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46(4), 1060-1106. http://doi.org/10.3102/0002831209333183 Williams, M. (2006). Will tablet PCs replace textbooks? PC World. Retrieved from http://www.pcworld.com/article/125492/will_tablet_pcs_replace_textbooks.html Wineburg, S. S. (1998). Reading Abraham Lincoln: An expert / expert study in the interpretation of historical texts. Cognitive Science, 22(3), 319–346. Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83(1), 73–87. Zawilinski, L., Caleb, A., O’Byrne, I., McVerry, G., Nierlich, T., & Leu, D. J. (2007, November). Toward a taxonomy of online reading comprehension strategies. In 53rd Annual Meeting of the National Reading Conference (Austin, TX). Zhang, M., & Quintana, C. (2012). Scaffolding strategies for supporting middle school students’ online inquiry processes. Computers & Education, 58(1), 181–196. http://doi.org/10.1016/j.compedu.2011.07.016 Zwiers, J. (2008). Building academic language: Essential practices for content classrooms, grades 5-12. San Francisco: Jossey-Bass. 167