MAJOR TOM, YOU’VE REALLY MADE THE GRADE: SCIENCE & ENGINEERING PRACTICES IN VIDEOGAME AFFINITY SPACES By Elizabeth Owens Boltz A DISSERTATION Michigan State University in partial fulfillment of the requirements Submitted to for the degree of Educational Psychology and Educational Technology—Doctor of Philosophy 2019 ABSTRACT MAJOR TOM, YOU’VE REALLY MADE THE GRADE: SCIENCE & ENGINEERING PRACTICES IN VIDEOGAME AFFINITY SPACES By Elizabeth Owens Boltz Research suggests that videogames can promote science learning; however, studies have largely focused on conceptual knowledge rather than the development of the science and engineering practices emphasized in current standards for science education. Previous studies have also tended to exclude informal learning that can occur in the spaces surrounding videogames. Given the crucial role that social context plays in scientific sensemaking, affinity spaces like online discussion forums seem to offer particular affordances for players to engage in science and engineering practices within a sphere of public reasoning. This study investigated how science and engineering practices manifested in these kinds of communities through a mixed-methods case study of two online gaming forums: the Portal 2 Steam Community and the official forum for Kerbal Space Program. After collecting the 1000 most recent posts made to each forum, I employed content analysis with an a priori protocol coding scheme based on the Next Generation Science Standards’ (NGSS) Science and Engineering Practices (SEP) and Nature of Science Connections (NOS) to interpret and code forum posts. I identified whether evidence of SEP was focused on science and/or engineering, rated posts for their depth of engagement with the practices and for quality, and characterized posts as being nurturing, elitist, or neutral in terms of tone. I then used descriptive statistics to identify patterns in the data, and rich description to qualitatively analyze each online community and the games with which they are affiliated. The resulting study combined quantitative statistics to describe trends in the forums with qualitative analysis to ground the results within the context of the two specific affinity spaces. Across both forums, I found that roughly half of the posts showed evidence of science and engineering practices. Most of that evidence had an engineering focus. The most common practices evidenced in the forums were Asking Questions and Defining Problems, Constructing Explanations and Designing Solutions, and Obtaining, Communicating, and Evaluating Information. I found very little evidence of engagement with the Nature of Science in either forum. Posts from KSP were twice as likely to engage with science and engineering practices compared to those for Portal 2. KSP posts showed significantly more evidence of seven out of the eight Science and Engineering practices and significantly more evidence of Nature of Science Connections overall. The depth and quality of KSP posts were significantly higher than those from the Portal 2 forum. The results of this study offer practical implications for the design of learning environments and for teacher education. The differences between posts from Portal 2 and KSP suggest that some features of games and their affiliated affinity spaces are more effective in fostering engagement with science and engineering practices than others. Educators and designers may therefore wish to leverage features that appear to be most conducive for their intended learning goals. Additionally, these spaces may offer opportunities for teachers to identify and notice authentic ways learners engage with the practices in interest-driven spaces. This study also contributes to theory by further complicating our understanding of affinity spaces and addressing the nuanced ways that learning can occur outside formal educational structures. Licensed under a CC BY-SA 4.0 International License by ELIZABETH OWENS BOLTZ 2019 To Brandon, for loving and believing in me even when I don’t, for your strength and support as we lived apart from each other, for reminding me that happiness is more important than prestige or publications, and for never once asking how my dissertation was going. v To my committee— ACKNOWLEDGEMENTS Dr. Matt Koehler, my advisor and committee chair, for his patience in guiding someone who wasn’t always quite sure where she wanted to go; Dr. Cindy Okolo, for her dedication to supporting all learners and for helping me find my footing on this journey; Dr. Rand Spiro, whose cheer and sparkling wit were often a beacon of light on gloomy days; and Dr. Christina Schwarz, for her warmth, generosity, and commitment to equity in science learning. To my colleagues and friends—Leigh, Punya, and Aman for their confidence; my EPET classmates; the MAET students, alumni, and instructors who continually inspire me; and especially my Board of Advisors, Candace, Maria, and Brittany. To all of my friends for their support and understanding—especially Heather, Amanda, Kate, Kris, Ardin, Emily, and Crystal. To Pawn Pawn and my rock star friends for providing much- needed creative breaks. In memory of Princess, Luci, Faramir, and especially Scorpy—who didn’t make it all the way through this journey with me, but were by my side during the loneliest of times. And to Mike Carter, whose kindness and love of Miss Marvel will never be forgotten. The Force will be with you…always. vi To my parents, Ralph and Emilie Owens—for their unconditional love; for instilling creativity, curiosity, and compassion in their children; for that VHS tape of Star Wars; and for the Apple IIGS. To my siblings, Colleen (and Adam, Fiona, and Lucas) and Mike—some of the only people I know who (like me) communicate through movie references and remember the tooth brushing game on the Commodore. And to my extended family, Terry, Gina, Deb, and the Wenzler and Owens families, for always making me feel welcome. To those who are oppressed, harassed, and harmed for who they are—I will continue to interrogate my privilege and stand with you in solidarity and activism. To those who worked so hard to convince me of my worthlessness because I was different—that’s Doctor Liz to you. vii TABLE OF CONTENTS LIST OF TABLES ...................................................................................................................... x LIST OF FIGURES ................................................................................................................... xi Introduction ................................................................................................................................ 1 Literature Review ....................................................................................................................... 4 Science as a Way of Knowing ................................................................................................. 4 Science and Engineering Practices .......................................................................................... 6 Videogame Affinity Spaces and Science and Engineering Practices ........................................ 9 Context, Purpose, and Research Questions ................................................................................ 14 Method ..................................................................................................................................... 16 Research Design ................................................................................................................... 16 Positionality Statement .......................................................................................................... 16 Definitions of Key Terms ...................................................................................................... 17 Data Sources and Collection ................................................................................................. 17 Coding Scheme ..................................................................................................................... 21 Quality and Ethics of Online Data ......................................................................................... 29 Reliability and Validity ......................................................................................................... 30 Results ...................................................................................................................................... 32 RQ1: To what extent do discussions in the Portal 2 forums show evidence of Science and Engineering Practices and Nature of Science Connections? ............................................... 32 RQ2: To what extent do discussions in the Kerbal Space Program forums show evidence of Science and Engineering Practices and Nature of Science Connections? ............................ 36 RQ3: How do the forums differ with respect to the extent to which users engage in Science and Engineering Practices and Nature of Science Connections? ......................................... 42 RQ4: How can the quality of conversations in these forums be characterized? ...................... 44 Discussion ................................................................................................................................ 54 Synthesis of Findings ............................................................................................................ 54 Implications for Practice ....................................................................................................... 57 Implications for Theory ......................................................................................................... 61 Limitations ............................................................................................................................... 63 Suggestions for Future Research ............................................................................................... 65 Conclusion ................................................................................................................................ 66 APPENDICES .......................................................................................................................... 67 viii APPENDIX A: Inter-rater Reliability Results........................................................................ 68 APPENDIX B: Correlations Between Codes ......................................................................... 71 APPENDIX C: Correlations Between KSP Subforums and Practices .................................... 73 REFERENCES ......................................................................................................................... 74 ix LIST OF TABLES Table 1: Science and Engineering Practices Codes .................................................................... 22 Table 2: Nature of Science Connections Codes ......................................................................... 23 Table 3: Depth Rubric ............................................................................................................... 26 Table 4: Quality Rubric ............................................................................................................. 26 Table 5: Affinity Space Characterization Codes ........................................................................ 28 Table 6: Final Round of Inter-rater Reliability for Top-level Categories .................................... 31 Table 7: Percentages of Posts Coded for Science and Engineering Practices in Portal 2 Forums 32 Table 8: Percentages of Posts Coded for Nature of Science Connections in Portal 2 Forums ..... 33 Table 9: Percentages of Posts Coded for Science and Engineering Practices in KSP Forums ..... 36 Table 10: Percentages of Posts Coded for Nature of Science Connections in KSP Forums ........ 37 Table 11: Coding Example ........................................................................................................ 40 Each Forum .............................................................................................................................. 43 Each Forum .............................................................................................................................. 44 Table 14: Comparison of Average Ratings for Depth and Quality Categories ............................ 45 Table 15: Comparison of Percentages for Characterization Codes ............................................. 47 Table 16: User Post Count Details Within Each Forum’s 1000-post Sample ............................. 52 Table 17: Final Round of Inter-rater Reliability Across All Categories ...................................... 68 Table 18: Correlations Between Top-level Categories ............................................................... 71 Table 19: Correlations Between KSP Subforums and Practices ................................................. 73 Table 12: Comparison of the Percentages of Posts Coded for Science & Engineering Practices in Table 13: Comparison of the Percentages of Posts Coded for Nature of Science Connections in x LIST OF FIGURES Figure 1: Screenshot of the Portal 2 community (from steamcommunity.com) ......................... 19 Figure 2: Screenshot of the KSP community (from forum.kerbalspaceprogram.com) ................ 20 Figure 3: Embedded screenshot of vehicle design (from forum.kerbalspaceprogram.com) ........ 38 Figure 4: Embedded image of Jupiter-C (from forum.kerbalspaceprogram.com) ....................... 41 Figure 5: Differences in total SEP and NOS across forums ....................................................... 42 Figure 6: Differences in individual SEP codes across forums .................................................... 43 Figure 7: Differences in depth and quality across forums .......................................................... 45 Figure 8: Differences in characterization codes across forums ................................................... 46 xi “Geordi, I have spent my whole life trying to figure out crazy ways of doing things. I'm telling ya, as one engineer to another: I can do this.” – Scotty (Star Trek) Introduction Roughly 30 years ago, famed astrophysicist Carl Sagan noted the discrepancy between our simultaneous reliance upon and ignorance of science and engineering: “We live in a society exquisitely dependent on science and technology, in which hardly anyone knows anything about science and technology. This is a clear prescription for disaster,” (1989, p. 10). More recently, astrophysicist Neil deGrasse Tyson has argued that being able to make sense of science can empower us to understand societal issues and protect us from disenfranchisement in a democratic society (Tyson, 2009). Jobs are increasingly technical and specialized, requiring critical thinking, reasoning, and problem-solving skills; our homes and workplaces demand the knowledge, use, and development of technological innovations; and many of the global challenges we face today will require scientific solutions. Products, services, and even political policy are marketed and shaped using what are often dubious “scientific” claims. In order to make informed decisions about everything from public policy to personal health, the ability to make sense of science holds relevance for scientists and nonscientists alike; and as these examples illustrate, science and engineering involve working within a broader community to build, explain, and come to consensus about theories and designs (Schwarz, Passmore, & Reiser, 2017). This focus on the authentic, real-world applications of science and engineering is reflected in the Next Generation Science Standards (NGSS). Science is a challenge for many Americans, as evidenced by poor performance on standardized tests for children and adolescents (National Assessment of Educational Progress, 2015; OECD, 2017) as well as large-scale surveys of adults (Funk & Goo, 2015). Although typical inquiry activities found in textbooks often fail to involve learners in sophisticated science 1 practices or to foster scientific habits of mind (Chinn & Malhotra, 2002; Singer et al., 2006), experiences that provide opportunities to engage in science and engineering practices can foster the kind of metacognition that supports scientific sensemaking (Clark et al., 2015; Lehrer, Schauble, & Lucas, 2008). There is now mounting evidence to suggest that videogames can be effective tools for science learning (Barab et al., 2009; Clark et al., 2015; Wouters et al., 2013), which makes intuitive sense, since videogames require players to make predictions, experiment, test strategies, and make revisions based on in-game feedback in order to succeed (Steinkuehler & Duncan, 2008; Tekinbas & Zimmerman, 2003). Learners are unlikely, however, to understand science and engineering solely by engaging in a particular activity; effective learning happens in a community context, moving back and forth between investigation and explanation as learners engage in shared practices and work towards consensus (National Research Council, 2011; Schwarz et al., 2017). In keeping with currently-accepted situated and sociocultural theories of learning, educational research may benefit from deeper examinations of the informal, self-initiated learning practices that occur in online spaces and interactive media (Barron, 2006; Duncan & Hayes, 2012). Videogame discussion forums appear to be a promising environment to support these aims. Online forums provide the reflective, social spaces where players can defend interpretations of feedback, debate differing explanations, and do so in a way that engages with a culture of public reasoning (NRC, 2011; Steinkuehler & Duncan, 2008; Zimmerman, 2014). Researchers have noted that the social scaffolding provided by online videogame communities can prompt players to engage with the more formal science underlying many commercial games (Martinez-Garza, Clark, & Nelson, 2013). To date, much of the research literature has conceptualized science narrowly—focusing 2 on conceptual knowledge and factual content rather than science practices and sensemaking (Martinez-Garza et al., 2013). This focus largely ignores the role of social interaction and the transdisciplinary connections that are crucial in understanding the nature of science. Similarly, the field has tended to conceptualize videogames just as narrowly—focusing on game content and gameplay, but often neglecting the online forums that are now a fundamental component of modern gaming. In order to be of value, future research must evolve to incorporate more current understandings of learning in social spaces. Since online videogame communities can serve as avenues for participants to evaluate and communicate information, solve problems collaboratively, and engage in argumentation—some of the very practices emphasized in NGSS—such spaces seem worthy of further study. To that end, this study explored whether and how videogame discussion forums may engage players in science and engineering practices. 3 Literature Review Science as a Way of Knowing Our current understanding of science is that it is not simply a body of knowledge, but a way of knowing or making sense of the world (Schwarz et al., 2017). This way of knowing has an assortment of constructs and labels associated with it, including scientific thinking, inquiry, literacy, habits of mind, practices, and sensemaking. Although there are distinctions amongst these different terms, in a general sense they reflect the view that science goes beyond the knowledge or ability that one has to also include what one does (Berland et al., 2016; Kuhn, 2011). A key metacognitive element of making sense of scientific phenomena involves the intentional coordination of theories and evidence. As Kuhn (2010) argued, “To seek knowledge is to acknowledge that one’s existing knowledge is incomplete, possibly incorrect—that there is something new to know,” (p. 3). Considering the persistence of misinformation in fields from education (e.g., the learning styles myth) to medicine (e.g., misinformation linking vaccines to autism) and outright falsehoods perpetuated in an era of “fake news”, the importance of theory- evidence coordination seems more relevant than ever. But Americans tend to fall short in national and international assessments related to science. In the 2015 National Assessment of Educational Progress—which was designed to measure not only content knowledge, but also the extent to which students are able to use scientific principles and scientific inquiry—only 22% of 12th grade students scored at or above the Proficient level. And according to the 2015 PISA, 15-year old students in the United States perform significantly below Canada, Estonia, Germany, and Hong Kong in measures of scientific literacy (OECD, 2017). Further, science-related fields suffer from a lack of diversity. Only 17.5% of bachelor’s degrees, 14% of master’s degrees, and 7% of doctoral degrees in 4 science were earned in the U.S. by black, Latinx, and indigenous students in 2008 (National Science Foundation, 2011). The implications of these figures are troubling. From an economic perspective, science is a necessary prerequisite for the engineering and technical jobs of the future. What’s more, an understanding of the nature of critical, scientific inquiry empowers individuals to make informed decisions in their daily lives and to fully participate as engaged citizens. Disparities in science achievement amongst today’s youth threaten to perpetuate existing patterns of privilege and marginalization in tomorrow’s adults (NRC, 2011), and both our classrooms and our society would benefit from a more diverse array of voices, ideas, and perspectives in science, engineering, and technology (Schwarz et al., 2017). If we seek to develop a diverse, gainfully employed, scientifically literate citizenry prepared to tackle scientific challenges and compete in a global economy, there is considerable work to be done. But encouraging scientific sensemaking isn't just about making students into scientists. Science and engineering practices, many educators argue, can allow people to solve problems that extend beyond the specific domain of science. This increased attention to the broader applications and value of a scientific mindset was evident in the measures used for the 2015 PISA, which were not focused exclusively on science content knowledge but instead were designed to test students’ abilities to identify appropriate questions, explain scientific phenomena, draw conclusions using evidence, and understand the nature of scientific inquiry (OECD, 2017). As defined by PISA, science involves content knowledge, procedural knowledge, and epistemic beliefs. In other words, science is not just about factual recall; it also requires an understanding of how science is done. 5 Science and Engineering Practices Of the many popular terms used in science education, I chose to examine science and engineering practices in this study. This focus is in keeping with the Framework for K-12 Education (NRC, 2012) and Next Generation Science Standards, which emphasize that engaging in science and engineering is not simply about a set of skills, content knowledge, or processes, but also requires an understanding of how scientific knowledge is constructed—using a variety of methods and approaches—and how scientific consensus is reached through discourse, argumentation, and consensus. Learners who use meaningful science practices are engaged in scientific sensemaking; in other words, they are actively constructing explanations to help them make sense of the world as opposed to simply replicating ‘correct’ outcomes defined by an authority (Reiser, Berland, & Kenyon, 2012). Science and engineering practices are both cognitive and social in nature, and take place in context. They move beyond prescriptive processes, skills, and accuracy to a deeper understanding of the reasons behind scientific ideas and the ways they are meaningfully applied in real-life contexts. As such, science practices are intended to engage learners in making sense of the world and in designing solutions for problems that are personally meaningful (Schwarz et al., 2017). Our current understanding of these practices is reflected in the NGSS, which focus on the transdisciplinary nature and real-life applications of scientific inquiry. They therefore emphasize practices as ways of building, evaluating, revising, and applying knowledge. The NGSS Science and Engineering Practices represent the different parts of the sensemaking process through which we can construct models, compare evidence, critique explanations, and engage in debate (Schwarz et al., 2017). Science practices reflect the types of activities scientists use to formulate questions about the natural world and investigate scientific phenomena. Engineering practices 6 provide an opportunity for learners to apply scientific knowledge in order to design solutions to authentic problems. The stronger emphasis on engineering practices reflected in NGSS (compared to previous standards) is intended to underscore how science, engineering, and technology interconnect to address meaningful, real-world challenges. Potentially, engaging in engineering practices offers pathways to strengthen scientific understanding and can also enhance interest in science by providing opportunities to apply it in context (NRC, 2012). Although science and engineering are certainly interrelated, some (e.g., Cunningham & Carlsen, 2014) have critiqued the ways that the NGSS standards have conflated science and engineering practices. For example, the standards neglect to address the important role that non- scientists (like the users of a product) play in engineering, as well as the variety of non-scientific criteria (e.g., cost, aesthetics) that are often used in the evaluation of engineering solutions. Additionally, another dimension of the standards called the disciplinary core ideas (DCIs) have been described as problematic with respect to engineering. For one, the DCIs for engineering are described using verbs (as opposed to the nouns used for the other DCIs) making them sound more like practices than ideas. Additionally, some of the Framework’s language indicates that engineering is an application of science, yet NGSS describes distinct engineering DCIs that suggest it has a unique disciplinary identity. The authors therefore argue that it makes more sense to focus on the practices rather than DCIs when exploring the distinguishing features of engineering (Cunningham & Carlsen, 2014). Since there is little consensus on the best way to teach science and engineering practices, and minimal evidence yet available to map their developmental trajectory (NRC, 2012), designing instruction and learning environments to support engagement in the practices has been a challenge for educators. But, as Barab & Hay (2001) suggest, effective science learning 7 experiences tend to share several key design characteristics: 1. Learners do domain-related practices to address domain-related dilemmas. 2. Scientific and technical knowledge/practice is situationally constructed and socially negotiated. 3. Learning is participatory, happening “at the elbows” of more knowledgeable others, including teachers, scientists, and peers. 4. Practices and outcomes are authentic to and owned by the learner and the community of practice, and are in response to real-world needs. 5. Participants become a part of (developing an identity as a member of) a community of practice. 6. Formal opportunity and support for reflection-in-action and reflection-on-action. (p. 77) Taken together, these design characteristics define participatory science learning environments. In contrast to the often prescriptive approaches taken in formal educational settings, research suggests that learning environments should establish a need for science practices: They should involve some level of uncertainty to spark curiosity, and the problems to be investigated should encourage learners to adopt shared goals (Manz, 2015; Schwarz et al., 2017). sustaining (Carlone, Haun-Frank, & Webb, 2011; Schwarz et al., 2017). This involves navigating Further, science learning should be socially and culturally meaningful, relevant, and the tension between the Western European-American, colonial approach to science that is often privileged in the U.S. education system—which, in turn, perpetuates normative views of what “doing science” should look like—and alternative pathways to meaning-making and knowledge production. For example, a number of indigenous scholars and educators have illustrated that 8 reciprocal, community-based storytelling often shapes scientific meaning-making; through narrative, indigenous communities have historically constructed explanations of natural relationships and engaged in theory-building (Cajete, 2000; Marin & Bang, 2015; Schwarz et al., 2017). Culturally sustaining pedagogies emphasize that authentic science is grounded in a community where it makes sense to build knowledge (Carlone et al., 2011). Videogame Affinity Spaces and Science and Engineering Practices As the characteristics of participatory science learning suggest, learning doesn’t only happen in school. It is also situated in context, can be enhanced by collaborative social interaction, and is often distributed across communities (Greeno, Collins, & Resnick, 1996; Lave & Wenger, 1991; Vygotsky, 1978). What’s more, it’s difficult for educators to design truly authentic, interest-driven learning that honors students’ varied lived experiences within the constraints of formal schooling. But by taking a broader, more ecological perspective on learning (Barron, 2006), we can explore the ways learners engage with science and engineering practices outside the bounds of formal educational structures. Indeed, mounting research shows that informal environments can qualitatively influence people's relationship with science, provide support for science learning, and offer opportunities to broaden participation in science (Bell & National Research Council, 2009). The concept of the affinity space was proposed by James Paul Gee (2004) in response to Lave & Wenger’s (1991) popular communities of practice framework. Both perspectives share a focus on social interactions and learning; however, Gee’s affinity space framework places a greater emphasis on the affordances and constraints of a particular environment (whereas Lave & Wenger’s work focuses primarily on the characteristics of the community) and views membership within those spaces as much more fluid (Duncan & Hayes, 2012; Pellicone & Ahn, 9 2014). Affinity spaces thus describe the dynamic physical, virtual, and hybrid environments that offer avenues for discussion, reflection, collaboration, and mentorship around a common endeavor. These cross-generational, fluid spaces allow participants to draw from “constellations” of literacy practices (Steinkuehler, 2007), online and offline information (Martin, 2012), and connections (Abrams & Gerber, 2014). Within these environments, users leave traces of both their own individual thought processes and the participatory, sociocultural efforts they make to construct knowledge from their experiences (Gerber, Abrams, Curwood, & Magnifico, 2016). It has been suggested that videogame affinity spaces may support informal science learning for individuals from a variety of backgrounds (National Science Foundation, 2015). As a 2011 NRC report on digital games and science learning noted, online videogame communities hold some of the greatest potential for facilitating the leap from tacit to explicit understanding in science. In keeping with the characteristics of participatory science learning, videogame players are more likely to engage in science practices (and certainly, to provide evidence of doing so) when games are complemented with opportunities for reflection (Morris et al., 2013). Similarly, informal science learning research suggests that informal science learning is most effective when it is interactive, provides multiple ways for learners to engage with concepts and practices, facilitates learning across settings, prompts learners to interpret and explore their learning experiences, and encourages learners to extend their learning over time (Bell & National Research Council, 2009). In a study of the game Physicus, for example, an educational game designed to promote understanding of physics concepts, Foster, Koehler, & Mishra (2006) found that players who made written and verbal comments that indicated reflection on the connections between their in- game actions and intentions for learning demonstrated significantly greater learning gains from 10 pre- to post-test when compared to non-players in a control condition. Even the videogame SPORE—a game that follows the development of a new species from a cellular organism to a technically-advanced civilization, and which has been criticized for inaccurate science content— was found to promote higher order thinking and led to higher learning gains for students who played it as part of an undergraduate course in evolution as compared to a non-gaming group (Poli et al., 2012). In the study, both the traditional and gaming groups employed readings, journal entries, and writing assignments. To foster critical thinking about the controversy, learners were also required to engage in an online debate about evolution and creationism and to build wikis about the subject. Emerging research on science in videogame affinity spaces suggests that the ways in which some players engage in the social construction of knowledge may be “parallel to what takes place in the scientific community” (Steinkuehler & Duncan, 2008, p. 531). For example, in their 2008 study of World of Warcraft (WoW) forums, Steinkuehler and Duncan found evidence of scientific discursive practices, systems and model-based reasoning, and tacit epistemologies— categories informed by the Benchmarks for Science Literacy (American Association for the Advancement of Science, 1993) and the National Science Education Standards (NRC, 1996). Scientific discursive practices codes related to argumentation, the use of scientific discourse, and supporting claims with evidence. Systems- and model-based reasoning codes included prediction and testing; analyzing feedback; and the use of mathematics, models, and systems. Tacit epistemologies codes reflected the stance one takes with regard to the nature of knowledge (e.g., that knowledge is subjective, objective, or shaped by evaluation and debate). Steinkuehler and Duncan found that 58% of the WoW forum posts showed evidence of systems-based reasoning, 10% displayed model-based reasoning, and 65% reflected an evaluative epistemology. These 11 findings suggest that online videogame forums do show evidence of science practices, and that many posts also reflect understandings of the nature of science. Importantly, many researchers have noted how the concept and dimensions of the affinity space has evolved considerably in recent years, heralding the need for new research that reconceptualizes the term (Duncan & Hayes, 2012; Lammers, Curwood, & Magnifico, 2012). Ten or more years ago, affinity spaces tended to be defined by a common, central portal like a single discussion forum; today, the relationships between portals is more symbiotic (Lammerset al., 2012). Since then, the boundaries between portals to contemporary affinity spaces have become even more porous and mutable, incorporate more multimodal content, are often integrated with social media, and are populated with user-generated content. Today, videogame affinity space portals include live streaming channels, wiki entries, cosplay and conventions, fan fiction, and online player forums—just to name a few. Since new portals continue to open, some portals close, and portal characteristics continue to evolve, research must continue to explore the practices that emerge in these spaces: “As new tools and spaces are developed and gain traction, the size, scope and practices of affinity spaces will change,” (Lammers et al., 2012, p. 55). Additionally, recent literature (e.g., Abrams & Lammers, 2017; Gee & Hayes, 2012; Pellicone & Ahn, 2014) problematizes affinity spaces, taking a more nuanced view that these spaces can be nurturing (cooperative, non-hierarchical, inclusive) and/or elitist (exclusive, hostile, abusive). Many online spaces, and especially those affiliated with gaming, can be particularly unwelcoming towards those from marginalized populations (see Salter & Blodgett, 2017). This is an especially important consideration if we hope to support all learners in a culturally-sustaining approach to science education. Additionally, the practices of science and engineering may be learned more effectively and authentically when a diversity of perspectives 12 are shared and negotiated. For example, Berland & Lee’s 2012 qualitative case study of group activities in science classrooms suggests that learners engaged with science and engineering practices are often better able to reach consensus when a variety of ideas are legitimized. Doing so creates a space in which dissenters can shift from persuasion to sensemaking and eventual consensus. Notably, in the classroom groups studied by Berland and Lee, these legitimization moves were not facilitated or enabled by a teacher, but were initiated by the learners themselves (Berland & Lee, 2012). The nature of online gaming affinity spaces are worth exploring in more detail, then, since this may hold implications for how likely individuals from diverse communities are to feel welcome enough to participate in any participatory science learning environment. From a methodological perspective, online affinity spaces offer opportunities to extract rich, naturalistic observational data; using computational techniques, such data can be collected efficiently on a very large scale (Landers, Brusso, Cavanaugh, & Collmus, 2016; Scott & Carrington, 2014). Online data is a resource increasingly being taken up by social science researchers, often using a method called web scraping. Put simply, web scraping involves the automated collection of online data. As Landers et al. (2016) note in their primer on this emerging method, “Web scraping, at its core, is about finding meaning in patterns of human behavior, the fundamental goal of all psychological research,” (p. 488). It allows researchers to get a glimpse of phenomena, or user reactions to phenomena, as they happen in a natural environment. 13 Context, Purpose, and Research Questions To expand our understanding of how science learning may manifest in videogame communities, I investigated science and engineering practices in the online forums for two popular videogames: Kerbal Space Program and Portal 2. Prior to conducting this research, I had already played both games and was familiar with both forums, having used them as resources while playing. The sequel to the popular game Portal (2007), Portal 2 (Valve, 2011) is an award- winning, top-selling first-person platformer/puzzle game that incorporates both single player and cooperative play modes. The gameplay of Portal 2 involves shooting a gun at two different surfaces to create portals that allow the main character (Chell) to progress through different game environments, to move objects, and avoid obstacles with the goal of escaping from the Aperture Science Enrichment Center facility. Successful navigation through puzzles requires players to determine the appropriate velocity and angle necessary to traverse from place to place. Since its initial release, additional content has been developed by Valve, including Puzzle Maker (2012), a level editor for players, and Teach With Portals (2012), an initiative designed to support lesson plans that use Portal 2 in classrooms. With an Entertainment Software Rating Board (ESRB) rating of E10+, the game is generally considered appropriate for ages 10 and up, and has been given a 5 star rating by Common Sense Media (a nonprofit dedicated to the evaluation of digital media). Portal 2 is playable on both desktop and console gaming systems. Kerbal Space Program, or KSP (2015), is a critically acclaimed, top-selling, multi-genre videogame created by the Mexico-based developer, Squad. Put simply, KSP is a space/engineering simulation game in which players construct, test, and use rockets, rovers, and other space vehicles. These crafts must be properly designed and built to safely transport a crew 14 of Kerbals (small, green aliens from the planet Kerbin) and to successfully complete in-game missions and challenges. The game is generally considered appropriate for ages 8 and up according to Common Sense Media, which rated the game at 4 out of 5 stars. KSP is playable on both desktop and console gaming systems. Portal 2 and KSP incorporate a stronger, more explicit emphasis on science and design than many other entertainment games. Both games allow players to act as designers; in KSP, players construct and test space vehicles, and in Portal 2 players can design and test their own levels. The online discussion forums associated with these games offer informal opportunities for participants to both engage in science and engineering practices and to make their sensemaking visible. As informal, interest-driven spaces affiliated with entertainment media, these internet- situated discussions may offer a glimpse into the ‘middle path’ described by Duncan & Hayes (2012), through which researchers may better understand the types of learning embedded within different kinds of affinity spaces. To examine whether and how science and engineering practices manifest in online videogame forums, I pursued the following research questions: ● RQ1: To what extent do discussions in the Portal 2 forums show evidence of Science and Engineering Practices and Nature of Science Connections? ● RQ2: To what extent do discussions in the Kerbal Space Program forums show evidence of Science and Engineering Practices and Nature of Science Connections? ● RQ3: How do the forums differ with respect to the extent to which users engage in Science and Engineering Practices and Nature of Science Connections? ● RQ4: How can the quality of conversations in these forums be characterized? 15 Research Design Method This was a descriptive, exploratory, mixed-methods study of two online affinity spaces. To answer my research questions, I conducted an in-depth case study using existing, public data from two online discussion forums bounded within a specific period of time. I began with a content analysis to interpret and code forum posts. My primary a priori protocol coding schemes were based on the NGSS Science and Engineering Practices and Nature of Science Connections. I used a secondary protocol coding scheme to characterize post content, and also rated the depth and quality of posts. I then performed a descriptive statistical analysis to identify patterns in the data. Positionality Statement My role in this study could be described as a peripheral participant observer. This study proceeds from the worldview that meaning is socially constructed and historically, culturally, and politically situated (Creswell, 2008) and that interpretation is, by nature, subjective (Merriam, 1998; Saldaña, 2016, Sipe and Ghiso, 2004). I therefore strove to maintain self-critical and introspective reflexivity by reflecting upon the ways my background and positionality influenced my decisions. I am a white, middle class, English-speaking, college-educated researcher who identifies as a woman. The disciplinary lens of educational psychology and educational technology has influenced the types of research questions I ask in this study as well as the methodology I used. At the same time, I attempted to use a trans-disciplinary approach that drew from (and ideally, may contribute to) other fields including games studies, science and technology studies, and teacher education. The resulting research project combines quantitative 16 statistics to describe patterns that appear to be common in the two forums with qualitative analysis that grounds the results within the context of these specific affinity spaces, offering preliminary insights about their implications. Definitions of Key Terms In this study I define user as an individual who is a member of an online discussion forum. Each user in a forum typically creates a username to identify themselves in the online space. Depending on a forum’s requirements for registration, users may choose or be required to create user profiles to provide others with more information about them (e.g., location, age, interests, and a photo or avatar). A post is a comment made by a user within a forum. An original post refers to the first post made in a topic (generally called a thread). A reply is a response made to a post. A quote is a quotation from a previous post—sometimes a post in its entirety, and sometimes a portion of a post. Some users choose to incorporate a quote to indicate the specific part of a previous post to which they are replying. A thread, therefore, can be seen as a conversation that starts with an original post followed by one or more replies on the same topic. Replies may or may not include quotes from previous posts. Data Sources and Collection For this study, I collected public data from online discussion forums for Portal 2 (2011) and Kerbal Space Program (2015) in March, 2018. The data source theory I used to support construct validity (Landers et al., 2016) rested on the premise that affinity spaces are a key part of modern gaming. These are dynamic and participatory environments in which players engage in debate, seek support, learn how to play or improve their game-related skills, share strategies, connect with other players, and discuss other aspects of the game. Although the majority of the 17 content consists of text, posts also often incorporate links and embedded content (some of it user- generated) that allow people to make connections between different online spaces. It is from this premise that the “digital traces” (Welser, Smith, Fisher & Gleave, 2008) left by forum posters provide evidence of the types of practices that occur within these spaces. Using web scraping techniques, posts from each forum were collected using the open-source programming language R. I collected the most recent 1,000 English-language posts from the General Discussion forums for Portal 2 on Steam (http://steamcommunity.com/app/620/discussions/). This encompassed an approximately 7 month time frame from Sept 11, 2017 through March 19, 2018. The Portal 2 Steam forums contained over 23,000 threads and over 250 active topics at the time of data collection. Steam is a popular content delivery system for videogames through which users can download, play, and in many cases modify thousands of different videogames. The Steam Workshop offers the Perpetual Testing Initiative, which allows players to develop and share user-created Portal 2 levels. Steam is also a gaming community composed of over 100 million users, many of whom choose to engage in online discussions about the games they play through discussion forums. The Steam community includes Rules and Guidelines and an Online Conduct Guide for forum posts, which encourage users to engage in civil discussion, to search for answers to questions before posting, to avoid abusive behavior, and to stay on topic. Posts that violate these rules can be removed by community moderators, and forum users can also report specific posts and replies to moderators for review by clicking on the flag icon that appears alongside each post. User profiles on Steam can be public, friends only, or private. Default user profile data includes the number of hours the user has played the game, workshop submissions, achievement 18 progress and badges, groups of which the user is a member, friends, and country/location. Users can also complete a “summary” field for their profile, which some users populate with a short bio or list of interests. Figure 1: Screenshot of the Portal 2 community (from steamcommunity.com) Similarly, I collected the most recent 1,000 English-language posts from the official Kerbal Space Program Forums (http://forum.kerbalspaceprogram.com/)1. This encompassed a two day period from March 6 through March 7, 2018. This community of over 180,000 users continues to be an active space for lively discussion, and contained nearly 160,000 total topics and over 3 million total posts at the time of data collection. Within the forums players discuss a variety of topics related to gameplay, including but not limited to strategies, troubleshooting, mission ideas, and space-related topics. Forum content and topics are highly categorized, and forum moderators often move content to more appropriate sub-forums to maintain well- organized, searchable resources for users. 1 KSP now also has an associated Steam Workshop and Discussion forum. 19 In terms of community values, the KSP forum strives to promote civil interactions between members, as evidenced by the forum’s Good Conduct Guide and Positive Forum Movement. These guides encourage users to stay on topic, to read forum rules and search for answers before posting, and to avoid personal attacks. This emphasis on a positive community is underscored by the presence of some of the game’s developers, as well as community managers and moderators, who participate in the forums and occasionally enforce community guidelines. In the KSP forum, some public user profile data is available, including: Content count (the number of posts and replies the user has made), rank (determined by a user’s post count), number of followers (other members of the community who ‘follow’ a user to be notified of new posts), community reputation (which is influenced by the number of likes a user’s posts have received and number of followers), and the date they joined the forum. Optionally, users can choose to include their interests, location, and contact information in their profile. Figure 2: Screenshot of the KSP community (from forum.kerbalspaceprogram.com) 20 The scraping process began with the first thread listed on the General Discussion page, and collected all of the posts on the initial page for the thread. If there was more than one page for a thread, the scraper worked through each page within the thread, pulling all posts until it reached the last post on the last page of the thread. This process was repeated for all threads in General Discussion Forum until the latest 1,000 posts were collected. The products of data collection were two spreadsheets of comma separated values, one for each forum. Each spreadsheet included the name of the user who made the post, a link to the user’s profile, the title of the post, and the URL of each post. Coding Scheme I analyzed and coded the data corpus using an a priori coding scheme based primarily on the eight Science and Engineering Practices (SEP) identified by NGSS. To better understand the ways in which posts may or may not have engaged with different aspects of the practices, and in light of recent critiques of the way NGSS tends to conflate science and engineering, each of my SEP coding categories was subdivided to classify posts as having a science and/or engineering focus—resulting in a total of 16 codes. The SEP codes, including descriptions and examples for each code, are described in Table 1. Codes based on the NGSS standards were conceptualized broadly—that is, the sensemaking and practices they describe were considered in a sense not limited to explicitly- stated scientific domains or concepts. This focus on scientific sensemaking and practices was in keeping with the approach taken previously by Steinkuehler & Duncan (2008) as well as recommendations by science education researchers (Schwarz et al., 2017). 21 Table 1: Science and Engineering Practices Codes Code Description Examples Asking Questions & Defining Problems Asking questions that lead to testable explanations of how the world works Developing & Using Models Using/building models as tools to represent ideas and explanations Planning & Carrying Out Investigations Analyzing & Interpreting Data Using Mathematics & Computational Thinking Constructing Explanations & Designing Solutions Engaging in Argument from Evidence Poses a scientific question, engages in prediction/hypotheses, identifies a design or engineering problem, a need for a particular design/engineering solution, or criteria for finding a successful solution Uses a model to represent a system/part of a system or phenomenon, to identify flaws, to refine or test a design, to communicate a design’s features to others (e.g., a prototype) or to test solutions to a problem References experiments intended to describe a phenomenon, test a theory about how the world works, or test a solution (or an improvement) to a design or engineering problem Presents data in order to communicate it to others (e.g., tabulating, graphing, statistics) and/or analyzes a design (e.g., by creating a model or prototype and collecting/analyzing data on how it performs) Planning/conducting systematic investigations and identifying data, variables, and parameters Identifying significant features / patterns in data, degree of certainty, and sources of error Using math and computation to represent variables and their relationships Uses computational thinking or mathematics to investigate or describe scientific phenomena or to solve a design or engineering challenge Science seeks explanations; engineering seeks solutions Includes a scientific explanation, claim, or theory about how a phenomenon works and/or a systematic, proposed solution to a design or engineering problem Using evidence to construct arguments in order to reach explanations/solutions Constructs an argument to explain a phenomenon, or in support of a particular design/engineering solution, using criteria, models/prototypes, measurements, or simulations to generate data to inform the solution 22 Table 1 (cont’d) Obtaining, Evaluating, & Communicating Information Communicating ideas and methods clearly Reflects critical consumption/evaluation of information, recognizes salient ideas, identifies sources of error and methodological flaws, etc., and/or refers to outside sources to evaluate the merit and validity of scientific claims, ideas, or methods Note: Codes based on NGSS Lead States (2013) Additionally, I looked for evidence of the eight Nature of Science Connections (NOS) as outlined by NGSS (see Table 2). As noted in the NGSS guidelines, “Science is the pursuit of explanations of the natural world, and technology and engineering are means of accommodating human needs, intellectual curiosity, and aspirations,” (NGSS Lead States, 2013, p. 430). This secondary scheme was intended to identify evidence that posts reflect an awareness of the nature of scientific explanations; the values of science and engineering; the ways that science and engineering are shaped by historical contexts, new discoveries, and human motivations; and other aspects of science not necessarily addressed by mere engagement in scientific activities. Table 2: Nature of Science Connections Codes Code Description Examples References to different modes or approaches to testing a scientific idea or engineering design Scientific Investigations Use a Variety of Methods Different methods and tools can be used in science; scientific discourse practices are used to ensure the accuracy of measurements, observations, and objectivity of findings; inquiry is characterized by a common set of values 23 Table 2 (cont’d) Science Knowledge Is Based on Empirical Evidence Scientists look for patterns and order when making observations about the world Scientific Knowledge Is Open to Revision in Light of New Evidence Science Models, Laws, Mechanisms, and Theories Explain Natural Phenomena Science Is a Way of Knowing Scientific Knowledge Assumes an Order and Consistency in Natural Systems Science Is a Human Endeavor Science Addresses Questions About the Natural and Material World Explanations can be probabilistic; knowledge/understandings may change based on new evidence and/or reinterpretation of existing evidence Facts must be repeatedly confirmed through observation and experiment, and validated by the scientific community Science as a body of knowledge as well as processes/practices; there are also other ways of knowing; science has a history that has evolved over time Science assumes consistent patterns in natural systems Individuals and teams from diverse backgrounds contribute to science; role of imagination, creativity, logic, persistence, ethics; reciprocal relationships with society and technology Science can help us understand natural and material phenomena (but not all questions can be answered by science) 24 References to the role of evidence and experimentation to test scientific ideas and engineering designs References to revisions in our understanding of science, or to the refinement / improvement of engineering designs References to scientific laws (e.g., laws of physics), what is generally accepted by the scientific community, and the importance of repeated testing to validate ideas References to other ways of knowing that play a role in science and engineering (e.g., philosophy) or the history of science Predictions based on established scientific patterns References to events that violate expected patterns References to the human aspects of science and engineering (e.g., personalities of tech moguls, politics, cost, intellectual property) References to differences between games and reality (e.g., comparing a videogame’s physics engine to “the real world”) Table 2 (cont’d) Note: Codes based on NGSS Lead States (2013) Individual posts served as the unit of analysis. I focused primarily on discourse in the form of written text as I interpreted the extent to which users engaged with science practices. I did consider multimodal content—in the form of embedded images, hyperlinks, and videos, for example—in my evaluation of the quality of forum discussions, but given the amount of data to be analyzed, it was not feasible to follow and fully evaluate every linked resource. Using the two spreadsheets of collected data, I used the URL for each post to view them using a web browser. For each of the 24 NGSS-based codes, I assigned each post a value: “1” if present, “0” if not present. As noted in the NGSS Lead States (2013), these themes are not mutually exclusive; in fact, many of them intentionally intersect. I therefore expected that some codes would overlap. Therefore, each post could potentially have from 0 to 24 of these codes applied during the coding process. To help characterize the conversations in these forums, I developed a code for Depth that reflected the level to which a post engaged with SEP and/or NOS. This was determined qualitatively and holistically to assess how deeply the post engaged with science and engineering practices. For example, posts engaging with SEP 1 might range from a simple question that has a declarative answer (which would indicate low Depth) to thoroughly defining and decomposing a problem (higher Depth). Similarly, posts coded for SEP 5 engaged with computational thinking and math practices to varying levels. In some cases they simply referred to relevant concepts rather than deeply analyzing or practicing them (in which case I would assign a lower rating for Depth). Based on this assessment, posts were assigned a rating from 0 to 2 to characterize their 25 overall depth. A rubric for the Depth code is described in Table 3. Table 3: Depth Rubric 0 1 2 Depth of discussion No engagement with practices Shallow engagement with practices Deep engagement with practices The Quality code reflected the length, interactivity, and multimodality of each post. This was determined qualitatively by considering three main factors: the length of the post (to characterize the sophistication of thought demonstrated in the individual post), the number of posts in the thread (to characterize the interactivity and quality of discussion of which it was a part), and the type of content included in the post (to characterize the variety and quality of media/modalities it incorporated). Based on this holistic assessment, the post was assigned a number from 0 to 2 to characterize its overall quality. A rubric for the Quality code is described in Table 4. Table 4: Quality Rubric Length 0 < 3 lines 1 3-10 lines 2 >10 lines Interactivity 1 post in thread 2-10 posts in thread >10 posts in thread Multimodality Text Includes text and hyperlinks Includes text and other multimodal content Admittedly, terms like depth, quality, and sophistication (as well as the use of grade level bands to categorize practices) are somewhat problematic. Even “less sophisticated” versions of the practices can be appropriate depending on the context in which they are used. I therefore used context to help inform the Depth and Quality ratings to avoid approaching this work from a 26 deficit perspective, considering the nature of what individuals were doing when engaging with the practices. To characterize the nature of these affinity spaces, I drew from the work of Gee & Hayes (2012) to code posts as Nurturing and/or Elitist or Neutral in nature. Posts coded as Nurturing tended to offer help and support to other members of the affinity space, solicit knowledge from members of the affinity space in a respectful way, make efforts to sustain a friendly affinity space (e.g., stands up for others, makes mention of community standards, agrees to disagree in a civil manner), and/or encourage collaboration by inviting others to play, write, or otherwise cooperate. Posts coded as Elitist tended to direct negatively charged language at others, use elitist language, and/or reinforce a perceived normative identity about who ‘belongs’ in the affinity space. Posts could contain both Nurturing and Elitist elements, and therefore were coded for both in some cases. Importantly, hierarchical elements could be present in both Nurturing and Elitist posts; that is, a post could be hierarchical and Nurturing (for example, a user with high levels of engineering expertise thoroughly and helpfully answering another user’s question) or hierarchical and Elitist (for example, belittling another community member for their lack of expertise). Posts whose content could not be classified as Nurturing or Elitist were coded as Neutral. Posts therefore could be coded as Nurturing only, Elitist only, Nurturing and Elitist, or Neutral only. Descriptions and examples of the characterization codes are described in Table 5. In sum, each post was coded for the presence or absence of SEP and/or NOS, assigned a rating for Depth and for Quality, and was characterized as Nurturing and/or Elitist or Neutral. 27 Table 5: Affinity Space Characterization Codes Code Nurturing Elitist Neutral Description Example Cooperative, non- hierarchical, inclusive Offers help and/or support to other members of the affinity space, solicits knowledge from members of the affinity space in a respectful way, makes efforts to create/sustain a friendly affinity space, and/or invites others to play, write, or otherwise cooperate Exclusive, hostile, abusive The post may direct negatively charged language at others, use elitist language, reinforce a perceived normative identity about who ‘belongs’ in the affinity space The post does not meet criteria for Nurturing or Elitist Does not contain text, or contains text that could not be interpreted Note: Codes based on Gee & Hayes (2012) In keeping with recommendations for qualitative coding, I conducted an iterative and rigorous analysis. I began with a preliminary review of forum posts to get a sense of the data, taking notes on initial impressions, interesting quotations, and questions they raised about my coding scheme. As I continued to cycle through the data—coding, questioning, taking notes, and recoding—I built a coding manual informed by qualitative coding guidelines that included, for each code: a description, decision guide, inclusion and exclusion criteria, typical examples, and when appropriate, atypical or “not quite” examples (Bernard & Ryan, 2010). I maintained field notes and analytic memos to encourage reflexivity and heuristic fluidity (as recommended by Saldaña, 2016) during the coding process. In the memos, I documented and reflected upon my key decision points and code choices; initial patterns I saw in the data; common trends that appeared in each of the forums; how I differed from and related to forum users and their 28 responses, how my coding might be influenced by my own positionality; and any emergent patterns or concepts (Creswell, 2008; Saldaña, 2016). Quality and Ethics of Online Data User profile data for online forums can be problematic, since many users choose to remain somewhat anonymous (e.g., in terms of gender, ethnicity, and age) and, even if revealed, such data cannot be verified. Neither forum requires or includes identifiable or demographic information. However, as a number of researchers have cautioned, the digital data points collected through internet research are abstractions of the identities and behaviors of individual human beings; therefore, as ethical researchers, our methods should still be informed by principles that respect privacy, agency, and the minimization of harm (Gerber et al., 2016; Markham & Buchanan, 2012; Markham, Tiidenberg, & Herman, 2018). In keeping with guiding principles and ethical questions for internet research (Markham & Buchanan, 2012), I examined the robots.txt file for each site. Put simply, this file contains a set of directives and/or filters using the Robots Exclusion Standard (RES) used to indicate which parts of a website should or should not be indexed by web crawlers. Both the Portal 2 Steam community forum and KSP forum, at the time of scraping, allowed for posts to be scraped using the crawler and method employed in this study. Although these forums were public, individuals who participate in online communities may not anticipate the aggregation and public use of their data by third parties (Markham & Buchanan, 2012). On the other hand, anonymizing an online utterance strips its author of credit for their words and ideas. To respect the voices of forum participants, I identified several users who tended to make exemplary posts on the Portal 2 and KSP forums. I then contacted five members of each forum privately through direct messaging to obtain consent. All of these 29 individuals granted permission for me to use quotations from their posts, attributed to their usernames, in this study and any resulting publications. Reliability and Validity The a priori coding I used was based on the comprehensively researched NGSS standards, supporting construct validity (Saldaña, 2016). The study’s ecological validity was strengthened by the use of naturalistic, existing public discussions. I employed researcher reflexivity through critical reflection on my own assumptions, worldview, and perceptions in my qualitative interpretations of these discussions. I also conducted extensive inter-rater reliability with an independent researcher. In keeping with this study’s focus on the broad and cross-cultural interpretation of science and engineering practices, this researcher brought experience and expertise that were different from my own in terms of cultural background, age, race/ethnicity, and education. I first provided her with the coding manual to be developed for this study, then met to discuss each code, engage in practice rounds of coding, and resolved questions about the coding scheme. I adjusted the coding manual to reflect our key decision points. I then randomly selected 200 posts (10% of the data corpus), which we coded independently. Agreement for most of the categories was above fair in this round, but I noted that the random sample of 200 posts we used contained very few examples of some low- incidence codes, which limited opportunities to discuss examples of certain codes and also had a noticeable effect on reliability measures. I therefore selected a purposeful sample with greater density to determine whether this would change inter-rater reliability, particularly with regard to low-incidence codes. This resulted in a sample of 24 posts that included at least a 10% representation of each code, which we coded independently. 30 Recognizing the limitations of using just one measure, I calculated two indices for reliability: Cohen’s kappa and percent agreement. Landis and Koch’s (1977) benchmarks for Cohen’s kappa designate values less than 0 as no agreement, 0–0.20 as slight agreement, 0.21– 0.40 as fair agreement, 0.41–0.60 as moderate agreement, 0.61–0.80 as substantial agreement, and 0.81–1 as almost perfect agreement. I also calculated percent agreement between myself and the other rater. Although using percent agreement alone as a reliability measure can be problematic, in this case I hoped that providing this additional measure would help to illustrate a more complete and transparent picture of the reliability of this coding scheme. This final round resulted in above moderate agreement for all top-level categories, as described in Table 6 (a table with the results across all categories can be found in Appendix A). Table 6: Final Round of Inter-rater Reliability for Top-level Categories SEP (all) Science Engineering NOS(all) Nurturing Neutral Elitist Depth Quality Portal 2 & KSP % agreement Cohen’s kappa 100% 79% 83% 88% 100% 96% 100% 88% 92% 1.00 0.58 0.52 0.60 1.00 0.78 1.00 0.78 0.84 Note: Depth & Quality were coded on a scale of 0-2. 31 Results RQ1: To what extent do discussions in the Portal 2 forums show evidence of Science and Engineering Practices and Nature of Science Connections? Overall, the total percentage of posts coded for SEP in the Portal 2 forum data was 33.7%. Most of the posts coded for SEP had an engineering focus. In terms of percentages, the most frequently-appearing codes were SEP 1: Asking Questions and Defining Problems at 21.6%, SEP 3: Planning and Conducting Investigations at 10.0%, and SEP 6: Constructing Explanations and Designing Solutions at 13.9%. There was no evidence at all of SEP 2: Designing and Using Models (Science focus) and little evidence of science-focused practices in general (2.1%). For the total percentages of posts coded for each SEP, see Table 7. Table 7: Percentages of Posts Coded for Science and Engineering Practices in Portal 2 Forums SEP Code Total SEP SEP1: Questions & Problems SEP2: Models SEP3: Investigations SEP4: Data SEP5: Computational Thinking SEP6: Explanations & Solutions SEP7: Argument SEP8: Information Portal 2 Sci 2.1% 1.1% 0.0% 0.7% 0.2% 0.1% 0.7% 0.0% 0.1% Eng 31.8% 20.5% 0.5% 9.4% 7.5% 5.6% 13.3% 0.8% 4.2% Combined 33.7% 21.6% 0.5% 10.0% 7.7% 5.7% 13.9% 0.8% 4.3% Only a small percentage of posts (2.7%) were coded for evidence of NOS. Only one code, NOS 7: Science is a Human Endeavor, appeared in more than one percent of the posts, with even less evidence of any of the remaining NOS codes and no evidence of NOS 4: Science 32 Models, Laws, Mechanisms, and Theories Explain Natural Phenomena, NOS 5: Science Is a Way of Knowing, or NOS 8: Science Addresses Questions About the Natural and Material World. For the percentage of posts coded for each category, see Table 8. Table 8: Percentages of Posts Coded for Nature of Science Connections in Portal 2 Forums NOS Code Total NOS NOS1: Variety of Methods NOS2: Empirical Evidence NOS3: Revision NOS4: Laws & Theories NOS5: Way of Knowing NOS6: Order & Consistency NOS7: Human Endeavor NOS8: Natural World Portal 2 2.70% 0.40% 0.20% 0.10% 0.00% 0.00% 0.50% 1.50% 0.00% Many posts in the Portal 2 community reflect community members working together to troubleshoot problems with installation and publishing of the Portal 2 chambers they have created, or troubleshooting other technical issues with modification tools. The following exchange between two members of the forum illustrates some of the ways discussions engaged with science and engineering practices, as well as the ways in which I considered context in interpreting and coding posts from the forums. Community member Testsubject276 began by requesting help with the installation of a new version of BEEmod (Ben and Carl’s Extended Editor), an open-source application that can be used to modify Portal 2’s Puzzlemaker. Testsubject276 was experiencing a number of design challenges and glitches while trying to construct and run a particular Portal 2 chamber. In 33 response, Goldenschmidt replied: Make a folder for BeeMod 2.4.0 alone (That is, unless you already have a folder for BeeMod 2.4.0 alone (If the previous version of BeeMod 2.4.0 already exist in that folder, either delete it, or move it to another folder, or make a different folder for the new one) Download the .zip folder of the BeeMod 2.4.0, and extract it into the folder. When you've done that, do the same for the package folder. (Make sure that its filepath is not [Foldername]\Package\package, but just [Foldername]\package ) When you've done that, open the Bin folder, and create a shortcut of Bee2.exe, and place it wherever you want it to be, then doubleclick it, or hit enter, while it's selected. When you're prompted to, find the executable for Portal 2. (Found in Steam\Steamapps\common\Portal 2) and open it. Now Bee2 will ask what you'll name this game (In this case, Portal 2), then click Okay. Here you can edit your palette, change style, etc. Now just hit the button "Export to [GameName]" Portal 2 should now launch, after compiling it into the game. This response was part of a larger discussion that focused on solving an engineering/design problem (constructing a chamber with the use of the BEEmod tool), context that I considered in interpreting and coding its content. In their response, Goldenschmidt engages with the computational thinking concept of algorithms (SEP 5: Mathematics and Computational Thinking) by providing a concrete, ordered series of logical instructions to support a solution to a computing problem (SEP 6: Constructing Explanations and Designing Solutions). These instructions provide some amount of context to address the complexity of the different file structures that may or may not be present (“If the previous version of BeeMod 2.4.0 already exist 34 in that folder, either delete it, or move it to another folder, or make a different folder for the new one”) while still maintaining a level of simplicity in order to present ordered steps efficiently and concisely. Given its level of engagement with SEP, I assigned it a Depth rating of 2. The reply does not contain multimodal content, but is relatively lengthy and part of an extremely lengthy thread, so it was rated 2 for Quality. This was a Nurturing post that attempted to assist another member of the community and was free from elitist sentiments. Unfortunately, even with this assistance the original poster continued to experience problems and glitches during the design process that rendered their chamber unplayable. As the discussion delved into an investigation of the design problem (SEP 3: Planning and Carrying Out Investigations), Goldenschmidt asked additional probing questions such as “Did you try to create a completely new map, and add some of the Non-Beemod items that made these glitches?” and “What happens, if you try to create a completely new map? Try only using basic PeTI items. See if the problem persist”. These questions reflect an effort to gain more insight into the problem by testing different scenarios and isolating particular variables. Eventually, Testsubject276 arrived at a workaround solution: good news though, i found out how to suppress it, though annoying, i seems to show up after a number of actions, if i change one thing then run the chamber over and over after each change, it seems to stay stable for awhile, I've been walking up to the door and saving every time it works normally and leaving the editor when it comes back to restart, annoying, but so far usable. This response touches upon a few practices: SEP 3: Planning and Carrying Out Investigations (“if I change one thing then run the chamber over and over after each change”), SEP 4: Analyzing and Interpreting Data (in noticing the behavior of the glitch and when it does 35 or does not appear), and SEP 6: Constructing Explanations and Designing Solutions (saving and exiting the editor to get the chamber to function normally). Since this individual was trying to create a chamber, and the issues being tested and explored all had the overall goal of design in mind, I coded the practices for an engineering focus. I coded this post as Nurturing since it offers to share the workaround with the wider community, and rated it as 1 for both Depth and Quality given that its connections with science practices are less explicit and the post is relatively brief but part of a lengthy discussion thread. RQ2: To what extent do discussions in the Kerbal Space Program forums show evidence of Science and Engineering Practices and Nature of Science Connections? In the KSP forums, the total percentage of posts coded for evidence of SEP was 66.3%. Engineering-focused SEP codes were more common, and appeared more than twice as often as science-focused codes. The most frequently-appearing codes were SEP 1: Asking Questions and Defining Problems (36.6%), SEP 4: Analyzing and Interpreting Data (18.3%), SEP 6: Constructing Explanations and Designing Solutions (28.1%), and SEP 8: Obtaining, Evaluating, and Communicating Information (22.5%). For the percentage of posts coded for each category, see Table 9. Table 9: Percentages of Posts Coded for Science and Engineering Practices in KSP Forums SEP Code Total SEP SEP1: Questions & Problems SEP2: Models SEP3: Investigations KSP Eng 50.5% 28.4% 12.4% 12.3% Combined 66.3% 36.6% 12.4% 14.1% Sci 18.3% 8.8% 0.0% 2.5% 36 Table 9 (cont’d) SEP4: Data SEP5: Computational Thinking SEP6: Explanations & Solutions SEP7: Argument SEP8: Information 2.7% 1.1% 6.6% 1.7% 6.4% 15.9% 6.0% 22.2% 3.6% 16.8% 18.3% 6.9% 28.1% 4.9% 22.5% The total percentage of posts in the KSP forums coded for NOS was 12.9%. The most frequently-appearing NOS code in the KSP forums was NOS 7: Science is a Human Endeavor (6.6%). Posts engaging with NOS 7 tended to reflect ideas about the human aspects of science and engineering such as the personalities of tech moguls (e.g., Elon Musk), financial concerns, and cautions for impatient forum members to be aware that mod/game developers are human (e.g., that mistakes/bugs will occur, that developers have other commitments and cannot be working on updates 24/7). There was some evidence of the remaining codes, but they appeared infrequently (less than two percent). The percentages for each of the NOS codes in KSP can be found in Table 10. Table 10: Percentages of Posts Coded for Nature of Science Connections in KSP Forums NOS Code Total NOS KSP 12.9% NOS1: Variety of Methods NOS2: Empirical Evidence NOS3: Revision NOS4: Laws & Theories NOS5: Way of Knowing NOS6: Order & Consistency 0.2% 1.4% 1.3% 0.9% 0.7% 0.8% 37 Table 10 (cont’d) NOS7: Human Endeavor NOS8: Natural World 6.6% 1.0% By way of example, this post by KSP forum user Rizzotherat was a reply to an original post (OP) describing design changes intended to address problems with aerobraking large spacecraft. Briefly, aerobraking is a maneuver intended to reduce velocity of a craft by flying through the atmosphere at a low point of its orbit, thereby leveraging atmospheric drag in order to slow the craft. Definitions for terms and acronyms referenced in the post are included in brackets. Rather than a drag plate, I've been experimenting with a wing design to keep me facing the right way during Eve entry. Figure 3: Embedded screenshot of vehicle design (from forum.kerbalspaceprogram.com) This one didn't actually work as the moment was still too small, but I don't have a pic of my latest version which does work, is longer, and has an ISRU [In Situ Resource 38 Utilization: allows the craft to collect and use resources encountered during space exploration] and drills just behind the heat shield, which coupled with not a lot of fuel on board keeps the centre of mass nice and low. It's scary just how much force you need to keep straight braking in to Eve from a low orbit. Obviously this isn't going to give me as much drag as the OP's [Original Poster’s] design, but it's a lot easier to launch from Kerbin as the box wing segment is attached via a docking port so be launched low down on a separate launcher and docked later. The rest of the ship can then hit LKO [Low Kerbin orbit] as an SSTO [single-stage-to-orbit: a craft that can achieve orbit without requiring multiple stages]. In this reply, the Rizzotherat engaged with a number of practices with an engineering and design focus. To begin, they engaged with SEP 6: Constructing Explanations and Designing Solutions by describing an alternative solution—a different wing design—to both improve aerobraking and stabilize the direction of the craft: “Rather than a drag plate, I've been experimenting with a wing design to keep me facing the right way during Eve entry”. They explained how this design helps to address the problems that have been defined, such as the amount of force needed for aerobraking (SEP 1: Asking Questions and Defining Problems). Additionally, the user engaged with SEP 3: Planning and Conducting Investigations, describing the results of experiments (including an embedded image of a model that was unsuccessful in testing, and references to its features, thereby touching on SEP 2: Developing and Using Models). Throughout, the user showed evidence of engagement with SEP 4: Analyzing and Interpreting Data by analyzing performance data and the influence of variables such as the amount of fuel on board: “my latest version which does work, is longer, and has an ISRU and drills just behind the heat shield, which coupled with not a lot of fuel on board keeps the centre 39 of mass nice and low”. Finally, the post engaged with SEP 8: Obtaining, Evaluating, and Communicating Information by both illustrating points with an embedded image and comparing affordances and constraints of this design to those described in the original post: “this isn't going to give me as much drag as the OP's design, but it's a lot easier to launch from Kerbin”. This post contributed knowledge and ideas to the KSP community and was free from elitist language; I therefore coded it as Nurturing. Given the number of practices touched upon by this post as well as the depth of that engagement, the post’s length, its participation in a rather lengthy thread, and its incorporation of multimodal elements, I rated its Depth and Quality as 2. Another example from the KSP forums was posted by a member of the community named NSEP in response to a question about the design of a particular rocket (specifically, the original post asked: “Why are R-7 boosters inset?”). For this example, I have created a table to align excerpts from the post, the codes that were assigned, and explanations of how I coded them (see Table 11). Although this doesn’t fully account for the overlapping nature of these codes, it may serve to better illustrate my coding process. Table 11: Coding Example Post R-7 was made during the dawn of spaceflight, when orbital rockets are were completely new. There was no other orbital rockets to compare it to, and that ends up in pretty odd designs. Just look at any early rocket design. The Juno I for example was a ballistic missle and had multiple solid rocket upper stages. Its like fireworks strapped together. Code SEP 8 (Engineering) Rationale Draws from the history of spaceflight and rocket design. SEP 2 (Engineering) Incorporates and refers to a model (an embedded blueprint of the Jupiter-C rocket) to compare the R-7 with an example of “pretty odd” early rocket design. 40 Table 11 (cont’d) Figure 4: Embedded image of Jupiter-C (from forum.kerbalspaceprogram.com) Eventually the most efficient and effective rockets stayed, and that’s why most rockets are just cilinders with cones. SEP 6 (Engineering) NOS 3 But the R-7, just kind of sticked around, because it worked and there is no need for a replacement. SEP 7 (Engineering) Notes that as rocket design advanced, some experimental design elements were modified while efficient rocket designs were maintained. Reflects on the ways engineering advances as new evidence emerges (regarding the efficiency and effectiveness of new rocket designs). The post uses these practices to construct an argument for why some aspects of rocket design tended to remain stable for many years. This post engages beyond a surface level with a number of practices, so I rated it as a 2 for Depth. The post itself is approximately 5 lines long, but is part of a lengthy thread and includes embedded multimodal content. I therefore rated it as 2 for Quality. Because it contributes answers to another forum member’s question and is free from elitist discourse, I coded it as Nurturing. 41 RQ3: How do the forums differ with respect to the extent to which users engage in Science and Engineering Practices and Nature of Science Connections? The posts in the KSP forums showed significantly more evidence than the Portal 2 posts of SEP overall (66.3% compared to 33.7%) as well as with regard to each specific practice. Posts in both forums tended to show more evidence of engineering-focused practices than science- focused practices, but KSP had significantly more posts with a science focus (18.3% compared to only 2.1%). KSP also had more posts coded for NOS overall (12.9% compared to 2.7%). A visualization of these differences is illustrated in Figure 5. Figure 5: Differences in total SEP and NOS across forums KSP also had more total posts coded for all SEP categories. These differences were statistically significant (p < .05) for all categories except SEP 2 (Science): Developing and Using Models (in fact, no posts were coded for the science focus of SEP 2 in either forum) and SEP 5 (Engineering): Using Mathematics and Computational Thinking. A comparison of the 42 percentages of posts coded for specific SEPs in each forum is illustrated in Figure 6, while exact percentages and significant differences across forums can be found in Table 12. Figure 6: Differences in individual SEP codes across forums Table 12: Comparison of the Percentages of Posts Coded for Science & Engineering Practices in Each Forum SEP Code Total SEP Portal 2 Sci 2.1%** KSP Sci 18.3%** Portal 2 Eng 31.8%** KSP Eng 50.5%** KSP Portal 2 Combined Combined 33.7%** 66.3%** 1.1%** 0.0% 0.7%* 0.2%** 0.1%* 0.7%** 0.0%** 0.1%** SEP1 SEP2 SEP3 SEP4 SEP5 SEP6 SEP7 SEP8 * Statistically significant difference at the p < .05 level ** Statistically significant difference at the p < .001 level 20.5%** 0.5%** 9.4%* 7.5%** 5.6% 13.3%** 0.8%** 4.2%** 8.8%** 0.0% 2.5%* 2.7%** 1.1%* 6.6%** 1.7%** 6.4%** 28.4%** 12.4%** 12.3%* 15.9%** 6.0% 22.2%** 3.6%** 16.8%** 21.6%** 0.5%** 10.0%* 7.7%** 5.7% 13.9%** 0.8%** 4.3%** 36.6%** 12.4%** 14.1%* 18.3%** 6.9% 28.1%** 4.9%** 22.5%** 43 KSP had more posts coded across all NOS categories except NOS 1: Scientific Investigations Use a Variety of Methods (in which Portal 2 had more, although this difference was not significant). The differences across forums were statistically significant (p < .05) for all categories except NOS 1: Scientific Investigations Use a Variety of Methods and NOS 6: Scientific Knowledge Assumes an Order and Consistency in Natural Systems. NOS 7: Science is a Human Endeavor was the most common of the NOS practices to appear (6.6%). A comparison of the percentages of posts that were coded for NOS in each forum can be found in Table 13. Table 13: Comparison of the Percentages of Posts Coded for Nature of Science Connections in Each Forum NOS Code Total NOS NOS1 NOS2 NOS3 NOS4 NOS5 NOS6 NOS7 NOS8 * Statistically significant difference at the p < .05 level ** Statistically significant difference at the p < .001 level Portal 2 2.7%** 0.4% 0.2%* 0.1%* 0.0%* 0.0%* 0.5% 1.5%** 0.0%* KSP 12.9%** 0.2% 1.4%* 1.3%* 0.9%* 0.7%* 0.8% 6.6%** 1.0%* RQ4: How can the quality of conversations in these forums be characterized? KSP posts tended to be higher in both Quality and Depth. On the 2 point scale, the average Depth of KSP posts was 0.53 compared to 0.33 in Portal 2, and the average Quality of KSP posts was 1.01 compared to 0.71 in Portal 2. These differences are illustrated in Figure 7. 44 Figure 7: Differences in depth and quality across forums Unlike the dichotomous values for other categories, values for the Quality and Depth categories were on a scale of 0-2 (ordinal, and not a proportion). I therefore conducted an independent samples t-test to determine whether differences between forums were significant, and found that differences for both the Quality and Depth categories were significant at the p < .001 level. The standard deviation was similar for both forums and both categories (ranging from 0.54 to 0.58), with the exception of a slightly larger standard deviation for Depth in the KSP forums (0.73). The results for the Depth and Quality are described in Table 14. Table 14: Comparison of Average Ratings for Depth and Quality Categories Portal 2 KSP Mean Rating 0.33** 0.71** SD 0.58 0.56 Mean Rating 0.53** 1.02** SD 0.73 0.54 Depth Quality Note: Depth & Quality were coded on a scale of 0-2. 45 Table 14 (cont’d) * Statistically significant difference at the p < .05 level ** Statistically significant difference at the p < .001 level To further characterize and differentiate the two affinity spaces, I also compared the percentage of posts in each forum that were coded as Nurturing, Neutral, and Elitist. Both spaces could be similarly characterized as Nurturing, since the majority of the Portal 2 (96%) and KSP (97%) posts were coded as such. Differences in the percentages of posts coded for each characterization category are illustrated in Figure 8. Figure 8: Differences in characterization codes across forums I conducted a z-test of proportions to determine whether any differences between forums were significant for these categories (see Table 15). The Portal 2 forums contained significantly more posts coded as Neutral (3.1%) than the KSP forums (0.7%). This difference was significant at the p < .001 level. Additionally, more posts were coded as Elitist for KSP (4.1%) as compared to Portal 2 (1.2%). This difference was significant at the p < .001 level. 46 Table 15: Comparison of Percentages for Characterization Codes Nurturing Neutral Elitist Portal 2 96% 3%** 1%** KSP 97% 1%** 4%** * Statistically significant difference at the p < .05 level ** Statistically significant difference at the p < .001 level Posts coded as Elitist were not significantly correlated with any other category. The Depth and Quality codes were positively correlated with each other (using Spearman's rho for ordinal variables at a significance level of p < .001). Neutral posts were negatively correlated with SEP codes (p < .001) as well as Depth and Quality (at a significance level of p < .001). Nurturing posts were positively correlated with Depth and Quality (p < .001). A correlation table for all of the codes can be found in Appendix B. The KSP forums were, and continue to be, well-organized, active affinity spaces with strong community norms. The forums are structured by purpose, including a subforum for new players to get acquainted with the game and with other players; a subforum that explores different parts of the KSP network; and a subforum called The Lounge, dedicated to off-topic (non-KSP) discussions. Not only do moderators actively monitor and review content (taking it upon themselves, for example, to move posts to more relevant threads when deemed necessary) but the community itself tends to self-police where inappropriate or impolite conversation is concerned. Additionally, member profiles include a Community Reputation badge. New members are initially assigned a Neutral reputation and must have their first five posts to the forums approved by moderators (and posts made to off-topic forums like The Lounge do not count toward that total). Once these first five posts have been approved, the user advances to a 47 Good reputation and can post without approval as well as send direct messages to other forum members. Members of the forum can “like” posts by clicking a thumbs-up icon, or, conversely, can report posts that violate community Guidelines. Reports are reviewed by moderators, who can issue Warning Points accordingly (those points appear on a member’s profile and can negatively affect their reputation). Notably, The Lounge is one of the most active spaces in the forums; in this area, forum visitors discuss favorite foods, complain about things, and play games (such as constructing collaborative rhymes, engaging in “picture wars,” and inventing silly reasons to “ban the user above you”). A notable tendency on the KSP forums throughout the duration of this study was the emergence of discussions sparked by events that occurred during the specific time frame which data was captured. Some of these events were directly related to the game—for example, a new end user license agreement (EULA) for KSP was released—while others had to do with current events in spaceflight—for example, China’s Tiangong-1 space station re-entry and the launch of SpaceX’s Falcon Heavy. An example of this were replies to the question “Will StratoLaunch fly?”. In this thread, users explored the feasibility of the American space transportation project started by Microsoft co-founder Paul Allen, which began ground testing in spring 2017. KSP community member Wumpus posted a reply to this thread to address both the expense and feasibility of single-stage vs. two-stage shuttles (I have added explanations for some references in brackets to further contextualize the content of the post): A SSTO [single-stage-to-orbit] shuttle isn't just expensive, it may well be on the wrong side of physical limits. If you don't drop the fuel tank (and stop being "fully reusable"), it almost *certainly* is on the wrong side. Staging is simply required for the Isp used in currently available fuels, and there is no reason to believe Allen has anything up his 48 sleeve beyond a whole lot of money. A two stage shuttle (presumably both landing like planes/gliders) still has problems, but might be considered easier/cooler/less "me-to" than a vertical landing. I don't know Stratolaunch's cost/schedule, but building a rocket for it would presumably be a project of similar scope. I'd expect a rocket sized for stratolaunch would be a more complex project, but presumably Stratolaunch already knows from working with Scaler (who built both the White Knight and Space Ship 1). BFR [a reusable rocket system developed by SpaceX] is well along the way in design. Presumably New Glen [an orbital launch vehicle being developed by Blue Origin] is also coming along. I'd assume at least one of them will be flying while Stratolaunch is grounded/reduced to ferrying Pegasus [an air- launched rocket developed by Orbital Sciences Corporation] regardless of how fast Allen spends money. There's also the issue that the Electron [a two-stage, expendable launch vehicle developed by Rocket Lab] can already send more payload to orbit than any SSTO light enough for Stratolaunch to carry. If any work starts on the design, that "nearly SSTO" will drop down to "just getting out into space." This post illustrates a common trend on the KSP forums to not only discuss science and technology in a wider, real-world sense that extends beyond the game—indeed, there is an entire subforum devoted to “Science & Spaceflight”—but to do so in a way that reflects awareness of past, current, and future space-related projects; news and popular media; design differences across companies and projects; and the very human concerns associated with such projects (e.g., cost, relationships between corporations, scope creep, tech celebrities). To examine the KSP data in greater detail, I identified the subforums to which each of the 49 collected posts were made. In total, the collected data included posts made to 16 of the 21 subforums in existence at the time. Of the 16 subforums, I identified those representing 5% or more of the data; the resulting 5 subforums reflected the most active areas of the community at the time of data collection. This included KSP Discussion (comprising 42% of the KSP posts), Add-on Releases (18%), Science & Spaceflight (10%), The Lounge (9%), and Technical Support (5%). KSP Discussion was devoted to general discussion of the game; Add-on Releases to discussion and announcements of new add-ons for the game; Science & Spaceflight to discussions of space exploration and science; The Lounge to off-topic chat not related to KSP; and Technical Support to reports of problems, bugs, and issues with the game (note: this included posts from modded and unmodded PC installations as well as console installations of KSP). Since a more organized structure is one of the affordances of the KSP community, I was curious whether there appeared to be significant relationships between these subforums and any evidence of Science and Engineering Practices (AnySEP) and/or Nature of Science Connections (AnyNOS). An examination of Pearson’s correlations revealed that the Science & Spaceflight subforum was positively correlated with AnySEP and AnyNOS (p < .001). The Lounge was negatively correlated with AnySEP (p < .001) and AnyNOS (p < .05). The Technical Support subforum was positively correlated with AnySEP (p < .001), and the KSP Discussion subforum was negatively correlated with AnyNOS (p < .01). A correlation matrix describing these relationships can be found in Appendix C. The Portal 2 forums differ from the KSP forums in a number of ways. These spaces are less active, less structured, and less regulated than the KSP forums, without designated moderators or specific subforums to organize content. Community members can choose to reveal 50 a good deal about themselves and their Steam play or keep their user profiles private. Some elements of public profiles include Level (as defined by experience), games owned, friends, achievement badges, group memberships, reviews posted, and guides and workshop items they created. While KSP forum profiles contain the more holistic “Community Reputation” assessment, Portal 2 user profiles contain a variety of different metrics that could indicate how much they contribute to the community (in the form of guides, reviews, and workshop items, for example) and how far they have advanced in the game (hours played, achievements earned, etc.). Like the KSP forums, members of the Portal 2 Steam community can also report posts for inappropriate content. Some of the common trends, or norms, in the Portal 2 forums include a variety of threads dedicated to sharing/advertising user-generated maps, forum users looking to connect with others interested in cooperative play, and threads in which players offer to record and share video recordings of “speedruns” of other users’ maps (a speedrun is a player’s attempt to complete the level as quickly as possible, and can also be considered a form of playtesting since it often includes feedback and suggested improvements for the map creator). Content in these threads tends to follow a particular format; the original post often starts a thread with a title such as “Advertise your maps here,” and is followed by many (often 50 - 100) very brief replies in which members share embedded links to the playable chambers they have created within the Portal 2 Steam Workshop. Requests for cooperative play tend to originate with a post entitled “Looking for co-op”. Unlike map-sharing threads, these discussions tend to be very short (often with few or no replies). The Portal 2 forums contained significantly more posts that were coded as Neutral (though still a small percentage, at 3.10%). The Neutral code was assigned to posts that were 51 difficult or impossible to classify as Nurturing or Elitist. Typical posts of this type contained one word (e.g., “test”), a string of letters or numbers that could not be interpreted, or no content at all. This kind of content may have been more prevalent in this space because it was less actively moderated. Also, since these forums are part of the wider Steam Community forums (of which thousands of different games and associated communities are a part), it seems possible that posts from casual visitors who are less actively involved in the specific game being discussed may be more common. In terms of user activity, data from the Portal 2 forums originated from a smaller number of users (222) with higher average post counts (4.51 posts per user) represented in the 1000-post sample. The KSP sample represented posts from 366 users, with an average post count of 2.74. The post count for the most active member of the Portal 2 community was 97, compared to just 21 posts made by the most active user in the data I collected from the KSP forums. Thus, the Portal 2 data comprises a smaller number of active members, but those individuals tended to post frequently. The KSP data, in turn, reflects a larger number of moderately active members. The median number of posts for users in each forum was 2 posts per user (see Table 16). Table 16: User Post Count Details Within Each Forum’s 1000-post Sample Number of users Post count for most active user Average post count Median post count Portal 2 222 97 4.51 2 KSP 366 21 2.74 2 It’s worth taking a closer look at the user count data to take context into account. For example, the most active user in the Portal 2 forums posted 97 times during the 7 month time 52 frame bounding the 1000 posts collected for this study. Nearly all of that user’s posts were replies to “Advertise your maps here” threads, meaning they were very short posts with little to no text, often only containing links to chambers that the poster had recently created. The most active member of the KSP forums made 21 posts during the two days over which the 1000 posts collected from that forum were made. Similarly, nearly all of that user’s posts were brief responses to threads within KSP’s informal, off-topic sub-forum, The Lounge (the majority of these posts were contributions to threads like “Ban the user above you”, a game in which users invent fanciful and humorous reasons why the previous poster might be banned from the forums, or “One-word story”, a collaborative story to which each participant contributes one word). On the one hand, the user count data outlined above indicates a difference between the forums: There were fewer users making more posts about Portal 2, and more users making fewer posts about KSP. On the other hand, a closer qualitative examination suggests a similarity between the forums; that is, in both spaces, some very active threads (like those dedicated to map sharing in the Portal 2 forums and threads within The Lounge subforum in the KSP community) tended to have the least amount of text per post and little if any engagement with science practices. 53 Synthesis of Findings Discussion I found evidence that science and engineering practices emerged “naturally” in these informal spaces. Overall, roughly half of the posts I collected for this study showed evidence of engagement with science and engineering practices. That engagement tended to focus much more on engineering than on science. It makes intuitive sense that this would be an affordance taken in these spaces, reflecting the emphasis both games place on design (constructing new chambers in Portal 2, and building spacecraft in KSP). There was far less evidence of engagement with science-focused practices, and even less engagement with the Nature of Science. Both spaces could be characterized as nurturing. The most common practices in the forums were Asking Questions and Defining Problems, Constructing Explanations and Designing Solutions, and Obtaining, Communicating, and Evaluating Information. The frequency of SEP 1: Asking Questions and Defining Problems reflects an affordance taken: Forums are frequently used as an outlet for reporting and troubleshooting technical issues and asking questions about gameplay. The second most common practice in both forums was SEP 6: Constructing Explanations and Designing Solutions. This makes sense in light of both in- game goals (designing chambers or spacecraft) as well as the common goals of forum users (to share those designs with the larger game community, and/or to report and troubleshoot issues). Examples of SEP 5: Mathematics and Computational Thinking were similarly infrequent in both forums, but when the practice did manifest it tended to occur in posts relating to troubleshooting technical issues—users engaged in abstraction, problem decomposition, algorithms, and debugging to address programmatic issues, isolate glitches, and provide instructions related to chamber development; examples of mathematics tended to relate to 54 distance, time, and other calculations. These types of discussions were common in both forums, and the lack of significant differences here may reflect the similar ways forum members tend to engage with this practice. One of the least common practices in either forum was SEP 7: Engaging in Argument from Evidence. Where there was evidence for this practice, it tended to relate to the historical development of designs or current trends in design. Notably, this aligns with the High School (9-12) grade band, which suggests that arguments may “come from current scientific or historical episodes in science” (NSTA, 2014). Although SEP 7 does include references to design, NGSS primarily describes the practice in relation to scientific reasoning and scientific argumentation (e.g., claims, supporting evidence, and evaluation). It seems possible, then, that this practice was overshadowed by SEP 6: Constructing Explanations and Designing Solutions. That is, since these games tended to foster practices with an engineering focus, discussions in the forums tended to focus more on designing solutions than on making scientific arguments. Posts in the KSP forum were twice as likely to engage with science and engineering practices, and were also significantly higher in depth and quality compared to posts in the Portal 2 forums. The KSP forums showed significantly more evidence of each individual practice with the exception of SEP 5: Mathematics and Computational Thinking. Evidence of SEP 2: Developing and Using Models was more common in KSP and virtually absent from Portal 2 forums. Discussions about modeling in KSP reflected the focus of the game’s mechanics; in the game, players build, test, evaluate, and refine spacecraft designs, and not surprisingly, tended to reference modeling more explicitly in the associated forums. The greater amount of evidence of SEP 3: Planning and Conducting Investigations in the KSP forums reflected features of the game as well as aspects of the forums: The game presents players with specific missions, and users 55 often engaged with SEP 3 while discussing these missions. Additionally, engagement with SEP 3 may have been influenced by the affordance of a subforum in the KSP community entitled “Challenges & Mission Ideas,” which encouraged players to design and tackle in-game challenges and report back with results. Similarly, the results of missions and analyses of performance data that occurred within the KSP forums—again, likely encouraged by both the game’s features and mechanics as well as the types of discussions happening online— contributed to the more frequent occurrence of SEP 4: Analyzing and Interpreting Data. Although both forums showed a good deal of evidence of SEP 6, posts within the KSP community showed significantly more. This may have been a result of the multiple pathways KSP offers for design. In the Portal 2 forums, members tended to engage in this practice either by addressing a technical problem or in discussions about designing chambers. But in the KSP forums, users often went beyond basic technical issues to discuss the construction of space- worthy vehicles, mods and mod designs, and the real-life applications of design. SEP 8: Obtaining, Evaluating, and Communicating Information was again significantly more common in the KSP forums. This example of an affordance taken in the KSP forums may have been influenced by a number of factors, including the culture of the community (with members who tended to be more invested in discussions), but perhaps more likely is the fact that KSP has such strong connections to real-life science and engineering—connections that appeared to spur more discussion of current trends in spacecraft design, the history of space exploration, and other topics that encouraged members to draw from, evaluate, and share outside sources of information. It's interesting that, at least within the KSP forums, SEP 4, SEP 6, and SEP 8 were some of the most frequently-coded practices, since studies have shown that these higher practices are 56 “elusive, diminished, or even absent in K-12 science classrooms” (Koomen et al., 2018). Instructional practices in the classroom tend to support students in asking questions and conducting investigations, but are often insufficient in providing learners opportunities to analyze or interpret data, construct explanations or design solutions, or communicate and evaluate information (Forbes, Biggers, & Zangori, 2013; Koomen, Blair, Young-Isebrand, & Oberhauser, 2014). This suggests that perhaps we can learn from the way these higher practices manifest outside the confines of formal education. The results of this study both reinforce and complicate previous findings. While there is growing evidence that members of online gaming communities engage in scientific discursive practices, systems-based reasoning, and model-based reasoning (Steinkuehler & Duncan, 2008) and that such communities can foster the kinds of skills, dispositions, and identities necessary for scientific inquiry (Asbell-Clarke et al., 2012; Gee, 2008), results of the current study caution against the oversimplification of affinity spaces. The extent to which participants engage in particular practices, take part in social interactions, and take advantage of technical affordances appears to be more contextual than much of the affinity space literature suggests. It may be more effective to conceptualize such spaces as sociotechnical assemblages to capture the complex interplay between the features of online spaces, how games are designed, the characterization of communities, and other factors that can influence learning in these interest-driven environments. Implications for Practice Design of learning environments. Online gaming forums are inextricably linked to the videogames they support and extend; it follows, then, that the features of those games are likely to influence the ways science and engineering practices manifest in the forums. While both games are framed in a fictional context, KSP makes more overt connections with science (with 57 missions and challenges focused on space exploration) and engineering (requiring players to design, build, test, and pilot spacecraft) compared to Portal 2. KSP also introduces specialized discourse that was often carried through and extended in the forums. Thus, playing KSP may prime members of the community to engage in deeper discussion of the practices. These findings align with research-based recommendations that learning games should focus on an authentic, complex problem from professional practice (Liang, Lee, & Chou, 2010) so that doing (game play) and knowing (the learning goals) are one and the same (Charsky & Mims, 2008; Squire, 2011). The significant positive relationship between higher quality posts and higher depth of engagement with practices, though not causal, suggests that multimodal affordances—that is, the ability to enhance text artifacts with multimodal content in the form of hyperlinks, images, screenshots, embedded video, etc.—should be encouraged. This supports previous research indicating that gaming forums can encourage participants to engage with academic practices in constructionist ways. Similarly, the online gaming forums I analyzed for this study provided affordances for learners to co-construct solutions to authentic problems which were shared publicly in online forums as digital artifacts (Papert, 1980; Papert & Harel, 1991). This study also illuminated affinity space characteristics that may be conducive for engagement with the practices. The more organized and active KSP forums showed greater evidence of the practices as a whole, and there were significant correlations between certain subforums and evidence of practices. Given that there was minimal engagement with NOS overall (an affordance not taken), it’s possible that the games did not make strong enough connections to the nature of science to encourage players to talk about it online, or that affinity spaces alone may not be enough to foster higher-level discussions about the nature of science. 58 Since the topics of some subforums appeared to influence the likelihood that users engaged with the nature of science, more strategic prompting might be necessary. This aligns with studies showing that particular types of teacher prompts (Cunningham & Carlsen, 2014) and thoughtfully designed topics in online gaming forums are better suited to encourage metacognition and the social construction of knowledge (Davis & Marone, 2016). Additionally, Nurturing posts tended to engage with practices and had higher Depth and Quality. Proactive moderation of the KSP community (an affordance taken) may have played a role in these differences. KSP moderators tended to proactively remove, penalize, or not approve “trolling” posts or posts with no real content; this, in turn, may contribute to the higher Depth and Quality of posts within the KSP community. Taken together, these findings suggest that educators should focus on games that make overt connections to science and engineering, introduce players to specialized discourse, and require players to solve in-game problems that have real-world applications. They may also wish to select or design forums that are organized by subtopics that focus discussion in desired directions (for example, topics that issue in-game mission assignments to encourage players to tackle challenges and report back with visual representations of models, as well as interpretations and analyses of performance data; and topics that prompt members to reflect more deeply on the nature of science). These spaces may be more effective when they are highly active, nurturing, well moderated, and support multimodality. Teacher education. Science learning needs to provide opportunities for students to make their thinking visible, in all its messiness. Science teaching, in turn, should involve active noticing of students’ thinking as it unfolds (Luna, Selmer, & Rye, 2018). But this is no easy task, as evidenced by the complexities of coding for this study (and echoed by other educational 59 researchers, e.g., Clegg & Kolodner, 2014). Within formal academic settings, in- and pre-service teachers often focus on noticing the accuracy of science content or whether students successfully complete phases of the science process (Talanquer, Tomanek, & Novodvorsky, 2013) instead of students’ thinking about that content. Additionally, teachers’ normative views of science discourse are often limited, and can devalue diverse ways of knowing (Carlone et al., 2011; Nasir, Rosebery, Warren, & Lee, 2006; Schwarz et al., 2017). Teachers’ epistemologies can also be misaligned with current reforms in science learning (e.g., viewing science as positivist, or as a concrete series of steps), and since what teachers notice influences what is learned, those misalignments matter (Plakitsi, Piliouras, & Efthimiou, 2016; Talanquer et al., 2013). One way to address this challenge is to provide teachers with more opportunities to practice noticing what learners say/write/do as they engage with science and engineering (Luna, et al., 2018; Talanquer et al. 2013). Some affinity spaces offer digital artifacts that could allow teachers to notice the messy, diverse, authentic ways learners engage with science and engineering practices in a participatory science learning environment, free from distractions of classroom management. In keeping with current science education reforms, teacher education and professional development programs may also wish to include games and affinity spaces that originate from a variety of cultures to explore different ways of making sense of the world.2 These interest-driven spaces may provide teachers examples of the diverse ways learners engage with the practices, as well as learners’ existing literacies so that teachers can more readily recognize, validate, and leverage the knowledge and experience learners bring to the classroom. 2 For example, Thunderbird Strike (LaPensée, 2017) is a game that draws from indigenous narratives to explore the impact of toxins on the environment as well as the ecologies of the Alberta tar sands and the Great Lakes. The points of reflection offered on the game’s official website could provide starting points, or subforum topics, to promote engagement with science practices. 60 Implications for Theory This study contributes to our knowledge of affinity spaces as well as how learning occurs in such environments. Many studies have investigated classroom or lab settings, but our understanding of how peer learning occurs ‘in the wild’ is still nascent (Barron, 2006; Hutchins, 1995). As many scholars have noted, research must evolve to accommodate the changing characteristics of such spaces (Hudson, Duncan, & Reeve, 2015; Lammers et al., 2012). Whether by design, affiliation with particular games, the nature of the community, or—most likely— interactions between these dimensions, all affinity spaces are not created equal. Indeed, even these two seemingly similar online gaming forums were noticeably different in terms of affordances taken/not taken. This reinforces the need to complicate the concept of the affinity space, recognize the multitude of factors that influence learning in these environments, and consider what this means for science learning. Classroom science is often abstract, disconnected from students’ lived experiences, and insufficient to promote interest in science and engineering; this suggests a need for environments that integrate the home lives and personal interests of learners—in other words, learning that draws from learners’ funds of knowledge (González, Moll, & Amanti, 2013) to foster awareness of meaningful applications for science in our everyday lives (Barron, 2006; Clegg & Kolodner, 2014). In a study of the Kitchen Science Investigators program, for example, researchers found that learners developed scientific dispositions and understandings while engaging in cooking activities—though the ways they engaged in inquiry could be more subtle and challenging to recognize (Clegg & Kolodner, 2014). The flexibility and scaffolding of the program—which (like the forums) sat at the “intersection of formal and informal science learning” (p. 40)—forged connections between science concepts and learners’ own values, interests, and social spheres. 61 Many out-of-school activities—like cooking at home, gardening, and playing videogames—hold a great deal of potential for informal science learning. This and other research suggests that the tacit knowledge explored through such activities can be more effectively leveraged and extended when learners are prompted to interpret, explain, and reflect on their learning in a participatory environment like an affinity space (Barab & Hay, 2001; Bell & National Research Council, 2009). 62 Limitations There are several limitations to mention for this study. For one, affinity spaces tend to be porous; there is rarely a clearly defined ‘boundary’ between one affinity portal and another. The forums I examined represent individual points of entry into the affinity spaces surrounding two specific videogames, but the networks players leverage for learning often include Twitch streaming, YouTube channels, social media, wikis, and more. For example, a post on the Portal 2 forum may contain links to external sites or embedded Steam Workshop items. This is a limiting factor that may enforce the idea of a distinction between these interconnected spaces that forum participants would not recognize, and also presents theoretical challenges. As Duncan and Hayes (2012) have recognized, the inadequacies of sociocultural and situated learning theories offer pressing questions for educational researchers studying these dynamic spaces: “How does one even define the learning environment in such instance? What are the salient dimensions of such environments, given this ‘mash-up’ of spaces and tools?” (p. 6). Therefore this study’s results may not generalize across the wider dimensions of videogame affinity spaces. Findings may also not be generalizable to all videogame affinity spaces, particularly if they do not incorporate engineering/design mechanics. Given the large amount of data to be analyzed, I focused primarily on discourse in the form of written posts as I interpreted the extent to which users engaged with science practices (although I did consider multimodal content in the form of visual images, hyperlinks, and videos in evaluating the quality of discussions). But multimodal theories of communication increasingly reject the idea that non-linguistic semiotics must always be relegated to a supporting role. In multimodal spaces, meaning emerges from the interactions between different semiotic modes— and the relationship is not necessarily of one mode replicating or reinforcing the other—for 63 example, an image that reproduces the meaning of written text (Roderick, 2016). Thus, I acknowledge that my approach privileged text over other semiotic systems, and therefore any conclusions may not extend to the rich and varied ways that meaning-making can occur in multimodal discourse. Another limitation relates to the diversity of perspectives included in these spaces. Demographic details for forum users were neither required nor verifiable, but it is possible that active users tend to be male. This is a perception implied by some forum discussions in which users debate whether videogame forums in general tend to be male-dominated, whether these particular forums tend to be even more so (reflecting U.S. disparities in STEM fields), or whether these perceptions are inaccurate since some non-male users may use masculine or gender-neutral usernames. This reflects yet another challenge for affinity space research, since traditional social markers like gender, race, ethnicity, age, and even location are less visible in virtual spaces (Buckingham, 2007; Duncan & Hayes, 2012). Regardless, it is possible that fewer members of underrepresented groups are participating actively on these forums; therefore, this data may not fully capture a diverse group of perspectives. It’s also important to acknowledge the way each of the games examined in this study are marketed, and the types of players each game tends to attract. KSP is marketed more heavily as an engineering-based game, and therefore may attract players who are drawn to (and more likely to discuss) the science and engineering aspects of the game. Portal 2 does contain science- related content, but is generally marketed as an entertainment game. As a result, KSP players may have already been predisposed to greater engagement with the practices. This limitation should be kept in mind when evaluating characteristics of affinity spaces that may be more conducive for engagement with specific practices. 64 Suggestions for Future Research This was an exploratory study focused on two specific videogame forums. Although the results offer implications for both theory and practice, more research is needed to fully understand the multifaceted and nuanced ways that learners engage with science and engineering practices in affinity spaces, as well as the affordances and constraints that the various features and types of affinity spaces may offer for science learning. It would be useful to incorporate multimodal discourse analysis (see Roderick, 2016) in future studies to more deeply explore how linked and embedded multi-modal content contributes to scientific sensemaking within affinity spaces. Doing so could further explore how learners make authentic connections to science and engineering as they move through the ecology of social interactions, resources, and spaces that comprise their experiences (Barron, 2006). And could help to illuminate the more dynamic and fluid aspects of the “constellations” of practices that occur in online affinity spaces (Abrams & Gerber, 2014; Martin, 2012; Steinkuehler, 2007). This study addressed two of the dimensions of the Next Generation Science Standards. For feasibility’s sake I did not code for Crosscutting Concepts, Disciplinary Core Ideas, or Engineering Connections. Future research should investigate these other dimensions of the standards, and might also incorporate a wider variety of games and forums, including games with a stronger focus on science. It would also be fruitful to expand such research to game communities that originate from or are affiliated with traditions outside of Western European- American views of science. 65 Conclusion The nuanced and multifaceted nature of affinity spaces require that we pay close attention to the complex interplay between the technical affordances of games, the features of affiliated online communities, and the types of social interactions that occur within them. As the differences between the two forums examined in this study indicate, not all affinity spaces are equally suited for particular practices. Affinity spaces are influenced by a number of factors including the mechanics and goals of the games themselves, the way online spaces are structured, community standards and norms, and technical affordances. Educators interested in their potential for science learning should therefore purposefully design or select games and forums whose features align with intended outcomes. Teacher education programs may also be interested in leveraging the practices that occur within affinity spaces as opportunities to notice the ways individuals engage with science and engineering outside of the classroom. Similarly, research should acknowledge the complexity of affinity spaces and continue to explore the social and technical affordances they offer for science and engineering practices. 66 APPENDICES 67 APPENDIX A: Inter-rater Reliability Results This appendix reports the final results of inter-rater reliability across all categories based on a purposeful, high-density sample of 24 posts (see Table 17). Table 17: Final Round of Inter-rater Reliability Across All Categories Portal 2 % agreement Cohen’s kappa % agreement Cohen’s kappa % agreement Cohen’s kappa 100% 1.00 100% 1.00 100% 1.00 KSP Portal 2 & KSP SEP (All) SEP (1) SEP (2) SEP (3) SEP (4) Engineering Science Engineering Science Engineering Science 100% 100% 100% 100% 100% 100% Engineering 100% NaN NaN NaN NaN NaN NaN NaN 67% 81% 76% 100% 81% 76% 86% 68 0.31 0.48 0.30 NaN 0.62 0.35 0.71 79% 83% 79% 100% 83% 79% 88% 0.58 0.52 0.32 NaN 0.66 0.36 0.73 Table 17 (cont’d) SEP (5) SEP (6) SEP (7) SEP (8) Engineering Science Engineering Science Engineering Science Engineering Science NOS (All) NOS (1) NOS (2) NOS (3) NOS (4) NOS (5) NOS (6) Portal 2 KSP Portal 2 & KSP % agreement Cohen’s kappa % agreement Cohen’s kappa % agreement Cohen’s kappa 100% 100% 100% 100% 100% 100% 100% 100% 100% 67% 100% 100% 100% 100% 100% NaN NaN 1.00 NaN NaN NaN NaN NaN 1.00 0.40 NaN NaN NaN NaN NaN 91% 86% 71% 71% 71% 91% 81% 67% 86% 91% 95% 95% 91% 100% 91% 69 0.77 0.35 0.40 0.29 0.36 0.70 0.60 0.27 0.50 0.46 0.83 0.86 0.77 1.00 0.74 92% 88% 75% 75% 75% 92% 83% 71% 88% 88% 96% 96% 92% 100% 92% 0.78 0.36 0.47 0.31 0.40 0.70 0.63 0.30 0.60 0.50 0.83 0.86 0.78 1.00 0.75 Table 17 (cont’d) NOS (7) NOS (8) Nurturing Neutral Elitist Depth Quality Portal 2 KSP Portal 2 & KSP % agreement 100% 100% 100% 100% 100% 67% 100% Cohen’s kappa % agreement Cohen’s kappa % agreement Cohen’s kappa 1.00 NaN 1.00 1.00 NaN 0.50 1.00 91% 86% 100% 95% 100% 91% 91% 0.70 0.50 1.00 0.64 1.00 0.83 0.79 92% 88% 100% 96% 100% 88% 92% 0.75 0.51 1.00 0.78 1.00 0.78 0.84 Note 1: Depth & Quality were coded on a scale of 0-2. Note 2: NaN = Not a Number, indicating a divisor of 0 because no posts were flagged for the code. 70 APPENDIX B: Correlations Between Codes This appendix reports correlations across top-level SEP, NOS, and characterization, depth, and quality codes for both forums (see Table 18). Table 18: Correlations Between Top-level Categories SEP1 SEP2 SEP3 SEP4 SEP5 SEP6 SEP7 SEP8 AnySEP NOS Nurturing Neutral Elitist Depth Quality Pearson's r SEP1 — p-value — 0.226 Pearson's r SEP2 — < .001 p-value — Pearson's r SEP3 0.234 0.267 — < .001 < .001 p-value — SEP4 Pearson's r 0.286 0.268 0.441 — < .001 < .001 < .001 p-value — Pearson's r SEP5 0.261 0.207 0.108 0.142 — < .001 < .001 < .001 < .001 p-value — SEP6 Pearson's r 0.151 0.204 0.179 0.242 0.301 — < .001 < .001 < .001 < .001 < .001 p-value — 0.038 0.062 Pearson's r SEP7 0.151 0.166 0.079 0.185 — p-value 0.005 < .001 0.088 < .001 < .001 < .001 — 0.226 0.170 0.284 0.161 0.291 0.155 Pearson's r SEP8 0.285 p-value < .001 < .001 < .001 < .001 < .001 < .001 < .001 — — 71 Table 18 (cont’d) NOS Neutral — — — — AnySEP Pearson's r 0.393 0.171 0.516 0.259 0.387 0.370 0.263 0.641 — p-value < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 — Pearson's r 0.115 0.130 0.129 0.160 0.081 0.182 0.373 0.229 0.164 p-value < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 Pearson's r Nurturing -0.008 0.124 0.039 0.034 0.082 0.030 0.068 0.065 0.041 0.086 p-value < .001 < .001 < .001 0.066 0.004 0.002 0.187 0.131 0.081 0.706 Pearson's r -0.081 -0.037 -0.052 -0.054 -0.036 -0.063 -0.024 -0.044 -0.125 -0.010 -0.705 p-value < .001 0.660 < .001 0.049 0.287 0.005 0.107 0.016 0.021 0.102 < .001 -0.590 -0.034 -0.001 -0.028 -0.047 -0.004 -0.036 -0.032 -0.005 -0.030 Pearson's r -0.000 0.008 p-value 0.175 0.813 0.148 0.107 0.846 0.036 0.207 0.967 0.126 0.737 < .001 0.994 -0.069 -0.082 Spearman's rho 0.105 0.235 0.667 0.335 0.194 0.541 0.380 0.483 0.473 0.380 0.492 < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 p-value < .001 < .001 -0.170 Spearman's rho 0.126 0.196 0.167 0.387 0.177 0.190 0.128 0.249 0.199 0.283 0.120 < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 < .001 Note: Spearman’s rho used for ordinal categories. 0.002 0.002 Quality Elitist Depth — — — — p-value — — 0.292 — — < .001 0.934 72 APPENDIX C: Correlations Between KSP Subforums and Practices This appendix describes the relationships between SEP, NOS, and the top four KSP subforums (see Table 19). Table 19: Correlations Between KSP Subforums and Practices Discussion Science & Spaceflight Lounge Tech Support SEP NOS Discussion Pearson's r p-value Science & Spaceflight Pearson's r Lounge Tech Support SEP NOS p-value Pearson's r p-value Pearson's r p-value Pearson's r p-value Pearson's r p-value — — -0.281 < .001 -0.260 < .001 -0.195 < .001 0.022 0.481 -0.086 0.006 — — -0.070 0.026 -0.332 < .001 -0.080 0.011 — — 0.125 < .001 0.006 0.859 — — 0.144 — < .001 — — — -0.102 0.001 -0.076 0.016 0.116 < .001 0.264 < .001 73 REFERENCES 74 REFERENCES Abrams, S. S., & Gerber, H. R. (2014). Cross-literate digital connections: Contemporary frames for meaning making. English Journal, 103(4), 18. Abrams, S. S., & Lammers, J. C. (2017). Belonging in a Videogame Space: Bridging Affinity Spaces and Communities of Practice. Teachers College Record, 119, 1-34. Achieve (2010). International Science Benchmarking Report - Taking the Lead in Science Education: Forging Next-Generation Science Standards. Retrieved from https://www.achieve.org/files/InternationalScienceBenchmarkingReport.pdf American Association for the Advancement of Science (1993). Benchmarks for science literacy. Oxford University Press, New York. Asbell-Clarke, J., Edwards, T., Rowe, E., Larsen, J., Sylvan, E., & Hewitt, J. (2012). Martian Boneyards: Scientific Inquiry in an MMO Game. International Journal of Game-Based Learning, 2(1), 52-76. Barab, S., & Hay, K. (2001). Doing Science at the Elbows of Experts: Issues Related to the Science Apprenticeship Camp. Journal of Research in Science Teaching, 38(1), 70-102. Barab, S., Scott, B., Siyahhan, S., Goldstone, R., Ingram-Goble, A., Zuiker, S., & Warren, S. (2009). Transformational Play as a Curricular Scaffold: Using Videogames to Support Science Education. Journal of Science Education and Technology, 18(4), 305-320. Barron, B. (2006). Interest and self-sustained learning as catalysts of development: A learning ecologies perspective. Human Development, 49, 193 – 224. Bell, P., & National Research Council (U.S.). (2009). Learning science in informal environments: People, places, and pursuits. Washington, DC: National Academies Press. Berland, L. K., & Lee, V. R. (2012). In Pursuit of Consensus: Disagreement and legitimization during small-group argumentation. International Journal of Science Education, 34(12), 1857–1882. https://doi.org/10.1080/09500693.2011.645086 Bernard, H. R., & Ryan, G.W., (2010). Analyzing Qualitative Data: Systematic Approaches. Thousand Oaks, CA: Sage. Berland, L. K., Schwarz, C. V., Krist, C., Kenyon, L., Lo, A. S., & Reiser, B. J. (2016). Making scientific practices meaningful for students: Epistemologies in practice. Journal of Research in Science Teaching, 53(7), 1082–1112. https://doi.org/10.1002/tea.21257 75 Braga, B., Pope, B., Druyan, A., Hanich, L., Holtzman, S., Soter, S., Silvestri, A. (2014). Cosmos: A spacetime odyssey [Television series]. United States: Twentieth Century Fox Home Entertainment Ltd. Bransford, J., Brown, A. L., & Cocking, R. R. (2000). How People Learn: Brain, Mind, Experience, and School. Washington DC: National Academy Press. Buckingham, D. (Ed.) (2007). Youth, identity, and digital media. The John D. and Catherine T. MacArthur Foundation Series on Digital Media and Learning. Cambridge, MA: The MIT Press. Light. Cajete, G. A. (2000). Native science: Natural laws of interdependence. Santa Fe, NM: Clear Carlone, H. B., Haun-Frank, J. & Webb, A. (2011). Assessing equity beyond knowledge- and skills-based outcomes: A comparative ethnography of two fourth-grade reform-based science classrooms. Journal of Research in Science Teaching, 48: 459-485. doi:10.1002/tea.20413 Clark, D. B., Sengupta, P., Brady, C. E., Martinez-Garza, M. M., & Killingsworth, S. S. (2015). Disciplinary integration of digital games for science learning. International Journal of STEM Education, 2(1). https://doi.org/10.1186/s40594-014-0014-4 Clegg, T., & Kolodner, J. (2014). Scientizing and Cooking: Helping Middle-School Learners Develop Scientific Dispositions. Science Education, 98(1), 36–63. https://doi.org/10.1002/sce.21083 Charsky, D., & Mims, C. (2008). Integrating commercial off-the-shelf video games into school curriculums. TechTrends, 52(5), 38–44. Chinn, C. A., & Malhotra, B. A. (2002). Epistemologically authentic inquiry in schools: A theoretical framework for evaluating inquiry tasks. Science Education, 86(2), 175-218. doi:10.1002/sce.10001 Creswell, J. W. (2008). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (3rd ed.). Upper Saddle River, N.J: Pearson/Merrill Prentice Hall. Cunningham, C. M., & Carlsen, W. S. (2014). Teaching Engineering Practices. Journal of Science Teacher Education, 25(2), 197–210. https://doi.org/10.1007/s10972-014-9380-5 Davis, D., & Marone, V. (2016). Learning in Discussion Forums: An Analysis of Knowledge Construction in a Gaming Affinity Space. International Journal of Game-Based Learning, 6(3), 1-17. 76 Duncan, S. & Hayes, E. (2012). Expanding the affinity space: An introduction. In E.R. Hayes & S. C. Duncan (eds) Learning in video game affinity spaces: new literacies and digital epistemologies. New York: Peter Lang Publishing, p.p. 1-22. Forbes, C. T., Biggers, M., & Zangori, L. (2013). Investigating essential characteristics of scientific practices in elementary science learning environments: The Practices of Science Observation Protocol (P-SOP). School Science and Mathematics, 113(4), 180–190. Foster, A., Koehler, M., & Mishra, P. (2006, June). Game-based learning of physics content: the effectiveness of a physics game for learning basic physics concepts. In EdMedia+ Innovate Learning (pp. 2119-2125). Association for the Advancement of Computing in Education (AACE). Funk, C., & Goo, S. K. (2015, September 10). A Look at What the Public Knows and Does Not Know About Science. Retrieved from http://www.pewinternet.org/2015/09/10/what-the- public-knows-and-does-not-know-about-science/ Gee, J. P. (2004). Situated language and learning: A critique of traditional schooling. New York, NY: Routledge. Gee, J. P. (2008). Learning and games. In K Salen (Ed.), The Ecology of Games: Connecting Youth, Games, and Learning. The John D. and Catherine T. MacArthur Foundation Series on Digital Media and Learning (pp. 21–40). Cambridge, MA: The MIT Press. doi:10.1162/dmal.9780262693646.021. Gee, J. P., & Hayes, E. (2012). Nurturing affinity spaces and game-based learning. In Games, Learning, and Society: Learning and Meaning in the Digital Age (pp. 129-153). Cambridge University Press. doi: 10.1017/CBO9781139031127.015. Gerber, H., Abrams, S., Curwood, J., & Magnifico, A. (2016). Conducting Qualitative Research of Learning in Online Spaces. Beaverton: Ringgold Inc. González, N., Moll, L. C., & Amanti, C. (Eds.). (2013). Funds of knowledge: Theorizing practices in households, communities, and classrooms. Routledge. Greeno, J., Collins, A., & Resnick, L. (1996). Cognition and learning. In D. Berliner & R. Calfee (eds.), Handbook of educational psychology (pp. 15-46). New York, NY: Macmillan. Hudson, R., Duncan, S., & Reeve, C. (2015). Affinity Spaces for Informal Science Learning: Developing a Research Agenda. [Report]. Games+Learning+Society Conference. Hutchins, E. (1995). Cognition in the Wild. Cambridge, MA: MIT Press. Koomen, M. H., Blair, R., Young-Isebrand, E., & Oberhauser, K. S. (2014). Science professional development with teachers: Nurturing the scientist within. Electronic Journal of Science Education, 18(6), 1–28. 77 Koomen, M. H., Rodriguez, E., Hoffman, A., Petersen, C., & Oberhauser, K. (2018). Authentic science with citizen science and student-driven science fair projects. Science Education, 102(3), 593–644. https://doi.org/10.1002/sce.21335 Kuhn, D. (2011). What is Scientific Thinking and How Does it Develop? In Goswami, U. C. (Ed). The Wiley-Blackwell handbook of childhood cognitive development (2nd ed.). Malden, MA: Wiley-Blackwell. Kuhn, D. (2010). Teaching and learning science as argument. Science Education, 94(5), 810– 824. https://doi.org/10.1002/sce.20395 Lammers, J. C., Curwood, J. S., & Magnifico, A. M. (2012). Toward an affinity space methodology: Considerations for literacy research. English Teaching, 11(2), 44. Landers, R. N., Brusso, R. C., Cavanaugh, K. J., & Collmus, A. B. (2016). A primer on theory- driven web scraping: Automatic extraction of big data from the Internet for use in psychological research. Psychological Methods, 21(4), 475–492. https://doi.org/10.1037/met0000081 Landis, J.R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics. 33 (1): 159–174. doi:10.2307/2529310. JSTOR 2529310. LaPensée, E. (2017). Thunderbird Strike. [Videogame]. East Lansing, MI. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cambridge University Press. Lehrer, R., Schauble, L., & Lucas, D. (2008). Supporting development of the epistemology of inquiry. Cognitive Development, 23(4), 512–529. https://doi.org/10.1016/j.cogdev.2008.09.001 Liang, C., Lee, Y.-Z., & Chou, W.-S. (2010). The design considerations for game-based learning. Educational Technology, 50(2), 25-28. Luna, M. J., Selmer, S. J., & Rye, J. A. (2018). Teachers’ Noticing of Students’ Thinking in Science Through Classroom Artifacts: In What Ways Are Science and Engineering Practices Evident? Journal of Science Teacher Education, 29(2), 148–172. https://doi.org/10.1080/1046560X.2018.1427418 Manz, E. (2015). Resistance and the Development of Scientific Practice: Designing the Mangle Into Science Instruction. Cognition and Instruction, 33(2), 89–124. https://doi.org/10.1080/07370008.2014.1000490 Marin, A., & Bang, M. (2015). Designing pedagogies for indigenous science education: finding our way to storywork. Journal of American Indian Education, 54(2), 29. 78 Markham, A., & Buchanan, E. (2012). Ethical decision-making and Internet research: Version 2.0. Recommendations from the AoIR Ethics Working Committee. Retrieved from http://www.aoir.org/reports/ethics2.pdf Markham, A. N., Tiidenberg, K., & Herman, A. (2018). Ethics as methods: Doing ethics in the era of big data Research—Introduction. Social Media + Society, 4(3), 205630511878450. doi:10.1177/2056305118784502 Martin, C. (2012). Video Games, Identity, and the Constellation of Information. Bulletin of Science, Technology, & Society, 32, 384-392. Martinez-Garza, M., Clark, D. B., & Nelson, B. C. (2013). Digital games and the US National Research Council’s science proficiency goals. Studies in Science Education, 49(2), 170– 208. https://doi.org/10.1080/03057267.2013.839372 Merriam, S. B. (1998). Qualitative research and case study applications in education. San Francisco, Jossey-Bass. Morris, B., Croker, S., Zimmerman, C., Gill, D., & Romig, C. (2013). Gaming science: the “Gamification” of scientific thinking. Frontiers in Psychology, 4. https://doi.org/10.3389/fpsyg.2013.00607 Nasir , N. S., Rosebery , A. S., Warren, B., & Lee, C. D (2006). Learning as a cultural process: Achieving equity through diversity. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (489–504). New York, NY: Cambridge University Press. National Assessment of Educational Progress (NAEP): Science Assessment (2015). (n.d.). U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Retrieved from https://nationsreportcard.gov/science_2015/ National Research Council (1996). National science education standards. National Academy Press, Washington, DC. National Research Council (2011). Honey, M., Hilton, M. L., & National Academies Press (Eds.). Learning science through computer games and simulations. Washington, D.C: National Academies Press. National Research Council (2012). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. Washington, DC: The National Academies Press. https://doi.org/10.17226/13165. National Science Foundation (2011). Women, minorities, and persons with disabilities in science and engineering: 2011. Arlington, VA: NSF. National Science Foundation (2015). Advancing Informal Science Learning (AISL) Solicitation nsf155593. Washington, D.C. 79 National Science Teachers Association (2014). NGSS at NSTA. Retrieved from https://ngss.nsta.org NGSS Lead States (2013). Next Generation Science Standards: For States, By States. Retrieved from http://www.nextgenscience.org/ Organization for Economic Co-operation and Development (2017). Science performance (PISA) (indicator). doi: 10.1787/91952204-en (Accessed on 17 May 2017). Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York, NY: Basic Books. Papert, S., & Harel, I. (1991). Situating constructionism. Constructionism. Retrieved from http://namodemello.com.br/pdf/tendencias/situatingconstrutivism.pdf Pellicone, A., & Ahn, J. (2014). Construction and community: Investigating interaction in a minecraft affinity space. In Proceedings of the Tenth Conference for Games + Learning + Society - GLS. Madison, WI: ETC Press. Plakitsi, K., Piliouras, P., & Efthimiou, G. (2016). Discourse Analysis: A Tool for Helping Educators to Teach Science. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 18(1). doi:http://dx.doi.org/10.17169/fqs-18.1.2397 Poli, D., Berenotto, C., Blankenship, S., Piatkowski, B., Bader, G. A., & Poore, M. (2012). Bringing Evolution to a Technological Generation: A Case Study with the Video Game SPORE. The American Biology Teacher, 74(2), 100–103. https://doi.org/10.1525/abt.2012.74.2.7 Reiser, B. J., Berland, L. K., & Kenyon, L. (2012). Engaging students in the scientific practices of explanation and argumentation. The Science Teacher, 79(4), 34. Roderick, I. (2016). Critical discourse studies and technology: A multimodal approach to analysing technoculture. London, UK: Bloomsbury Publishing. Saldaña, J. (2016). The coding manual for qualitative researchers. London: Sage. Salganik, M. J. (2018). Bit by bit: Social research in the digital age. Princeton: Princeton Sagan, C. (1989). Why we need to understand science. Parade. Sept. 10, p. 10. University Press. Schwarz, C., Passmore, C., & Reiser, B. J. (2017). Helping Students Make Sense of the World Using Next Generation Science and Engineering Practices. National Science Teachers Association. https://doi.org/10.2505/9781938946042 80 Singer, S. R., Hilton, M. L., Schweingruber, H. A., & National Research Council (U.S.). Committee on High School Science Laboratories (2006). America's lab report: Investigations in high school science. Washington, DC: National Academies Press. Sipe, L. R., & Ghiso, M. P. (2004). Developing conceptual categories in classroom descriptive research: Some problems and possibilities. Anthropology and Education Quarterly, 35(4), 472-85. Squad (2015). Kerbal Space Program. [Videogame]. Mexico City, Mexico: Monkey Squad. Squire, K. (2011). Video games and learning: teaching and participatory culture in the digital age. New York: Teachers College Press. Steinkuehler, C. (2007). Massively Multiplayer Online Gaming as a Constellation of Literacy Practices. E-Learning and Digital Media 4(3), 297 - 318. https://doi.org/10.2304/elea.2007.4.3.297 Steinkuehler, C., & Duncan, S. (2008). Scientific Habits of Mind in Virtual Worlds. Journal of Science Education and Technology, 17(6), 530–543. https://doi.org/10.1007/s10956-008- 9120-8 Talanquer, V., Tomanek, D., & Novodvorsky, I. (2013). Assessing students’ understanding of inquiry: What do prospective science teachers notice? Journal of Research in Science Teaching, 50(2), 189–208. https://doi.org/10.1002/tea.21074 Tekinbas, K. S., Zimmerman, E. (2003). Rules of play : game design fundamentals. Cambridge, Mass.: MIT Press. Tyson, N.D. (2009). Neil DeGrasse Tyson [Television series episode]. In E. Cerna (host) Conversations at KCTS9. Seattle, WA: Cascade Public Media. Valve Corporation. (2011). Portal 2. [Videogame]. Bellevue, WA: Valve Corporation. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Welser, H., Smith, M., Fisher, D. & Gleave, E. (2008). Distilling digital traces: computational social science approaches to studying the internet. In N. Fielding, R. M. Lee, & G. Blank The SAGE Handbook of Online Research Methods (pp. 116-140). SAGE Publications, Ltd doi: 10.4135/9780857020055.n7 Wouters, P., van Nimwegen, C., van Oostendorp, H., & van der Spek, E. D. (2013). A meta- analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249–265. doi: 10.1037/a0031311 81 Zimmerman, C. (2014). Developing Scientific Thinking in the Context of Video Games: Where to Next? In F. Blumberg (Ed). Learning by playing: frontiers of video gaming in education (pp. 54–68). Oxford: Oxford University Press. 82