EXPLORING THE DYNAMICS OF MOTIVATION AND ENGAGEMENT IN MODEL- BASED LEARNING By Bethany Furqueron A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Plant Biology – Doctor of Philosophy 2024 ABSTRACT Attracting and retaining diverse individuals is a core goal for reform efforts within in Science, Technology, Engineering, and Mathematics (STEM) gateway courses. Evidence-based instructional strategies in STEM, including those that incorporate scientific practices, have demonstrated promising findings regarding student learning gains. Inspired by previous findings in an introductory course taught through model-based instruction (MBI), my dissertation aims to gain a better understanding of affective mechanisms involved in the way students across all achievement levels are learning in this context. My dissertation measures student’s motivation, and cognitive and emotional engagement in a model-based, introductory biology context. Using validated survey scales contextualized for a model-based context, I generated student motivational profiles at the beginning and end of a semester to examine how profiles remain stable or change over time, and the relationship between motivational profiles and student achievement level. I also developed and applied tools to measure students’ cognitive and emotional engagement during model-based tasks to study engagement in different model contexts and explore the relationship between engagement and student achievement level. My results found that achievement measures (i.e., grades) did not predict levels of motivation and engagement, which may suggest these as potential mechanisms that explain how and why MBI and other practice-based instructional methods are successful. Continued research into understanding mechanisms that may explain performance differences across student achievement levels can advance pedagogical approaches in STEM and promote persistence and diversity among STEM learners. v This dissertation is dedicated to my son, Luke, my parents, Roger and Cheryl, my husband, Jake, and our soon-to-arrive, Baby F - my gratitude for each of you is immeasurable. iii ACKNOWLEDGEMENTS Everyone’s journey through graduate school is unique; undoubtedly filled with challenges and successes along the way. My one piece of advice for those who come after me is to not let anyone or any situation make you feel unworthy or uncapable of reaching your goals. My journey to obtain my Ph.D. is one that I could not have traveled alone, and it is with great honor and privilege that I share gratitude to those who made reaching this goal of mine possible. First, I wish to express my deepest appreciation to Tammy Long, my advisor, for her guidance and support in research, academia, and life. There are people in this world that deserve much more credit than what they get: she is one of them. Tammy - so much of who I am now has been inspired by you. The way I teach is a reflection of learning by your side in the classroom and observing your true dedication to learning for all. Your empathy for those around you, creativity (yes, you are creative, whether you want to admit it or not!) in designing learning and research experiences, and ability to find solutions for everything is some sort of magical power you have. You are a fierce protector of those you care about, and I thank you for advocating for me in so many ways these last seven years. I would not have been able to balance graduate school while keeping my head above water in life without you. You inspire many around you without even realizing it. Thank you for everything. I was truly blessed with the best committee members: Melanie Cooper, Amelia Gotwals and Shinhan Shiu. Melanie - your knowledge and passion for research and learning in college science courses has challenged me to think critically in my own work and become inspired by opportunities for growth and change. Amelia – you allowed me to be creative in my ideas while also keeping me grounded in the true nature of qualitative work. I thank you for challenging me to see my research through alternative lenses and gain a deeper appreciation for the stories of our iv students. Shinhan – I cannot help but get the biggest smile when I see you! Your positivity and support for new ideas is unmatched. I am thankful for the opportunity I had to work with you and learn from you in the IMPACTS program. I also want to express deep appreciation for your ‘check-ins’ on Luke and me. These small acts of care played a big role in my perseverance through graduate school. Melanie, Amelia, and Shinhan- I am a better person and scientist because of each of you. Thank you to our one-of-a-kind and hardworking undergraduates in the Long Lab. A special thank you to Hunter Hicks, Devin Babi and Josh Marwede for their work in transcribing interviews. In addition, I thank the anonymous undergraduate students who participated in my surveys and interviews used for this study. It is known that the Long Lab is one of the best labs on campus collectively made up of some of the coolest people around. I am grateful for my fellow peers in the Lab who provided constructive criticism, lots of good food, and many, many laughs. I was blessed to work with two amazing post-docs, Jyothi Kumar and Steve Bennett– thank you for being sources of critical feedback and emotional support. Steve - you have such a big heart and give the best hugs out of anyone I know. I am fortunate to be able to call you a friend. Joelyn de Lima and Seth Hunt - first, thank you both for your help in facilitating the interviews used in my research. Joelyn - your resilience and strength inspired my own resilience and strength. There may not be a human alive that cares more for the wellbeing of those around them than you. I am so incredibly grateful for your unwavering support, advice, and care for Luke for me. I also thank you for your willingness and selflessness to contribute countless of hours on top of your own work to help me develop and apply the cognitive engagement in modeling (CEM) framework. Seth - thank you for always having a listening ear and being such a great beer-drinking buddy. I am so happy and thankful we v get to continue working together, because it means there will be many more beers and laughs to be shared! While on the topic of beer, I want to recognize Ironbark Brewing Company. Thank you for being a space for me to work while enjoying a good beer, tasty food, and meeting some of the coolest bar tenders and patrons in Jackson. Space love. To my husband, Jake, who has shown me what true love is supposed to be like. I am so thankful you came into my life. You will never know how truly grateful I am to you for rebuilding my confidence and providing me unwavering support as I finished graduate school. You understood my priorities and never questioned them. I appreciate your enthusiasm for my research, even if I do a terrible job of explaining it. Thank you for joining Luke and I on life’s journey, and I cannot wait for our next steps as a family of four and our many fun adventures to come! To my son and built-in best friend, Luke. Raising you as a single Mom from early in my Ph.D. journey was scary, and I doubted my ability to fill the roles of a mother, researcher, student, and teacher. Yet, you made every single moment worth it. You truly are my biggest blessing and my greatest accomplishment. Thank you for being the best motivation to not give up. I love you, buddy. To our soon-to-arrive baby, I hope you will always know your worth and never give up on your dreams. I love you and I can’t wait to meet you! Finally, to my parents, Roger and Cheryl, I made it here because of you. There are no words to express my appreciation for your support throughout my entire life, but especially over the past seven years. There were sure some major changes that brought unimaginable challenges but you both were there, with unwavering support, for every single step. Thank you for always vi being willing to help with Luke, even at the drop of a hat, providing me extra time and space in order to be successful in graduate school. We are so lucky to have you, and I am so proud to be your daughter. vii TABLE OF CONTENTS INTRODUCTION ............................................................................................................................. 1 REFERENCES ...................................................................................................................... 5 CHAPTER ONE: Exploring motivational profiles in a model-based introductory biology course ............................................................................................................................................... 11 REFERENCES .................................................................................................................... 51 APPENDIX ......................................................................................................................... 68 CHAPTER TWO: Measuring student cognitive engagement in modeling (CEM): development and application of a CEM framework ............................................................................................. 72 REFERENCES .................................................................................................................. 119 APPENDIX ....................................................................................................................... 141 CHAPTER THREE: Measuring emotional engagement in modeling (EEM): development and application of an emoji-based EEM scale ....................................................................................143 REFERENCES .................................................................................................................. 163 APPENDIX ....................................................................................................................... 174 CONCLUSION ............................................................................................................................176 REFERENCES .................................................................................................................. 181 viii INTRODUCTION Despite the growing need for diverse scientists entering the science, technology, engineering, and mathematics (STEM) workforce, attrition (i.e., switching to a non-STEM pathway or leaving college altogether) remains high among STEM undergraduates (Chen, 2015; Lytle et al., 2021; National Science Foundation [NSF], 2012). Attracting and retaining diverse individuals in STEM has been a persistent problem for decades (Kennedy et al., 2021; National Research Council [NRC], 2010; Olson & Riordan, 2012; President’s Council of Advisors on Science and Technology [PCAST], 2012). Hunter (2019) identified three commonalities among students’ decisions to leave STEM: (1) poor quality of teaching; (2) issues with curricular design, such as content overload, pace of delivery, and poor alignment between content taught and assessed; and, (3) trouble with conceptual understanding. Their findings echo a persistent theme that emerges from the collective of research on STEM attrition - that is, if we are to increase STEM retention, the quality of pedagogy must improve (e.g., American Association for the Advancement of Science [AAAS], 2015; Cooper et al., 2015; Dagley, et al., 2015; Seymour et al., 2019; Sithole, et al., 2017; Xu, 2016). Indeed, research tells us that students are more likely to demonstrate improved learning gains and persist in courses that use evidence-based active-engagement instructional approaches grounded in research on how students learn (e.g., Cooper et al., 2015; Freeman et al., 2014; Minner et al., 2010; NRC, 2012; Wiggins et al., 2017 Freeman et al., 2014). Model-based instruction (MBI) is an evidence-based pedagogical approach that engages students in the construction, interpretation, revision, and evaluation of scientific models (Clement, 2000; Gilbert & Justi, 2016; Justi & Gilbert, 2002a, 2002b; Long et al., 2014; Louca & Zacharia, 2012; Schwarz et al., 2009). MBI can reduce achievement gaps, particularly for 1 students traditionally underrepresented in science and those that typically underachieve on standard or rote assessments (Bierema et al., 2017; Brewe et al., 2010; Manthey & Brewe, 2013; Reinagel & Bray Speth, 2016; Verhoeff et al., 2008). My dissertation research was inspired by findings from four related MBI studies that showed prior academic achievement was a poor predictor of modeling-based performance and that there may be additional benefits for students from lower achievement groups (Bennett et al., 2020; Dauer et al., 2013; Dauer & Long, 2015; de Lima, 2020). The work of my dissertation aims to explore potential affective mechanisms that may explain differences in learning outcomes for students in an introductory biology course taught through MBI. Chapter one focuses on student motivation. Student motivation is not well understood in practice-based contexts, yet its research in these contexts, such as MBI, is valuable to informing changes to instructional approaches that can have meaningful impacts on STEM retention (National Academies of Sciences, Engineering, and Medicine [NAESM], 2018). My study applies a person-centered-approach (Bergman & Magnusson, 1997) and identifies motivational profiles (Conley, 2012; Hong et al., 2020) present among students in an introductory biology course taught through MBI and explores how those profiles change over a semester. In this study, I also examine the relationship between student achievement level and motivational profile stability or change. My second- and third-chapters center on dimensions of student engagement. Whereas motivation is comprised of private, internal processes, engagement consists of external, observable manifestations of those internal, motivational processes (Connell & Wellborn, 1991; Eccles & Wang, 2012; Finn & Zimmer, 2012; Fredricks & McColskey, 2012; Maehr & Meyer, 1997; Schunk & Mullen, 2012; Skinner et al., 2009; Wang & Degol, 2014). Specifically, I 2 examine student cognitive and emotional engagement during semi-structured in-person interviews. In Chapter Two, I focus on the development of a novel Cognitive Engagement in Modeling (CEM) framework that measures students’ use of learning strategies during model- construction tasks. Cognitive engagement is an important factor in student learning, as students who are cognitively engaged invest significant effort in understanding content and being successful on a task (Rotgans & Schmidt, 2011). The way students are cognitively engaged during practice-based tasks, such as modeling, remains less understood, however. Additionally, research remains unclear on specific learning strategies students deploy, and when, to complete practice-based tasks. My CEM framework is derived from a plethora of research on observable and linguistic indicators of learning strategies that evidence cognitive engagement (e.g., Barlow & Brown, 2019; Chi et al., 2018; Helme & Clarke, 2001) and is validated through the interview study. The CEM framework aims to fill a gap within the literature and advance research on cognitive engagement as a tool to qualitatively measure students’ cognitive engagement. Chapter three focuses on my development of an Emotional Engagement in Modeling (EEM) framework during model-based tasks. The EEM framework derives from research within Experience Sampling Methodology (ESM) (Csikszentmihalyi & Larson, 1987; Csikszentmihalyi & Csikszentmihalyi, 2006) to measure students’ emotions during a task. Although research has established emotions impact multiple components in students’ learning, such as performance outcomes, mental health, career decisions, and dropout rates (e.g., see Barroso et al., 2021 for review; Camacho-Morles et al., 2021; Cheng & McCarthy, 2018; Loukidou et al., 2009), students’ emotions remain understudied, particularly in STEM (Murphy et al., 2019). To address this gap, I created the emoji-based EEM framework to be relatable and accessible for students, 3 and easily adaptable for practitioners. I then applied the framework during interviews to evaluate and compare students’ emotional responses to model-construction and model-evaluation tasks, and examine the relationship between emotional responses and student achievement level. Collectively, my dissertation aims to serve three goals to further our understanding of how students of all achievement levels are learning in a model-based instructional context: 1) generate student motivational profiles that describe groups of students according to their combinations of motivational variables and examine motivational stability and change over a semester; 2) construct tools, specifically the CEM and EEM framework, that enable researchers to measure students’ cognitive and emotional engagement during learning tasks; and 3) apply the CEM and EEM frameworks to conduct research on students’ use of learning strategies during, and emotional responses to, model-based tasks. 4 REFERENCES The American Association for The Advancement of Science [AAAS]. (2015). Vision and Change in Undergraduate Biology Education: Chronicling change, inspiring the future. AAAS: Washington, DC. Barlow, A. J., & Brown, S. A. (2019, June). Work in progress: Measuring student cognitive engagement using the ICAP framework in and outside of the classroom. In 2019 ASEE Annual Conference & Exposition. Barroso, C., Ganley, C. M., McGraw, A. L., Geer, E. A., Hart, S. A., & Daucourt, M. C. (2021). A meta-analysis of the relation between math anxiety and math achievement. Psychological Bulletin, 147(2), 134–168. https://doi.org/10.1037/bul0000307 Bennett, S. W., Gotwals, A. W., & Long, T. M. (2020). Assessing students’ approaches to modeling in undergraduate biology. International Journal of Science Education, 42(10), 1697-1714. https://doi.org/10.1080/09500693.2020.1777343 Bergman L.R, & Magnusson D. (1997). A person-centered approach in research on developmental psychopathology. Development and Psychopathology, 9, 291–319. Bierema, A. M. -K., Schwarz, C. V., & Stoltzfus, J. R. (2017). Engaging undergraduate biology students in scientific modeling: analysis of group interactions, sense-making, and justification. CBE - Life Sciences Education, 16(68), 1-16. https://doi.org/10.1187/cbe.17-01-0023 Brewe, E., Sawtelle, V., Kramer, L. H., O’Brien, G. E., Rodriguez, I, & Pamelá, P. (2010). Toward equity through participation in modeling instruction in introductory university physics. Physical Review ST Physics Education Research, 6(1). https://doi.org/10.1103/PhysRevSTPER.6.010106 Camacho-Morles, J., Slemp, G. R., Pekrun, R., Loderer, K., Hou, H., & Oades, L. G. (2021). Activity achievement emotions and academic performance: A meta-analysis. Educational Psychology Review, 33(3), 1051–1095. https://doi.org/10.1007/s10648-020-09585-3 Chen, X. (2015). STEM attrition among high-performing college students in the United States: Scope and potential causes. Journal of Technology and Science Education, 5(1), 41–59. Cheng, B. H., & McCarthy, J. M. (2018). Understanding the dark and bright sides of anxiety: A theory of workplace anxiety. Journal of Applied Psychology, 103(5), 537–560. https://doi.org/10.1037/apl0000266 Chi, M. T. H., Adams, J., Bogusch, E. B., Bruchok, C., Kang, S., Lancaster, M., Levy, R., Li, N., McEldoon, K. L., Stump, G. S., Wylie, R., Xu, D., & Yaghmourian, D. L. (2018). Translating the ICAP theory of cognitive engagement into practice. Cognitive Science, 42, 1777-1832. https://doi.org/10.1111/cogs.12626 5 Clement, J. (2000). Model based learning as a key research area for science education. International Journal of Science Education, 22(9), 1041-1053. https://doi.org/10.1080/095006900416901 Conley, A. M. (2012). Patterns of motivation beliefs: Combining achievement goal and expectancy-value perspectives. Journal of Educational Psychology, 104(1), 32-47. Connell, J. P., & Wellborn, J. G. (1991). Competence, autonomy, and relatedness: A motivational analysis of self-system processes. In M. R. Funnar & L. A. Sroufe (Eds.), Self-processes and development: Minnesota symposium on child psychology (Vol. 23, pp. 43-77). Chicago: University of Chicago Press. Cooper, M. M., Caballero, M. D., Ebert-May, D., Fata-Hatrley, C. L. Jardeleza, S. E., Krajcik, J. S., Laverty, J. T., Matz, R. L., Posey, L. A., & Underwood, S. M. (2015). Challenge faculty to transform STEM learning: Focus on core ideas, crosscutting concepts and scientific practices. Science, 350(6258), 281-282. Csikszentmihalyi, M., & Larson, R. (1987). Validity and reliability of the experience-sampling method. Journal of Nervous and Mental Disease, 175(9), 526–536. https://doi.org/10.1097/00005053-198709000-00004 Csikszentmihalyi, M., & Csikszentmihalyi, I. S. (Eds.). (2006). A life worth living: Contributions to positive psychology. Oxford University Press. Dagley, M, Georgiopoulos, M., Reece, A., Young, C. 2015. Increasing retention and graduation rates through a STEM learning community. Journal of College Student Retention: Research, Theory & Practice, 18(2), 167-182. https://doi.org/10.1177/1521025115584746 Dauer, J. T., Momsen, J. L., Bray Speth, E., Mokohon-Moore, S. C., & Long, T. M. (2013). Analyzing change in students’ gene-to-evolution models in college-level introductory biology. Journal of Research in Science Teaching, 50(6), 639-659. https://doi.org/10.1002/tea.21094 de Lima, J. (2020). Contextual Influences on Undergraduate Biology Students’ Reasoning and Representations of Evolutionary Concepts. [Doctoral dissertation]. Michigan State University. Eccles, J., & Wang, M. T. (2012). Part I commentary: So what is student engagement anyway? In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 133-145). New York, NY: Springer. Finn, J. D., & Zimmer, K. S. (2012). Student engagement: What is it? Why does it matter? In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 97-131). New York, NY: Springer. 6 Fredricks, J. A., & McColskey, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 763-782). New York, NY: Springer. Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoafor, N., Jordt H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering and math. Proceedings of the National Academies of Science., 111(23), 8410-8415. https://doi.org/10.1073/pnas.131903011 Gilbert, J. K., & Justi, R. (2016). Modelling-based teaching in science education (Vol. 9). Basel, Switzerland: Springer international publishing. http://dx.doi.org/10.1007/978-94-010-0876-1 Helme, S., & Clarke, D. (2001). Identifying cognitive engagement in the mathematics classroom. Mathematics Education Research Journal, 13(2), 133-153. https://doi.org/10.1007/BF03217103 Hong, W., Bernacki, M. L., & Perera, H. N. (2020). A latent profile analysis of undergraduates' achievement motivations and metacognitive behaviors, and their relations to achievement in science. Journal of Educational Psychology, 112, 1409–1430. https://doi.org/10.1037/edu0000445 Hunter, A-B. (2019). Why undergraduates leave stem majors: changes over the last two decades. In E. Seymour & A-B Hunter (Eds.) Talking about leaving revisited: persistence, relocation and loss in undergraduate stem education, pp. 87-114. https://doi.org/10.1007/978-3-030-25304-2_3 Justi, R. S., & Gilbert, J. K. (2002a). Science teachers’ knowledge about and attitudes toward the use of models and modelling in learning science. International Journal of Science Education, 24(12), 1273-1292. https://doi.org/10.1080/09500690210163198 Justi, R. S., & Gilbert, J. K. (2002b). Models and modelling in chemical education. In: Gilbert, J. K., De Jong, O., Justi, R., Treagust, D. F., Van Driel, J. H. (eds). Chemical Education: Towards Research-based practice. Science & Technology Education Library (Vol. 17). Springer: Dordrecht. https://doi.org/10.1007/0-306-47977-X_3 Kennedy, B., Fry, R., & Funk, C. (April, 2021). “6 facts about America’s STEM workforce and those training for it.” PEW Research Center. Retrieved February 10, 2022 from: https://www.pewresearch.org/fact-tank/2021/04/14/6-facts-about-americas-stem- workforce-and-those-training-for-it/ Lytle, A., & De Rosa, A. J., & Fisher, F. T. (2021, November), A Review of Psychosocial Factors Associated with Undergraduate Engagement and Retention in STEM. Paper presented at 2021 Fall ASEE Middle Atlantic Section Meeting, Virtually Hosted by the 7 section. Retrieved from: https://peer.asee.org/a-review-of-psychosocial-factors-associated- with-undergraduate-engagement-and-retention-in-stem Long, T. M., Dauer, J. T., Kostelnik, K. M., Momsen, J. L., Wyse, S. A., Bray Speth, E., & Ebert-May, D. (2014). Fostering ecoliteracy through model-based instruction. Frontiers in Ecology and the Environment, 12(2), 138-139. https://doi.org/10.1890/1540-9295-12.2.138 Louca, L. T., & Zacharia, Z. C. (2012). Modeling-based learning in science education: Cognitive, metacognitive, social, material and epistemological contributions. Educational Review, 64(4), 471-492. https://doi.org/10.1080/00131911.2011.628748 Loukidou, L., Loan-Clarke, J., & Daniels, K. (2009). Boredom in the workplace: More than monotonous tasks. International Journal of Management Reviews, 11(4), 381–405. https://doi.org/10.1111/j.1468-2370 .2009.00267 Maehr, M. L., & Meyer, H. A. (1997). Understanding motivation and schooling: Where we've been, where we are, and where we need to go. Educational psychology review, 9, 371-409. https://doi.org/10.1023/A:1024750807365 Manthey, S., Brewe, E. (2013). Toward university modeling instruction- biology: adaption curricular frameworks from physics to biology. CBE- Life Sciences Education, 12, 206-214. https://doi.org/10.1187/cbe.12-08-0136 Minner, D.D.; Levy, A.J.; Century, J. (2010). Inquiry-based science instruction: what is it and does it matter? Results from a research synthesis years 1984 to 2002. Journal of Research on Science Teaching, 47(4), 474− 49. Murphy, S., Wang, C. A., & Danaia, L. (2019). Towards an understanding of STEM engagement: a review of the literature on motivation and academic emotions. Canadian Journal of Science, Mathematics and Technology Education, 19, 304-320. https://doi.org/10.1007/s42330-019-00054-w National Academies of Sciences Engineering and Medicine [NAESM]. (2018). How people learn II. Washington, DC: National Academies Press. National Research Council. (2010). Rising above the gathering storm, revisited: Rapidly approaching category 5. Washington, DC: The National Academies Press. National Research Council. (2012). Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering. Washington, DC: National Academies Press. National Science Foundation. (2012). Science and engineering indicators 2012. Washington, DC: National Science Board. 8 Olson, S. and Riordan, D.G. (2012). Engage to excel: producing one million additional college graduates with degrees in science, technology, engineering, and mathematics. Report to the president. Executive Office of the President. President’s Council of Advisors on Science and Technology [PCAST]. (2012, February). Engage to excel Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics. Washington, DC: US: Government Office of Science and Technology (Report to the President), Retrieved from: https://eric.ed.gov/?id=ED541511 Reinagel, A., Bray Speth, E. (2016). Beyond the central-dogma: model-based learning of how genes determine phenotype. CBE- Life Sciences Education, 15(1), 1-13. https://doi.org/10.1187/cbe.15-04-0105 Rotgans, J. I., & Schmidt, H. G. (2011). Cognitive engagement in the problem-based learning classroom. Advances in Health Sciences Education, 16(4), 465-479. https://doi.org/10.1007/s10459-011-9272-9 Schunk, D. H., & Mullen, C. A. (2012). Self-efficacy as an engaged learner. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 219-235). New York, NY: Springer. Schwarz, C. V., Reiser, B. J., Davis, E. A., Kenyon, L., Archér, A., Fortus, D., Shwartz, Y., Hug, B., & Krajcik, J. (2009). Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners. Journal of Research in Science Teaching, 46(6), 632-654. https://doi.org/10.1002/tea.20311 Seymour, E., Hunter, AB., Weston, T.J. (2019). Why We Are Still Talking About Leaving. In: Seymour, E., Hunter, AB. (Eds) Talking about Leaving Revisited. Springer, Cham. https://doi.org/10.1007/978-3-030-25304-2_1 Skinner, E. A., Kindermann, T. A., Connell, J. P., & Wellborn, J. G. (2009). Engagement as an organizational construct in the dynamics of motivational development. In K. Wentzel & A. Wigfield (Eds.), Handbook of motivation at school (pp. 223-245). Malwah, NJ: Erlbaum. Sithole, A., Chiyaka, E.T., McCarthy, P., Mupinga, D. M., Bucklein, B. K., Kibrige, J. (2017). Student attraction, persistence and retention in STEM programs: successes and continuing challenges. Higher Education Studies, 7(1), 46-59. Verhoeff, R. P., Boersma, K. T., Waarlo, A., J. (2008). Systems modeling and the development of coherent understanding of cell biology. International Journal of Science Education, 30(4), 543-568. https://doi.org/10.1080/09500690701237780 9 Wang, M-T., & Degol, J. (2014). Staying engaged: Knowledge and research needs in student engagement. Child Development Perspectives, 8(3), 137-143. https://doi.org/10.1111/cdep.12073 Wiggins, B. L., Eddy, S. L., Grunspan, D. Z., & Crowe, A. J. (2017). The ICAP active learning framework predicts the learning gains observed in intensely active classroom experiences. AERA Open, 3(2). https://doi.org/10.1177/2332858417708567 Xu, Y. J. (2016). Attention to Retention: Exploring and addressing the needs of college students in STEM majors. Journal of Education and Training Studies, 4(2), 67-76. 10 CHAPTER ONE: Exploring motivational profiles in a model-based undergraduate introductory biology course INTRODUCTION The demand for Science, Technology, Engineering and Math (STEM) jobs in the United States (US) economy and continual advancement of technology in STEM fields perpetuates the need for diverse, well-prepared STEM graduates. National projections from over a decade ago suggested the need for approximately one-million more STEM professionals, equating to a 34% annual increase in the number of students receiving STEM undergraduate degrees (National Research Council [NRC], 2010; Olson & Riordan, 2012; President’s Council of Advisors on Science and Technology [PCAST], 2012). The PEW Research Center indicates that there has been a “dramatic growth” in STEM graduates from US Colleges since 2010, however, challenges still exist for the issue of diversity in STEM occupations (Kennedy et al., 2021). In “Talking about Leaving Revisited”, Seymour, Hunter, and Weston (2019) reiterate that in order for the US to build a sufficient and competent STEM workforce, we must attract and retain STEM majors through graduation. The authors claim that although there has been an increasing number of students entering STEM disciplines, including those from underrepresented groups (URMs), we continue to see alarming rates of attrition. Studies estimate that only 40-50% of students entering college intending to major in a STEM field complete a STEM degree (Chen, 2015; National Science Foundation [NSF], 2012; Pedraza & Chen, 2022). Growth in Gateway Courses “Gateway courses'' are defined as foundational courses required for completion of a degree and typically taken during the first two years of college (Atanda, 1999). Successful completion of these courses is a strong predictor of persistence to graduation in STEM majors 11 (e.g., Flanders, 2017; Espinoza & Genna, 2021; Weston et al., 2019), but negative experiences in gateway courses may prevent graduation entirely (Bailey, Jeong, & Cho, 2010; Silva & White, 2013). Studies have suggested poor teaching, rigid curricula, and negative classroom climates as significant variables contributing to attrition from STEM gateway courses (e.g., Biggers, Braur, & Yilmaz, 2008; DeAngelo et al., 2011; Suresh, 2007; Weston et al., 2019). Indeed, the ‘gateway’ moniker has come to symbolize the role of these courses in filtering students such that only the highest achievers pass through to more advanced coursework. In response, much research has been directed at identifying instructional changes in gateway courses that promote persistence (e.g., Association of American Universities [AAU], 2012; Cooper et al., 2015; Freeman et al., 2014; Graham et al., 2013; Henderson, Beach & Finkelstein, 2011). Research has established that students learn more and are more likely to persist in STEM introductory courses that use evidence-based, active-engagement instructional approaches that are grounded in the research on how students learn (e.g., The American Association for the Advancement of Science [AAAS], 2015; Freeman et al., 2014; Graham et al., 2013; President’s Council of Advisors on Science and Technology [PCAST], 2012; Seymour et al., 2019; Sithole, et al., 2017; Xu, 2016). Therefore, several national reports have stressed the importance of teaching STEM introductory courses using evidence-based instructional strategies (AAAS, 2010, 2011; President’s Council of Advisors on Science and Technology, 2012; National Academies of Sciences, Engineering, and Medicine [NAESM], 2016; NAESM, 2018). Traditional science learning environments tend to teach isolated facts around disparate concepts (Momsen et al., 2010; Freeman et al., 2014), but STEM students need to develop knowledge and skills that enable them to do more than recall factual information. Evidence- based pedagogies engage students as active participants in their own learning and provide 12 alternatives to traditional lecture and rote memorization. In STEM courses in particular, pedagogies incorporating science practices have become a focus of much of the work directed at reforming gateway courses (e.g., Laverty et al., 2016; Cooper et al., 2015; Matz et al., 2018; McDonald, 2015). Scientific practices describe behaviors that scientists engage in as they investigate and develop theories about the natural world (NRC, 2012a). Developing scientific practices at the college level can help promote student understanding of how scientific knowledge develops, increase interest, and deepen content knowledge (Brewer & Smith, 2011; Cooper et al., 2015; NRC, 2012b). Significant research supports improved learning gains in classrooms that incorporate scientific practices, such as modeling, explanation, and argumentation (Cooper et al., 2015; Freeman et al., 2014; Minner et al., 2010; NRC, 2012b; Wiggins et al., 2017). Modeling and Model-Based Instruction (MBI) Modeling is a foundational scientific practice (Gilbert, 1991; NRC, 2012a) and can be defined as the process of constructing and externalizing mental models (Jonassen & Strobel, 2006, Jonassen et al., 2005; Louca & Zacharia, 2012). Mental models are internal, cognitive interpretations that individuals use to represent relationships among various parts of the world and are used in reasoning and understanding phenomena (Buckley, 2000; Johnson-Laird, 1983; Kahn, 2011). Scientific models are externalized representations of mental models depicting a concept, process, or phenomenon that can be used to illustrate, explain, or make predictions (Harrison & Treagust, 2000). Just as they are used by scientists in practice, education researchers and science educators generally agree that engaging students in modeling-based practices is an effective way to generate, evaluate, and communicate scientific knowledge, and lends itself to both instruction and assessment (e.g., Krell et al., 2012; Long et al., 2014; Schwarz et al., 2009; 13 Wilson et al., 2020). Courses and curricula that use models and modeling as a framework or as a component of instruction are becoming more prevalent in K-12 and postsecondary education (e.g., AAAS, 2015; Achér et al., 2007; Bennett et al., 2020; Bryce et al., 2016; J. J. Clement & Rea- Ramirez., 2008; Constantinou et al., 2019; Hung, 2008; Liu & Hmelo-Silver, 2009; Long et al., 2014; NRC, 2012a; Schwarz et al., 2009; Wilson et al., 2020) Model-based instruction (MBI) engages students in iterative construction, application, and evaluation of scientific models (Aragón, Olivia, & Navarrete, 2014; Clement, 2000; Gilbert & Justi, 2016; Justi & Gilbert, 2002; Long et al., 2014; Louca & Zacharia, 2012; Namdar & Shen, 2015; Schwarz et al., 2009; Shen et al., 2014). Research on teaching and learning through MBI in science classrooms can lead to a greater understanding of unobservable phenomena in science (Kahn, 2011), promote systems thinking (e.g., Ben-Zvi Assaraf & Orion, 2005; Bergan- Roller et al., 2018; Hmelo-Silver et al., 2017; Hung, 2008; Momsen et al., 2022; Tripto et al., 2013; Wilson et al., 2020), and help students develop a deeper knowledge of core concepts and relationships within a system (e.g., Dauer, et al., 2013; Hmelo-Silver, 2007; Hmelo-Silver & Pfeffer, 2004; Jordan et.al, 2013; Long et al., 2014; Tripto, Assaraf, & Amit, 2013; Vattam et al., 2011 Schwarz, 2009; Wilson et al., 2020). Research has demonstrated the potential of MBI for reducing performance gaps and engaging students who tend to underperform on traditional assessments that require factual recall (Bierema et al., 2017; Dauer et al., 2013; Manthey & Brewe, 2013; Reinagel & Bray Speth, 2016; Verhoeff et al., 2008). How a student engages in learning through MBI is undoubtedly influenced by both extrinsic factors (e.g., classroom context, social interactions, and approachability of the instructor) and intrinsic factors (e.g., the students’ desire to understand versus their desire to perform; Buckley, 2012). For example, if students view models as products or processes to be memorized, they may be less motivated to 14 understand the represented phenomena and therefore less likely to integrate modeled concepts into their mental models (Gilbert & Boutler, 2000). However, students motivated by the desire to understand or develop expertise in the skills associated with their field may be more likely to integrate model-based information into their mental models (Buckley, 2000). Although research has identified the critical role of motivation in STEM persistence and achievement (e.g., Graham et al., 2013; NAESM, 2016, 2018), little is understood about the relationship between motivation and specific practice-based pedagogical approaches, such as MBI. Prior findings from MBI-based introductory biology courses suggest that MBI can improve outcomes for students most at risk for leaving STEM (Bennett, Gotwals, & Long, 2020; Dauer et al., 2013; Dauer & Long, 2015; de Lima & Long, 2023), but mechanisms explaining these outcomes are not well understood. In this study, we examine students’ motivation as a potential factor contributing to performance differences among students in an MBI-based introductory biology course. Motivation Motivation is generally defined as a personal and internal characteristic that activates and sustains a behavior toward a goal (Dweck, 1986; Graham & Weiner, 1996). A powerful link between motivation and learning has long been suggested (e.g., Dweck, 1986; Lepper, Greene, & Nisbett, 1973), particularly in higher education where motivation has been identified as a critical predictor of academic achievement and engagement (e.g., Lazowski & Hulleman, 2016; Robbins et al., 2004). In STEM, motivation has been identified as an important predictor of persistence and achievement generally, but less is known about its role in the specific context of gateway courses (e.g., Cromley et al., 2016; NASEM, 2016, 2017, 2018; Perez et al., 2014; Linnenbrink- Garcia et al., 2018; Robinson et al., 2019). Research that examines the influence of instructional 15 methods on student motivation could be especially valuable in informing changes to instructional approaches that have large and meaningful impacts on STEM retention (NAESM, 2018). To date, motivation research has addressed pedagogies such as web-based instruction (e.g., Joo & Choi, 2000), flipped instruction (e.g., Abeysekera & Dawson, 2015), project-based learning (e.g., Kuo, Tseng, & Yang, 2019), and game-based learning (see Byusa, Kampire, & Mwesigye, 2022 for review). In general, findings from these studies and others that have adopted evidence-based, active-learning strategies (e.g., Armbruster et al., 2009; Prince, 2004) have demonstrated an increase in student motivation and attitudes. However, motivation in MBI contexts has not been explored. Integrating two Theoretical Frameworks Modern motivation research adopts a multidimensional view that considers motivation to be a combination of internal characteristics and processes that underlie reasons for people’s actions (Pintrich, 2003). In this study, we integrate two dominant motivational theories thought to play complementary roles in predicting achievement-related outcomes: expectancy value theory and achievement goal theory (Harackiewicz & Linnenbrink, 2005; Linnenbrink-Garcia et al., 2018; Plante, O’Keefe, & Theoret, 2013). A conceptual model of the two motivational theories and the subcomponents measured in our study is shown in Figure 1.1. 16 Figure 1.1. Conceptual model showing the theoretical frameworks for motivation and their components. An asterisk (*) represents components that were explicitly measured in our survey and are included in students’ motivational profiles. 1. Expectancy-Value Theory According to contemporary expectancy-value (EV) theory (Wigfield & Cambria, 2010), two key factors influence behavior and predict achievement: (a) perceived competence (PC) is the degree to which individuals believe they will be successful if they try. Science academic perceived competence is one-way motivational researchers have conceptualized expectancy for success (Schunk & Pajares, 2005) and is defined as students’ perceptions about whether or not they will successfully learn the content and succeed at academic work in science (Schunk & Pajares, 2005; Robinson et al., 2019). (b) Task value is the degree to which one perceives a task to be enjoyable, useful, and important to their identity (Barron & Hulleman, 2015; Eccles, 2009; Eccles et al., 1983; Wigfield et al., 2016). Task value focuses on features of a task that attract an individual and maintain engagement on the task (Eccles et al., 1983). EV theory differentiates task value into three subcomponents that reflect why individuals engage in a task: (1) intrinsic value (IV) - the individual finds a particular task or domain enjoyable or interesting; (2) utility value (UV) - the task is useful to their current or future goals; and (3) attainment value (AV) - doing well on the task is important to one’s identity. 17 Research in EV theory has revealed positive relations between students’ perceived competence and task values to their academic success and persistence generally (see Trautwein et al., 2013; Wigfield, & Cambria, 2010; Wigfield & Eccles, 2000; Wigfield et al., 2009, 2016 for reviews) and in STEM domains (e.g., Acee & Weinstein, 2010; Chow et al., 2012; Cromley et al., 2016; Fong et al., 2021; Hulleman et al., 2010; Lauermann et al., 2015; Luttrell et al., 2010; NASEM, 2017; Schnettler, et al., 2020; Umarji et al., 2018; Watt et al., 2012). In general, competence beliefs have been found to be more strongly related to academic performance, whereas task values are more important in achievement-related choices, such as persistence in a major (e.g., Barron & Hulleman, 2015; Trautwein et al., 2013). However, EV research aimed at measuring these components in relation to a specific practice-based pedagogy, such as MBI, is missing. 2. Achievement Goal Orientation Theory Achievement goal (AG) theory emerged as a framework to account for students’ affect, cognition, and behavior in competence-relevant contexts (Dweck, 1986; Elliot and Church, 1997). An individual’s achievement goal orientation characterizes one’s purpose for engaging in achievement-related behaviors (e.g., Ames, 1984; Anderman and Patrick, 2012; Pastor et al., 2007; Pintrich, 2000). AG theory suggests two primary underlying goal orientations that vary as a function of how competence is defined: a mastery (M) goal focuses on acquiring new information and developing competence in a task while a performance goal focuses on demonstrating competence relative to, and outperforming, others (Ames, 1992; Dweck & Leggett, 1988; Maehr & Midgley, 1991). Elliot (1999) later distinguished performance-approach (PAP) from performance-avoidance (PAV). The approach-avoidance distinction considers whether the student prioritizes outperforming one’s peers (approach focus, or PAP) versus 18 avoiding negative outcomes and appearing incompetent (avoidance focus, or PAV) (Elliot, 2006, 2008; Elliot & McGregor, 2001). While other models of AG have been proposed, such as a four- factor model that differentiates mastery goal into mastery-approach and mastery-avoidance (Pintrich, 2000; for review, see Al-Harthy, 2016), we adopt the trichotomous model due to its acceptance and prevalence in the literature (for review, see Huang, 2016). Among the three goal orientations, M is hypothesized to be most desirable as it is associated with many positive academic characteristics, such as effort investment, perseverance, resilience, retention, transfer, and self-efficacy (e.g., Belenky & Nokes-Malach, 2012; Cerasoli & Ford, 2014; Elliot & Church, 1997; Jowkar et al., 2014). Evidence substantiates an association between M and academic achievement (e.g., Dull, Schleifer, & McMillan, 2015; Huang, 2012; Meece et al., 2006). Literature on the role of a performance goal orientation on academic achievement is mixed, however. In general, performance goals are theorized to produce undesirable outcomes, such as anxiety, challenge-avoidance, ineffective learning strategy use, and decreased academic achievement (Dweck, 1986; Linnenbrink, 2005; Maehr, 1984; Nicholls, 1984). A plethora of literature generally support this hypothesis, however, there are exceptions. Surprisingly, some research has found that performance goals have been linked more reliably with academic achievement than mastery goals (e.g., Durik, Lovejoy, & Johnson, 2009; Harackiewicz, Barron, & Elliot, 1998; Hulleman et al., 2010). Some suggest that PAP can foster a competitive-based approach to learning, (i.e., demonstrating competence by attempting to performing better on tasks than their peers) which can lead to positive learning outcomes, such as self-efficacy and self-regulation (e.g, Grant & Dweck, 2003; Hackel et al., 2016; Senko & Tropiano, 2016; see meta-analyses by Hulleman et al., 2010b; Senko & Dawson, 2017a). Avoidance orientations (i.e., PAV) are characterized by fears of failure and low self-competence, 19 which can lead to limited engagement and poor academic outcomes (e.g., Lau & Lee, 2008). Therefore, literature generally associates PAV with lower academic achievement (e.g., Cooper, 2014; Huang, 2012; Hseih et al., 2007; Ranellucci et al., 2015; Senko et al., 2011) and is considered to be the least desirable goal orientation of the three. Classroom structure and the values educators communicate about learning through their teaching practices can have an impact on the shifting or reinforcing of a learner’s goal orientation (e.g., Anderman & Midgley, 1997; Maehr & Midgley, 1996; NASEM, 2018). For example, Fortus and Touitou (2021) recently found that students adapt their individual goal orientation to align with their perception of what the environment is promoting. Specifically, they found evidence that when an environment emphasizes/de-emphasizes a particular goal, students’ can shift their personal goal orientation to align with what they perceive to be emphasized by the environment. In addition to shifting or reinforcing a learner’s goal orientation, students may adopt different simultaneous combinations of goal orientations, according to course and classroom context, which may result in creating different patterns of student engagement with science (e.g., Barron and Harackiewicz; 2001, Hulleman et al., 2010; Kubsch et al., 2022; Luo et al., 2011; Schmidt et al., 2018). Due to the understanding that motivational components can combine and interact, recent motivation research has deemed it crucial to take a more holistic approach to its measurement (e.g., Hong et al., 2020; Linnenbrink-Garcia & Patall, 2016). For example, when considering students’ expectancies for success, achievement goals, and values, Conley (2012) found student motivation is best characterized by unique patterns across its components. To capture these unique patterns across multiple motivational components, our study takes an emerging approach within motivational literature (e.g., Kubsch et al., 2023; Lazarides et al., 20 2016; Lazarides et al., 2019; Viljaranta et al., 2016) and visualizes motivational profiles consisting of students’ perceived competence, task value, and achievement goal orientation. Generating motivational profiles using a person-centered approach Although EV and AG theories are typically studied independently, each framework by itself poses limitations when studying academic motivation. An EV perspective focuses on individuals’ competence expectations and their value for an academic task or domain, but disregards how their specific achievement goals are involved in their motivation. On the other hand, an AG perspective overlooks how individuals’ task-values might be related to their achievement goals and outcomes. Research that integrates these theories could help identify conceptual overlap between frameworks and develop a more nuanced understanding of motivation (e.g., Anderman, 2020; Linnenbrink-Garcia et al., 2018; Linnenbring-Garcia & Wormington, 2019; Plante, O’Keefe, & Theoret, 2013). Indeed, it is important to explore the holistic phenomenon of student motivation since no component of motivation operates in isolation from others (Hattie et al;, 2020; Higgins, 2012; Kaplan, 2014). This is consistent with more recent research that considers student motivation as a ‘motivational system’ with multiple components that interact in complex ways and influence each other (Kanfer, 2015; Kubsch et al., 2023; Linnenbrink-Garcia & Patall, 2016; Yeager & Walton, 2011). Motivational profiles consider the co-occurrence of multiple motivational factors in an individual to generate unique patterns (Conley, 2012; Hong et al., 2020). Hong et al. (2020) agree, and argue that to work toward gaining a better understanding of student motivation, research is required that models the components simultaneously in order to account for the complexity and systemic nature of motivation. Motivational profiles derive from a systems perspective and ensure that student motivation is modeled holistically, rather than being 21 composed of separate, additive parts (e.g., Hong et al., 2020; Kubsch et al., 2023; Linnenbrink- Garcia & Patall, 2016; Magnusson, 2015; Tuominen-Soini et al., 2011; Vansteenkiste et al., 2009). Student motivational profiles have been used in various contexts, including K-12 science classrooms to observe students’ transition between profiles following an intervention (Kubsch et al., 2023), and in an undergraduate anatomy and physiology course to measure motivational influences on learning in a technology-enhanced setting (Hong et al., 2020). In our study, we generate motivational profiles to better understand students’ motivation for learning introductory biology through MBI. Person-centered analyses (Bergman & Magnusson, 1997) are growing in popularity within educational psychology for capturing complex relationships between motivational constructs (e.g., Fong et al., 2018; Kubsch, et al., 2022; Wormington & Linnenbrink-Garcia, 2017). A person-centered, exploratory approach examines within-student combinations of motivational constructs to identify subgroups of students that display similar patterns (Bergman & Magnusson, 1997; Bergman, Magnusson & Khouri, 2003; Howard & Hoffman, 2017). Unlike a variable-centered approach that investigates relationships among variables, a person-centered approach explores how variables are grouped within individuals to identify combinations of motivational constructs that define subgroups (Hayenga & Corpus, 2010). Person-centered analyses apply a motivational systems perspective to integrate constructs from multiple motivational theories to generate students’ motivational profiles (Magnusson, 2015; Hong et al., 2020; Tuominen-Soini et al., 2011). Assessing and modeling the complex interplay of motivational variables can more appropriately demonstrate the heterogeneity between and within individuals, and is important to better understand how, and under which circumstances, learning takes place. When considering 22 research on practice-based pedagogies, such as MBI, a person-centered approach to thinking about student motivation provides many advantages. For example, through this approach, we can generate a better understanding of patterns of student motivation which will inform practitioners on best ways to support students based on students’ specific needs. In the context of adaptive teaching (Corno, 2008), findings from this study can inform modification of instruction to tailor to specific desired motivational components or specific groups of students. In addition, capturing if and how motivational profiles can change over time informs researchers and practitioners of motivational development and how profiles can be malleable over time (e.g., Lazarides et al., 2018). Research Questions In this study, we take a motivational systems perspective and bridge two theoretical frameworks to generate motivational profiles for students in an MBI-based introductory biology course. Profiles are assessed early and late in the semester to determine how and whether motivation changes over time. Specifically, we ask: 1. What motivational profiles are observed in an MBI-based undergraduate introductory biology course for life science majors? 2. In what ways do motivational profiles and student profile membership change? 3. What is the relationship between achievement metrics, such as course grades and student motivational profiles at the beginning and end of a semester? METHODS Course description Data were collected from a large research university in the midwest United States in two sections of a Fall 2020 introductory biology course. The course provides instruction on genetics 23 and inheritance, evolution, and ecology and is the second of a two-course sequence required for life science majors. Students typically enroll in their sophomore year following completion of the first course based on cellular and molecular biology. The sections were taught by two different instructors but both used MBI to teach the same core learning objectives. Therefore, we combine data from both sections into one for this study. Typically, the course is taught in-person in a large lecture hall, but due to the COVID-19 pandemic, both sections were taught virtually. Each class meeting consisted of brief lectures interspersed with cooperative learning activities in break-out groups. Students worked through content-related tasks, such as building or reasoning with a model, with their assigned team. Student- or team-generated examples of work were frequently used during the course to demonstrate multiple ways of thinking and representing ideas. Additionally, the online setting provided the opportunity for students to connect with each other and with the instructional team through an ongoing chat, where students could ask questions or pose comments related to the class content. The chat was monitored by a graduate teaching assistant and two undergraduate learning assistants who responded to questions and/or interjected students’ questions during lectures to be addressed by the instructors. Participants Participants (N=265) included female (58%) and male (42%) students of predominantly homogeneous ethnicity: 75% Caucasian (non Hispanic); 9% Asian; 1% African American; 5% Hispanic; 5% Multi-race; and 5% of participants chose not to report. First-generation students (13%) were also moderately represented in the sample. 24 Procedures Students were asked to complete an online questionnaire at two time points during the semester: Time 1 (T1) was selected to be after the first exam so that students had multiple experiences with constructing, applying, and revising models; Time 2 (T2) was at the end of the course and following the final exam. Low stakes course credit was provided for participation in the surveys. Motivation Survey Our survey measures three motivational constructs derived from two motivational theories: expectancy-value theory and achievement goal theory. From (1) expectancy-value theory, we measure (a) science academic perceived competence using items from a contextualized version of the patterns of adaptive learning survey scale (PALS; Midgley et al., 2000). The contextualized version we adopt for our study has been used in previous literature situated within a science-specific context (Robinson et al., 2019). Also from expectancy-value theory, we measure (b) task value using scales by Conley (2012). Within (2) achievement-goal theory, we measure students’ goal orientation using scales from the revised PALS (Midgley et al., 2000). All survey items used a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). Like previous context-specific research (e.g., Kubsch et al., 2022), all items were worded to focus on our specific context of a biology course taught through MBI. Table 1.1 shows a few select examples of items used in our study. The full questionnaire is provided in the Appendix (Table S1.1., pg. 68). 25 Table 1.1. Example survey items. Motivation Theory Expectancy- Value Component Item (Contextualized text) Science academic perceived competence (Robinson et al., 2019) I am certain I can master the skills taught in science classes. Task value: Intrinsic value (Conley, 2012) I enjoy the scientific practice of modeling biological systems. Achievement Goal Achievement goal: Mastery approach (Midgley et al., 2000) One of my goals in class is to learn as much about biology as I can. (1) Expectancy-value theory (a) Science Academic Perceived Competence Science academic perceived competence (PC) was assessed using a contextualized version of the PALS scale (Midgley et al., 2000) used in previous science context-specific research (Robinson et al., 2019). Five science academic perceived competence items measured the degree to which a student believes they will be successful in science if they try (n = 5; Cronbach’s ɑ = 0.89/0.90 (T1/T2); e.g., “I’m certain I can master the skills taught in science class.”). (b) Task Value Intrinsic (IV), utility (UV), and attainment value (AV) were assessed through 15 items using scales by Conley (2012). Similarly to items used in the achievement goals scales, items used to measure the three subcomponents of task value were adapted to fit the course-specific, model-based tasks. For example, an item from the unedited IV scale used to assess the degree to which a student finds a task interesting or enjoyable states, “I enjoy doing science.” The contextualized version used in this study states, “I enjoy modeling biological systems.” Five items measured students’ IV (n = 5; Chronbach’s ɑ = 0.89/0.92 (T1/T2)). Five UV items 26 measured the degree to which a student found the task useful to their current or future goals (n = 5; Chronbach’s ɑ = 0.83/0.86 (T1/T2); e.g, “Modeling biological systems will be useful or me later in life.”). Five AV items assessed whether the student believed being successful on the task was important to their identity (n = 5; Chronbach’s ɑ = 0.83/0.87 (T1/T2); e.g., “Being someone who is good at modeling biological systems is important to me.”). (See Appendix, page 68, for the full list of contextualized items.) (2) Achievement goal orientation Three goal orientations, mastery (M), performance-approach (PAP), and performance- avoidance (PAV), were assessed using items from the revised PALS (Midgley et al., 2000). Items were modified to focus on the biology context. For example, an item from the unedited M scale used to measure a student’s concern for mastering content states, “It’s important to me that I learn a lot of new concepts this year.” The same item contextualized to measure students’ focus on mastering biological content states, “It’s important to me that I learn a lot of new biological concepts this year.” Five items measured M orientation (n = 5; Chronbach’s ɑ = 0.86/0.88 (T1/T2)). Five PAP items measured students’ focus on demonstrating biological competence relative to their peers (n = 5; Chronbach’s ɑ = 0.88/0.92 (T1/T2); e.g., “It’s important to me that other students in my class think I am good at biology.”). Four PAV items assessed students’ focus on avoiding appearing incompetent with regards to biological content (n = 4; Chronbach’s ɑ = 0.83/ 0.87 (T1/T2); e.g., “It is important to me that I don’t look stupid in my biology class.”). ANALYSES Little’s test verified all missing data was completely at random (MCAR; T1= X2(263)=246.05, p=0.76, T2= X2(391)=391.54, p=0.48); thus, listwise deletions were used within 27 each variable to remove incomplete surveys allowing for full information maximum likelihood (FIML). Additionally, only students that completed surveys for both T1 and T2 were included in the analysis (N=265). Detection of outliers using Mahalanobis Distance brought the total sample size to 260 students for both times. Preliminary analyses including correlations between motivational components are provided in the Appendix (Table S1.2., page 71). To address our first research question, we use a type of person-centered analysis: latent profile analysis (LPA). The goal of a LPA is to identify clusters of observations that have similar patterns of variables, known as LPA indicators (Collins & Lanza, 2010, 2013; Vermunt & Magidson, 2002; Pastor et al., 2007). For our study, these clusters represent motivational profiles. Motivational profiles are representations of students’ motivational system as a whole, and, adopting Kubsch, et al.’s (2023) approach, we consider profiles to be on a spectrum ranging from higher to lower states of motivation. Motivational profiles were generated for the two time points: at the beginning and end of the semester to observe the types of profiles generated, and whether they changed over the course of the semester. For our test, we used the tidyLPA package (Rosenberg et al., 2018) in R (R Development Core Team, 2008). Fit indices were calculated for solutions with up to six profiles as in other studies (Hong et al., 2020; Kubsch et al., 2023; Pastor et al., 2007; Tuominen-Soini et al., 2011). Akaike’s Information Criterion (AIC; Akaike, 1973; 1987), Bayesian Information Criterion (BIC; Schwarz, 1978), and results of the Bootstrap Likelihood Ratio Test (BLRT; McCutcheon, 1987; McLachlan & Peel, 2000) were calculated to determine the best fitting solution. Lower values of fit indices, such as AIC and BIC indicate better model fit. A statistically significant BLRT p- value supports a solution with k+1 profiles over k-profiles (Bauer, 2022). Another way to consider this, a nonsignificant p-value supports the k-1 profile model. Furthermore, entropy and 28 classification probability were determined for each profile. Entropy values range from zero to one and are used to determine precision in group membership classification, with a higher value indicating better classification accuracy (Celeux & Soromenho, 1996; Kaplan & Keller, 2011; Masyn, 2013; Nylund, et al., 2007). For example, an entropy value of 0.87 indicates 87% of the individual cases being classified accurately in a latent class. In general, entropy values ≥ 0.8 are considered desirable (Asparouhov & Muthén, 2014). Motivation was measured and motivational profiles were generated at the beginning and end of the semester to explore whether students remained in the same profile, or if there was a shift. To address our third research question, students were first grouped into high-(n=165), middle- (n=76), and low- (n=19) achievement groups based on overall course grade. Students characterized as high-achieving earned a 4.0 overall course grade, middle-achieving students earned between a 3.0-3.5, and low-achieving students earned a 2.5 or lower. RESULTS RQ 1: Characterizing student motivational profiles Fit indices of the LPA solutions for Time 1 (Table 2.1a) and Time 2 (Table 2.1b) with up to six profiles are provided below. BLRT values remain significant for the entire series of models at both times, which some research suggests is not entirely uncommon (Bauer, 2022). Additional theoretical and methodological considerations (e.g., Masyn, 2013; Morin & Wang, 2016), including fit indices of AIC and BIC, and entropy values suggest three profile models for Time 1 and 2 are satisfactory. Further, graphical representation of class differentiation, including BIC elbow plots (provided in the Appendix, Table S1.3., page 71), support three-profile models for each time. Important to note, is that the three profile models at each time are characterized by 29 four unique profiles. Two profiles overlap between Time 1 and Time 2, and the other two profiles are unique to either Time 1 or Time 2. Table 1.2. Latent profile analysis fit statistics up to six profiles for Time 1(a) and Time 2(b). Abbreviations: AIC, Akaike’s information criterion; BIC, Bayesian information criterion; BLRT, bootstrap likelihood ratio test. Following the LPA recommendation of three-profile models at each time point, general patterns of the motivational profiles were graphically displayed (Figure 1.2 & 1.3). Figure 1.2 shows students’ profiles with all seven variables centered around the respective scale means and 95% confidence intervals at Time 1, whereas Figure 1.3 represents Time 2. As presented in Figure 1.2, profiles had different characteristics regarding motivational variables, and profiles at each time vary. We observed qualitatively different configurations of variables, known as shape differences, between profiles. For example, some profile indicators have relatively high levels above the sample mean in one profile, while in another profile the indicators have relatively low levels below the sample mean. Furthermore, quantitative differences in configurations of 30 variables, known as level differences, are shown. For example, we observed a profile in which all indicators have a relatively high level above the sample mean. Overall, our results show similar patterns to those identified in prior motivation research (e.g., Kubsch et al., 2023; Linnenbrink-Garcia et al., 2018; Wang et al., 2016), which have been adapted to characterize our motivational profiles. Figure 1.2. Bar graph of the three student motivational profiles at Time 1. All seven motivational variables are standardized to aid in interpretation, with 95% confidence intervals represented with black vertical lines. Note: PC, Science Academic Perceived Competence; MAP, Mastery Approach Goal Orientation; PAP, Performance Approach Goal Orientation; PAV, Performance Avoidance Goal Orientation; IV, Intrinsic Value; AV, Attainment Value; UV, Utility Value. 31 Figure 1.3. Bar graph of the three student motivational profiles at Time 2. All seven motivational variables are standardized to aid in interpretation, with 95% confidence intervals represented with black vertical lines. Motivational Profiles Through the LPA, three distinct motivational profiles were identified at time 1 and time 2. Figures 1.2 and 1.3 graphically display these profiles, and visually demonstrate similarities and differences between profiles and between indicators within the profiles at the two time points. Highly motivated The highly motivated profile (time 1, n = 65, 25%; time 2, n = 152, 59%) is characterized by students having consistently higher scores for all motivational variables compared to students in other profiles. This trend is especially notable for all task value components (i.e., Intrinsic Value [IV], Attainment Value [AV], and Utility Value [UV]). The characteristics of this profile indicate that the highly motivated students are motivated by both intrinsic factors and the desire 32 to appear competent and perform well in front of their peers, and the magnitude of each variable is greater than the others. Motivated, mastery and value driven The “motivated, mastery and value driven” (time 1, n = 73, 28%) is characterized as having higher than average scores for PC, M, and all three task value components, although slightly lower than the highly motivated profile. Students in this profile are also characterized as having lower than average performance-goal orientations (PAP and PAV). A motivational profile with similar characteristics was not observed at time 2. Average Motivation At time 2, a new motivational profile emerged which is characterized by close, yet, lower-than-average levels for all seven motivational variables, thus, is labeled as average motivation (n = 37, 14%). Unmotivated and performance driven An unmotivated and performance driven motivational profile appeared at both time 1 (n = 122, 47%) and time 2 (n = 71, 27%), and reflects nearly an inverse of the motivated, mastery, and value driven profile. In this profile, performance-goal orientations are the only above- average scores, and all others were lower than average, suggesting these students are generally, strongly motivated by the desire to appear competent and outperform their peers. RQ 2: Examining change in motivational profiles, and student profile membership 2.1 Changes within motivational profiles After selecting the best-fitting, three-profile, models and characterizing motivational profiles at the beginning and end of the semester, we compared the highly motivated and unmotivated and performance driven profiles to examine changes in individual indicators. 33 Within the highly motivated profile, a notable increase is observed for M, and UV whereas both performance goal orientation scores have decreased from the beginning to the end of the semester. This suggests that highly motivated students increased in their perceived present or future value for model-based tasks, therefore, fostering a greater mastery approach to learning through this style, and were less motivated by appearing competent and outperforming their peers. Within the unmotivated and performance driven profile, there is an evident increase above the average for performance-goal orientations, and a decrease below the average (becoming closer to the average) in IV and AV. This suggests that the unmotivated and performance driven students became even more performance-goal orientated, placing even more learning emphasis on demonstrating competence and outperforming their peers, but also became characterized as finding model-based tasks more interesting, and finding that doing well on the task to be important to their identity. 2.2 Changes in student profile membership Next, we examined whether and how students were changing motivational profiles from the beginning to the end of the semester. Figure 1.4 first displays the proportion of students within each profile at time 1 and time 2. Using the matrix provided in Table 1.3, we characterized students’ motivational profile shifts as increased, making a positive shift in profiles from time 1 to time 2, maintained, remaining within the same profile, or decreased, demonstrating a negative shift in their motivational profile. The data show that the majority of students (46%) experienced an increase in their motivation. A moderate proportion of students maintained their motivational profile within highly motivated (12%) or unmotivated and performance driven (14%), and 28% of students decreased in their motivational profile. 34 T1 (%) T2 (%) Figure 1.4. Sankey plot of the overall proportion (%) of students within each profile at Time 1 (T1) and Time 2 (T2). Table 1.3. Motivational profile shift matrix and the respective proportion of students. 35 RQ3. The relationship between achievement metrics, such as grades, on whether and how students’ change motivational profiles. Finally, we looked at the relationship between final course grades on motivational profile changes. Particularly, we investigated how students within high- (n=165), middle- (n=76), and low-achievement (n=19) groups maintained or shifted in profile membership from time 1 to time 2. Figure 1.5 displays the proportion (%) of high-, middle-, and low-achieving students and their motivational profile at time 1 (T1) and time 2 (T2). The Sankey plots show some stability in profile membership (i.e., highly motivated at T1 and T2), yet also indicate shifts across all achievement groups. 36 T1 (%) High T2 (%) T1 (%) Middle T2 (%) T1 (%) Low T2 (%) Figure 1.5. Sankey plots of high-, middle-, and low-achievement groups depicting the proportion (%) of students and their profile membership at Time 1 (T1) and Time 2 (T2). The greatest proportion of students across all achievement groups experienced an increase in motivation over the course of the semester. Specifically, 40% (n=66) of the students 37 who completed the course at a high-achievement level, 55% (n= 42) of the students who finished the course at a middle-achievement level, and 68% (n=13) of the students who finished at a low- achievement level increased in their motivation. A respectable number of students in all three achievement groups remained highly motivated from the beginning to the end of the semester. Specifically, 13% (n= 22) of high-achieving students, 8% (n=6) of middle-achieving students, and 16% (n=3) of low-achieving students were categorized as highly motivated at both times. There were zero low-achieving students who remained unmotivated and performance driven, compared to 18% (n=29) of high-achieving, and 9% (n=7) of middle-achieving students. A sizable number of high- (29%; n= 48), middle-(28%; n=21), and low-achieving (16%; n=3)) students decreased motivation over the course of the semester. The majority of high- and middle- achieving students that experienced a decrease in their motivation went from being highly motivated to unmotivated and performance driven, whereas the majority of low-achieving students who decreased in their motivation from highly motivated to average motivation. DISCUSSION Our study aimed to serve multiple purposes across multiple, interconnecting areas of research. Broadly, our work fills a gap within motivational literature by specifically investigating student motivation for modeling within a biology context. We generated student motivational profiles that describe individuals according to their combinations of motivational variables in an introductory biology course taught through MBI. In doing so, this study builds upon previous research considering motivation as a system, composed of variables that interact and influence each other (e.g., Kubsch et al., 2022; Linnenbrink-Garcia & Patall, 2016; Schmidt et al., 2018), and examines if and how student motivational profiles shift over a semester (Lazarides, et al., 2019). Assessing student motivational profiles at the beginning and end of a semester contributes 38 to a budding area of research examining how motivational profiles can change and how motivational components within profiles change over time. Our research explores motivation as a potential factor contributing to previously-documented performance differences among students in an MBI-based introductory biology course. Specifically, we characterize which motivational profiles high-, middle-, and low-achieving students are identified within, and examine if, and how, these students changed their motivation over the course of the semester. RQ1. Characterizing student motivational profiles Using a person-centered analysis, LPA, we identified four distinct motivational profiles: three motivational profiles at the beginning of the semester and three profiles at the end of the semester. In the beginning of the semester, students were characterized into three distinct profiles: highly motivated; motivated, mastery and value driven; and unmotivated and performance driven. At the end of the semester, we observed changes in profile characterization, including the disappearance of the motivated, mastery and value driven profile, and emergence of a new profile. The three profiles at the end of the semester are described as: highly motivated, unmotivated and performance driven, and average motivation. Our profiles resemble others that have been identified in previous motivational profile work. For example, the highly motivated profile is strongly related to Linnenbrink-Garcia et al. 's (2018) ‘very high all’ profile within a group of college science students, in which the levels of all motivational variables are positive, and, in most instances, of greater magnitude than what is observed in any other profile. Our motivated, mastery and value driven profile also resembles the ‘intrinsic and confident’ motivational profile reported by Linnenbrink-Garcia et al. (2018), which is characterized by high levels of task value, competence beliefs, and mastery approach but low levels of performance goal orientations. 39 The average motivation profile, characterized as having close to average levels across all variables, is similar to Linnenbrink-Garcia et al.’s (2018) ‘average all’ profile among college science students, and Wang et al.’s (2016) ‘moderate motivation’ profile found among grade- school and early college students in physical education courses. In each of these profiles, all motivational variables are reported at close-to-average levels. Finally, our unmotivated and performance driven most closely resembles Kubsch, et al.’s (2022) ‘unmotivated’ profile within middle-school science students, where most of our variables are lower than average, however, our findings differ in that students in this profile endorse both performance goal orientations. Across the motivational profiles, our findings confirm a complex interplay between components of motivation, and that the relationship between components can change over the course of a semester. Across all four motivational profiles, our results show a correlation between the two performance goal orientations, PAP and PAV. This finding is supported in previous research (e.g., Bong, 2009; Murayama et al., 2011; Wilbert et al., 2012) and further suggests that these two goals can become coactivated in classroom settings (Brophy, 2005; Urdan & Mestas, 2006). Murayama et al. (2011) report similar findings and suggest it may be in part due to redundancy in item wording (i.e., response bias; Murayama et al., 2011). Literature further suggests a complex interaction between performance goal orientations and PC (Kubsch et al., 2023; Sins et al., 2008). Our findings support this previous work, by demonstrating a relationship between performance goals and PC that differs for different groups of students. Similar to prior research, our findings show a moderate correlation within the highly motivated profile, where PAV, PAP, and PC are higher than average, and within the average motivation profile, where they are all around average (D’Lima et al., 2014; Hsieh, et al., 2008; Kubsch et al., 2023). For the other two profiles, however, PAV and PAP behave inversely to PC. For example, 40 the motivated, mastery, and value driven profile is characterized with lower than average performance goals and above-average PC, while the reverse is observed in the unmotivated and performance driven profile. These mixed results are consistent with previous findings (e.g., Kubsch et al.’s, 2023; Linnenbrink-Garcia et al., 2018), and may suggest that additional investigation into the relationship between performance goal orientation and other motivational aspects is needed. Our results generally document an inverse relationship between performance goal orientations and values. Literature suggests that trends of performance goals being greater than task value components suggests students that pursue a performance approach focus on whether they will succeed or fail upon completion of the task and may not become deeply engaged with the activity (e.g., Dewey, 1913; Flum & Kaplan, 2006; Renninger & Hidi, 2002). Across all four profiles, our findings support a strong interaction between PC, M, and all three sub-components of task value (IV, UV, AV). The connection between M and PC is consistent with previous research (e.g., Dorfman & Fortus, 2019; Kubsch et al., 2023), and suggests that students with greater levels of PC believe they are capable of learning the biology content and being successful on model-based tasks, therefore they will invest more time and energy into studying and practicing with model-based skills, fostering a M. The relationship with task value is in line with previous literature that suggests students who have positive beliefs about completing a task, and believe that they will be successful, tend to have a higher sense of value for the task (e.g., Bråten, and Olaussen, 2005; Lazarides et al., 2019; Eccles & Wigfield, 2002; Hulleman et al., 2008; Linnenbrink-Garcia et al., 2018; Linnenbrink & Pintrich, 2003; Sungur & Senler, 2009). Our findings also support prior research that has found M to be correlated with sub- components of task value (Bong, 2001; DeBacker & Nelson, 1999; Hulleman et al., 2008; 41 Linnenbrink, 2005). Indeed, students that pursue a M focus on learning and improvement, and therefore engage with the task in a way where they focus on the process of learning rather than the product of task engagement (e.g., Bong, 2001; Dewey, 1913; Flum & Kaplan, 2006; Renniger & Hidi, 2002; Wigfield & Eccles, 2002). RQ2. Examining change in motivational profiles, and student profile membership 2.1 Changes within motivational profiles Highly motivated From beginning to the end of the semester, not only was a change in the profiles identified, but motivational variables changed in magnitude within profiles as well. Within the highly motivated profile, there are notable positive changes, including an increase in students’ PC, M, and UV, and decreases in performance-goal orientations. This may not only suggest that students characterized as highly motivated find the model tasks to be more useful, but that students developed in their competence and belief of success if they try. Considering that the majority of students either shifted into (46%), or remained within (12%), the highly motivated profile demonstrates an increase in a considerable number of students’ drive to comprehend the subject material and improve one’s level of understanding (i.e., increased M). Previous research has suggested that there are positive effects on fostering a mastery goal orientation when courses emphasize interest, enjoyment, and challenge, and the student views success within the course as improvement of their learning process (e.g., McGregor & Elliot, 2002; Mouratidis et al., 2018; Roeser et al., 1996; Anderman & Anderman, 1999). This introductory biology course, taught through MBI, placed much emphasis on model-based tasks, which students can consider as both challenging and as an interesting way of learning. For example, students from a different semester of the same course, taught by the same instructor 42 with the same MBI curriculum, recorded stating, “Modeling was challenging, but I could better understand the content. The models made it more applicable.”; “Modeling kept me engaged and made me think more. It helped me remember stuff better since it was more visual.”; and “Modeling affected my interest level, because drawing the models helped me see connections and get a better understanding of the content.” (Furqueron, de Lima, & Long, in prep). Within the highly motivated profile, students also demonstrated a decrease in performance-goal orientations, which can have positive learning effects for students. Specifically, this may suggest that student engagement with the model-based tasks supersedes concerns about the appearance of low ability. In other words, highly motivated students may become less concerned with how they are perceived by their peers while also less likely to avoid challenging modeling tasks to prevent failure. Unmotivated and performance driven From the beginning to the end of the course, 14% of students remained within the unmotivated and performance driven profile, while 14% experienced a negative motivational shift into this profile (6% from motivated, mastery and value driven, and 8% from highly motivated). When compared to the highly motivated profile, motivational variables within the unmotivated and performance driven profiles display nearly the opposite trends over time. For example, the most notable change over time is an increase in performance-goal orientations. From this, we speculate that, over the course, these students perceived the focus of this class as not being an improvement to their learning process or skills, but to improve their performance (Roeser et al., 1996; Anderman & Anderman, 1999), and that there has been a greater shift in avoiding failure among these students. 43 Other notable changes within this profile are positive shifts for IV and AV components of task value. While still lower than average compared to the highly motivated and average motivation profiles, scores for these two variables become less negative, suggesting students characterized as unmotivated and performance driven became more interested and found greater enjoyment in the task (IV) while also seeing a greater personal importance of doing well on the model-based tasks (AV). Positive shifts for these two value components may suggest constructive characteristics of the MBI course context. For example, students may have perceived a sense of ownership in modeling tasks, and made meaningful connections between modeling in this class and other outside contexts. 2.2 Changes in student profile membership Across previous motivational profile research, findings are mixed with regards to stability and change in profile membership over time. For example, some work has indicated that adolescent and young adult learner’s motivational profiles remain relatively stable (e.g., Alexander & Murphy, 1998; Lazarides et al., 2019; Lazarides, et al., 2016), while other work suggests students can experience varied shifts in motivational profile membership (e.g., Kubsch et al., 2023; Tuominen-Soini et al., 2011). In this study, we examined changes in student profile membership following a similar ordination of motivational profiles implemented by Kubsch et al. (2023). Specifically, we posit the most desirable profile as the highly motivated profile, followed by motivated, mastery and value driven, average motivation, and the least desirable profile as unmotivated and performance driven. While this approach is in-line with previous research, we do call into question the interpretation of our rankings; particularly, with respect to different educational contexts. For example, our highly motivated profile is defined by having consistently higher scores for all motivation components, including both performance-oriented 44 goals. Performance-oriented goals are generally associated with negative learning outcomes, such as challenge-avoidance and ineffective learning strategy use (e.g., Dweck, 1986; Linnenbrink, 2005), however, some studies have linked PAP to positive outcomes, such as fostering self-efficacy and self-regulation (e.g, Grant & Dweck, 2003; Hackel et al., 2016; Senko & Tropiano, 2016). These mixed reports may be due to application of the research in different contexts, therefore, future work unpacking the relationships between context and components of performance-oriented goals are needed. Indeed, it’s important for one to consider that what might be most desirable in one setting, may not be true for another. Our findings first and foremost demonstrate that student motivation can change over the course of a semester. Specifically, we report that the majority of students (46%) transitioned into a higher level of motivation (i.e., going from unmotivated and performance driven to highly motivated) and a moderate number of students (28%) experienced a negative shift in their motivational profile (i.e., going from highly motivated to average motivation). Our results further show that a minimal number of students showed stability in their motivation, with 12% remaining highly motivated and 14% remaining unmotivated and performance driven. Overall, our findings support prior research that suggests student motivation can change over relatively short periods of time (e.g., Kubsch et al., 2023; D’Lima et al., 2014). Results such as these press the need for future research to take a closer look at why students are transitioning between profiles in order to gain a better understanding of the mechanisms underlying motivational shifts. RQ3. Achievement group motivational profile shifts In order to promote success for all students in gateway STEM courses, it is imperative to gain knowledge about the motivational differences among subgroups of students in STEM 45 classrooms. Our study is particularly interested in whether in-class performance, defined through high-, middle- and low-achieving groups of students, correlates with motivation. Across all achievement groups, the majority of students experienced a positive motivational profile shift (see Table 1.3 in Results). Namely, our findings show that the majority of students within each achievement group transitioned into the highly motivated profile at the end of the course. To our surprise, we found that all students within the low-achievement group who began the course as unmotivated and performance driven finished the course within the highly motivated profile. This finding is most impactful, as even though these students did not finish as top performers in the course, they are among the most motivated in the course. This counters the traditionally accepted argument in motivation literature that performance is a function of motivation and that greater motivation leads to improved performance (e.g., Cromley et al., 2016; NASEM, 2016, 2017, 2018; Perez et al., 2014; Linnenbrink-Garcia et al., 2018; Robinson et al., 2019). We feel this is an especially fruitful area for future research since our findings contradict prevailing assumptions about relationships between motivation and achievement. It is unclear to what we can attribute the cause underlying our findings, but we speculate two plausible, and potentially interrelated hypotheses: (1) the course in our study emphasized practice-based assessment (particularly, MBI), frequent collaborative learning activities, and multiple low-stakes opportunities for students to practice, reflect, and improve their performances on assessments. Studies that report positive associations between motivation and performance often do not report the specific nature of assessments that comprise performance scores. However, if these studies employ more ‘traditional’ assessment strategies (i.e., multiple choice items, few high-stakes exams, little to no collaboration), it is possible that students who typically don’t perform well with traditional approaches would be less motivated to 46 learn in those contexts. MBI and science practice-based assessments were a core feature in the courses in this study and may have engaged students’ motivation by providing alternative ways for students to represent their thinking and be successful, as well as through tasks that students perceived as relevant and useful to them personally as part of their training for future careers. Thus, it stands to reason that motivation could differ between courses that differ in their instructional approaches and nature of assessment underlying measures of ‘performance’. Additional studies are therefore warranted that explore motivational profiles in other courses that also emphasize MBI and/or practice-based assessment strategies. (2) Although we did not plan for it, this study was conducted during the Covid-19 pandemic and therefore was switched to an entirely online context. Studies of U.S. undergraduates adapting to online learning during the pandemic overwhelmingly report declines in attitudes and motivational attributes (e.g., Corpus et al., 2022; Hicks et al., 2023; Kalman et al., 2020; Marler et al., 2021; Parker et al., 2021; Usher et al., 2024). We were therefore surprised to see such positive gains in motivation and speculate that, again, contextual features of the course may have contributed to outcomes. Beyond MBI and shared approaches to assessment, both courses valued student collaboration and high- frequency interactions between students and instructional team members. Students from both courses reported a strong sense of community and belief that the instructional teams cared about their well-being. Both courses also had very high rates of attendance (>90%) most days, indicating students’ desire to be present and participate. It is possible that these factors also contributed to motivational outcomes and warrant further exploration. Overall, our research documents the complex nature of motivation and indicates specifics of how student motivation can change over the course of a semester. Our study examines two sections of an introductory biology course using MBI as a practice-based pedagogy. Future 47 studies assessing motivation in other disciplines and/or within a practice-based pedagogical contexts would help inform whether the trends we observed are generalizable and linked more broadly to pedagogical approaches rather than specific features unique to our context. In addition, more frequent measures of motivation would provide valuable insights into when student motivation shifts, for whom, and in response to what classroom variables. LIMITATIONS This study was conducted with undergraduate students from two sections of an MBI- based biology course for life science majors at a majority white Midwestern R1 university. Our data were collected in Fall 2020, during which, all university courses were being taught virtually due to the COVID-19 pandemic. It is entirely conceivable that characteristics unique to this population and/or context contributed to patterns in our findings. Our study design cannot claim a causal association that links any particular variable specifically to student motivation. Rather, we show that motivation can change over a semester of instruction, and that motivational improvements can be seen even among students with lower measures of academic performance. Research that explores motivational profiles and shifts across a range of populations, pedagogical approaches, and instructional contexts is essential for examining the generalizability of our findings and for identifying the specific contextual characteristics of courses that prompt motivational shifts. Research on motivation in STEM courses has found that value (e.g., Crisp, Nora, & Taggart, 2009; Jones et al., 2010; Zusho et al., 2003), mastery goal orientations (Zusho et al., 2003), and competence beliefs (Estrada et al., 2011; Perez et al., 2014) are positively related to persistence and achievement, whereas performance-avoidance goals are associated with a greater probability of leaving a STEM major (Hernandez et al., 2013). Given the importance of such motivational factors in predicting STEM retention, a greater 48 understanding about the features of instruction that promote or degrade motivation could be a critical step in retaining a larger and more diverse population in STEM careers. CONCLUSION Our study affirms that student motivation is a complicated interplay of multiple variables that can change over the course of a single semester. A person-centered approach applies a motivational systems perspective to capture complex relationships among variables that define subgroups and integrate constructs from multiple theories to generate profiles. Our study finds three motivational profiles that best characterize student motivation at the beginning of the semester: highly motivated; motivated, mastery and value driven; and unmotivated and performance driven. Two of these profiles were also present at the end of the semester: highly motivated and unmotivated and performance driven profiles; but a new, average motivation profile emerged and motivated, mastery and value driven was no longer present. Across high-, middle-, and low-achievement groups, we found that the greatest proportion of students increased in their motivation over the semester. This finding suggests that motivational profiles are not stable, and that grades are a poor predictor of trajectories of change. This research was conducted within the context of an introductory biology course taught through MBI. It is therefore imperative to test whether our findings are transferable to other science contexts using MBI or other evidence-based pedagogies. Motivation to learn in STEM courses, regardless of academic achievement, is a key factor related to student retention in STEM (NAESM, 2018). Previous research has shown that MBI can improve outcomes for students most at risk for leaving STEM ((Bierema et al., 2017; Dauer et al., 2013; Manthey & Brewe, 2013; Reinagel & Bray Speth, 2016; Verhoeff et al., 2008). Our findings may provide additional support for this by suggesting a mechanism for MBI 49 outcomes mediated through motivation, particularly for lower-achieving students. However, additional research will be necessary to establish a causal link between MBI and motivation and to test its relationship in other class settings. We re-emphasize that our study was conducted during the COVID-19 pandemic and in the context of online learning. The specific impacts of the class setting (i.e., online or in-person) on our findings remains unknown. Finally, we recall that this study was inspired by findings that show motivation as a predictor of long-term STEM outcomes. Unfortunately, measuring these are beyond the scope of this study, but more research is needed that can illuminate motivational shifts as students move through a curriculum and determine whether positive impacts on motivation ultimately translate into higher levels of STEM retention and degree completion. 50 REFERENCES Acee, T. W., & Weinstein, C. E. (2010). Effects of a value-reappraisal intervention on statistics students' motivation and performance. Journal of Experimental Education, 78(4), 487–512. https://doi.org/10.1080/00220970903352753 Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In: B. N. Petrov, F. Csaki (Eds.), Second international symposium on information theory. Budapest, Hungary: Akademiai Kiado. pp. 267–281. Akaike, H. (1987). Factor analysis and AIC. Psychometrika, 52(3), 317–332. Alexander, P. A., & Murphy, P. K. (1998). Profiling the differences in students’ knowledge, interest and strategic processing. Journal of Educational Psychology, 90(3), 435-447. https://doi.org/10.1037/0022-0663.90.3.435 Al-Harthy, I. (2016). Contemporary motivation learning theories: A review. International Journal of Learning Management Systems, 4(2), 1-6. The American Association for The Advancement of Science (AAAS). (2015). Vision and Change in Undergraduate Biology Education: Chronicling change, inspiring the future. AAAS: Washington, DC. Anderman, L. H., & Anderman, E. M. (1999). Social predictors of change in students’ achievement goal orientations. Contemporary Educational Psychology, 24, 21-37. Anderman, E. M. (2020). Achievement motivation theory: Balancing precision and utility. Contemporary Educational Psychology, 61, 1–7. https://doi.org/10.1016/j.cedpsych.2020.101864 Anderman, E. M., & Midgley, C. (1997). Changes in goal orientations, perceived academic competence, and grades across the transition to middle-level schools. Contemporary Educational Psychology, 22(3), 269-298. https://doi.org/10.1006/ceps.1996.0926 Anderman, E. M., & Patrick, H. (2012). Achievement goal theory, conceptualization of ability/intelligence, and classroom climate. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 173–191). Springer Science + Business Media. https://doi.org/10.1007/978-1-4614-2018-7_8 Aragón, M. D. M., Oliva, J. M., & Navarrete, A. (2014). Contributions of learning through analogies to the construction of secondary education pupils’ verbal discourse about chemical change. International Journal of Science Education, 36(12), 1960-1984. https://doi.org/10.1080/09500693.2014.887237 Armbruster, P., Patal., M., Johnson, E., & Weiss, M. (2009). Active learning and student- centered pedagogy improve student attitudes and performance in introductory biology. CBE-Life Sciences Education, 8, (203-213). 51 Association of American Universities (AAU). (2013). Framework for systemic change in undergraduate STEM teaching and learning. AAU Undergraduate STEM Education Initiative. Available at: http://www.aau.edu/WorkArea/DownloadAsset.aspx?id=14357 Last accessed 04/15/2021. Asparouhov, T. & Muthén, B. (2014). Auxiliary variables in mixture modeling: Three-step approaches using Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 22(2), 169-177. https://doi.org/10.1080/10705511.2014.935844 Atanda, R. (1999). Do gatekeeper courses expand educational options? Educational Statistics Quarterly, 1(1), 33–38. Bailey, T., Jeong, D. W., & Cho, S. W. (2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29(2), 255–270. Barron, K. E., & Harackiewicz, J. M. (2001). Achievement goals and optimal motivation: Testing multiple goal models. Journal of Personality and Social Psychology, 80(5), 706-722. Barron, K. E., & Hulleman, C.S. (2015). Expectancy-value-cost model of motivation. In J. S. Eccles, K. Salmelo-Aro (Eds.), International encyclopedia of social and behavioral sciences (2nd ed.), Motivational Psychology. Amsterdam, Netherlands: Elsevier. (pp. 261-271). Bauer, J. (2022). A primer to latent profile and latent class analysis. In, M. Goller, E. Kyndt, Paloniemi, S., & Damş (Eds.), Methods for Researching Professional Learning and Development: Challenges, Applications and Empirical Illustrations. Springer. (pp. 243-268). Belenky, D. M., & Nokes-Malach, T. J. (2012). Motivation and transfer: The role of mastery-approach goals in preparation for future learning. Journal of the Learning Sciences, 21(3), 399–432. Bennett, S. W., Gotwals, A. W., & Long, T. M. (2020). Assessing students’ approaches to modeling in undergraduate biology. International Journal of Science Education, 42(10), 1697-1714. https://doi.org/10.1080/09500693.2020.1777343 Bergman L.R, & Magnusson D. (1997). A person-centered approach in research on developmental psychopathology. Development and Psychopathology, 9, 291–319. Bergman, L. R., Magnusson, D., & El-Khouri, B. M. (2003). Studying individual development in an interindividual context: A person-oriented approach. Mahwah, NJ: Lawrence Erlbaum Associates. Biggers, M., Braur, A., & Yilmaz, T. (2008, March). Student perceptions of computer science: A retention study of comparing graduating seniors vs. CS leavers. Paper presented at the 52 39th SIGCSE technical symposium on computer science education, Portland, OR. https://doi.org/ 10.1145/1352135.1352274 Bong, M. (2001). Between- and within-domain relations of academic motivation among middle and high school students: Self-efficacy, task-value, and achievement goals. Journal of Educational Psychology, 93, 23-34. Bong, M. (2009). Age-related differences in achievement goal differentiation. Journal of Educational Psychology, 101, 879-896. Brewer, C. A., and Smith, D. (Eds). (2011). Vision and change in undergraduate biology education: A call to action. Washington, DC: American Association for the Advancement of Science. Brophy, J. (2005). Goal theorists should move on from performance goals. Educational Psychologist, 40, 167-176. https://doi.org/10.1207/s15326985ep4003_3 Buckley, B. C. (2000). Interactive multimedia and model-based learning in biology. International Journal of Science Education, 22(9), 895–935. Buckley, B. C. (2012). Model-based learning. In Seel, N. M. (eds) Encyclopedia of the Sciences of Learning. Springer: Boston, MA. https://doi.org/10.1007/978-1-4419-1428-6_589 Byusa, E., Kampire, E., & Mwesigye, A. R. (2022). Game-based learning approach on students’ motivation and understanding of chemistry concepts: A systematic review of literature. Heliyon, 8(e09541). Celeux, G., & Soromenho, G. (1996). An entropy criterion for assessing the number of clusters in a mixture model. Journal of Classification, 13, 195-212. Cerasoli, C. P., & Ford, M. T. (2014). Intrinsic motivation, performance, and the mediating role of mastery goal orientation: A test of self-determination theory. The Journal of Psychology, 148(3), 267-286. https://doi.org/10.1080/00223980.2013.783778 Chen, X. (2015). STEM attrition among high-performing college students in the United States: Scope and potential causes. Journal of Technology and Science Education, 5(1), 41–59. Chingos, M. (2018). What matters most for college completion? Academic preparation is a key predictor of success. American Enterprise Institute Papers & Studies, 3A. Retrieved from: https://www.thirdway.org/report/what-matters-most-for-college-completion- academic-preparation-is-a-key-of-success Chow, A., Eccles, J. S., & Salmela-Aro, K. (2012). Task value profiles across subjects and aspirations to physical and IT-related sciences in the United States and Finland. Developmental Psychology, 48(6), 1612–1628. https://doi.org/10.1037/a0030194 53 Clement, J. (2000). Model based learning as a key research area for science education. International Journal of Science Education, 22(9), 1041-1053. https://doi.org/10.1080/095006900416901 Clement, J., & Rea-Ramirez, M. A. (2008). Model based learning and instruction in science. New York: Springer. Collins, L. M., & Lanza, S. T. (2010). Latent class and latent transition analysis. Hoboken, NJ: John Wiley & Sons. Collins, L. M., & Lanza, S. T. (2013). Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences. Hoboken, NJ: John Wiley & Sons. Constantinou, C. P., Nicolaou, C. T., & Papaevripidou, M. (2019). A Framework for Modeling-Based Learning, Teaching and Assessment. In A. Upmeier zu Belzen, J. van Driel, & D. Krüger (Eds.), Towards a Competence-Based View on Models and Modeling in Science Education. Models and Modeling in Science Education, Vol 12. Springer, Cham. Conley, A. M. (2012). Patterns of motivation beliefs: Combining achievement goal and expectancy-value perspectives. Journal of Educational Psychology, 104(1), 32-47. Cooper, A. D. (2014). “Exploring the use of non-cognitive factors in predicting college academic outcomes.” Masters Theses and Doctoral Dissertations. The University of Tennessee at Chattanooga, Chattanooga, Tennessee. https://scholar.utc.edu/theses/117 Cooper, M. M., Caballero, M, D., Ebert-May, D., Fata-Hartley, C. L., Jardeleza, S. E., Krajcik., J. S., Laverty, J. T., Matz, R. L., Posey, L. A., Underwood, S. M. (2015). Challenge faculty to transform STEM learning: Focus on core ideas, crosscutting concepts, and scientific practices. Science, 350(6258), 281-282. Corno, L. (2008). On teaching adaptively. Educational Psychologist, 43, 161-173. https://doi.org/10.1080/00461520802178466 Corpus, J. H., Robinson, K. A., & Liu, Z. (2022, March). Comparing college students’ motivation trajectories before and during COVID-19: a self-determination theory approach. In Frontiers in Education (Vol. 7, p. 848643). Frontiers Media SA. Cotner, S., Thompson, S., & Wright, R. (2017). Do biology majors really differ from non-STEM majors? CBE-Life Sciences Education, 16(3), ar48. https://doi.org/10.1187/cbe.16-11-0329 Crisp, G. Nora, A., & Taggart, A. (2009). Student characteristics, pre-college, college, and environmental factors as predictors of majoring in and earning a STEM degree: an analysis of students attending a Hispanic serving institution. American Educational Research Journal, 46, 924-942. http://dx.doi.org/10.3102/0002831209349460 54 Cromley, J. G., Perez, T., & Kaplan, A. (2016). Undergraduate STEM achievement and retention: Cognitive, motivational, and instructional factors and solutions. Policy Insights from the Behavioral and Brain Sciences, 3(1), 4–11. https://doi.org/10.1177/ 2372732215622648 Dauer, J. T., & Long, T. M. (2015). Long-term conceptual retrieval by college biology majors following model-based instruction. Journal of Research in Science Teaching, 52(8), 1188-1206. https://doi.org/10.1002/tea.21258 Dauer, J. T., Momsen, J. L., Speth, E. B., Makohon‐Moore, S. C., & Long, T. M. (2013). Analyzing change in students’ gene-to-evolution models in college-level introductory biology. Journal of Research in Science Teaching, 50(6), 639-659. https://doi.org/10.1002/tea.21094 D’Lima, G. M., Winsler, A., & Kitsantas, A. (2014). Ethnic and gender differences in first-year college students’ goal orientation, self-efficacy, and extrinsic and intrinsic motivation. The Journal of Educational Research, 107(5), 341-356. De Lima, J. & Long, T. M. (2023). Students explain evolution by natural selection differently for humans versus nonhuman animals. CBE-Life Sciences Education, 22(4), ar48. https://doi.org/10.1187/cbe.21-06-0145 Dewey, J. (1913). Interest and effort in education. Cambridge, MA: Riverside Press. Dorfman, B. S., & Fortus, D. (2019). Students’ self-efficacy for science in different school systems. Journal of Research in Science Teaching, 56(8), 1037-1059. Dull, R., Schleifer, L., & McMillan, J. (2015). Achievement goal theory: The relationship of accounting students’ goal orientations with self-efficacy, anxiety, and achievement. Accounting Education, 24(2), 152–174. Durik, A., Lovejoy, C., & Johnson, S. (2009). A longitudinal study of achievement goals for college in general: Predicting cumulative GPA and diversity in course selection. Contemporary Educational Psychology, 34(2), 113-119. https://doi.org/10.1016/j.cedpsych.2008.11.002 Dweck, C. S. (1986). Motivational processes affecting learning. The American Psychologist, 41, 1040-1048. Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Physiological review, 95(2), 256-273. https://doi.org/10.1037/0033-295X.95.2.256 Eccles, J. S. (2009). Who am I and what am I going to do with my life? Personal and collective identities as motivators of action. Educational Psychologist, 44(2), 78-89. https://doi.org/10.1080/00461520902832368 55 Eccles, J. S., Adler, T. F., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J. L., & Midgley, C. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), Achievement and achievement motives: psychological and sociological approaches. San Francisco, CA: W. H. Freeman and Company, pp. 75-146. Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual review of psychology, 53(1), 109-132. https://doi.org/10.1146/annurev.psych.53.100901.135153 Elliot, A. J. (1999). Approach and avoidance motivation and achievement goals. Educational Psychologist, 34(3), 169-189. https://doi.org/10.1207/s15326985ep3403_3 Elliot, A. J. (2006). The hierarchical model of approach-avoidance motivation. Motivation and Emotion, 30, 111-116. https://doi.org/10.1007/s11031-006-9028-7 Elliot, A. J. (2008). Approach and avoidance motivation. In A. J. (Ed.), Handbook of Approach and Avoidance Motivation (pp. 3-14). New York, NY: Psychology Press. https://doi.org/10.4324/9780203888148 Elliot, A. J., & Church, M.A. (1997). A hierarchical model of approach and avoidance achievement motivation. Journal of Personality and Social Psychology, 72(1), 218-232. https://doi.org/10.1037/0022-3514.72.1.218 Elliot, A. J., & McGregor, H.A. (2001). A 2x2 achievement goal framework. Journal of Personality and Social Psychology, 80(3), 501-519. https://doi.org/10.1037/0022-3514.80.3.501 Espinoza, P., & Genna, G. (2021). Hi, I want to talk to you about your progress: a large course intervention for at-risk college students. Journal of College Student Retention: Research, Theory & Practice. 23(1), 2-27. https://doi.org/10.1177/1521025118790054 Estrada, M., Woodcock, A., Hernandez, P. R., & Schulz, P. W. (2011). Toward a model of social influence that explains minority student integration into the scientific community. Journal of Educational Psychology, 103, 206-222. http://dx.doi.org/10.1037/a0020743 Flanders, G. R. (2017). The effect of gateway course completion on freshman college student retention. Journal of College Student Retention: Research, Theory & Practice, 19(1), 2–24. https://doi.org/10.1177/1521025115611396 Flum, H., & Kaplan, A. (2006). Exploratory orientation as an educational goal. Educational Psychologist, 41, 99-110. https://doi.org/10.1207/s15326985ep4102_3 Fong, C. J., Acee, T. W., Weinstein, C. E. (2018). A person-centered investigation of achievement motivation goals and correlates of community college student achievement and persistence. Journal of College Student Retention: Research, Theory & Practice, 20(3), 369-387. https://doi.org/10.1177/1521025116673374 Fong, C. J., Kremer, K. P., Cox, C. H. T., & Lawson, C. A. (2021). Expectancy-value profiles in math and science: A person-centered approach to cross-domain motivation with academic 56 and STEM-related outcomes. Contemporary Educational Psychology, 65, 101962. https://doi.org/10.1016/j.cedpsych.2021.101962 Fortus, D., & Touitou, I. (2021). Changes to students’ motivation to learn science. Disciplinary and Interdisciplinary Science Education Research, 3(1). https://doi.org/10.1186/s43031-020-00029-0 Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoafor, N., Jordt H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering and math. Proceedings of the National Academies of Science, 111(23), 8410-8415. https://doi.org/10.1073/pnas.131903011 Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School Engagement: Potential of the Concept, State of the Evidence. Review of Educational Research, 74(1), 59-109. https://doi.org/10.3102/00346543074001059 Gilbert, S. W. (1991). Model Building and Definition of Science. Journal of Research in Science Teaching, 28(1), 73-79. Gilbert, J. K., & Boulter, C. J. (Eds.). (2000). Developing models in science education. Dordrecht: Kluwer. Gilbert, J. K., & Justi, R. (2016). Modelling-based teaching in science education (Vol. 9). Basel, Switzerland: Springer international publishing. https://doi.org/10.1007/978-3-319-29039-3 Graham, M. J., Frederick, J., Byars-Winston, A., Hunter, A.-B., & Handelsman, J. (2013). Increasing persistence of college STEM majors. Science, 341, 1455-1456. Graham, S., & Weiner, B. (1996). Theories and principles of motivation. Handbook of Educational Psychology, 4(1), 63–84. Grant, H., & Dweck, C. S. (2003). Clarifying achievement goals and their impact. Journal of Personality and Social Psychology, 85, 541-553. https://doi.org/10.1037/0022-3514.85.3.541 Guo, J., Parker, P. D., Marsh, H. W., & Morin, A. J. S. (2015). Achievement, motivation, and educational choices: A longitudinal study of expectancy and value using a multiplicative perspective. Developmental Psychology, 51(8), 1163–1176. https://doi.org/10.1037/a0039440 Hackel, T. S., Jones, M. K., Carbonneau, K. J., & Mueller, C. E. (2016)/ Re-examining achievement goal instrumentation: Convergent validity of AGQ and PALS. Contemporary Educational Psychology, 43, 73-80. https://doi.org/10.1016/j.cedpsych. 2016.04.005 57 Harackiewicz, J. A., Barron, K. E., & Elliot, A. J. (1998). Rethinking achievement goals: when are they adaptive for college students and why? Educational Psychologist, 33(1), 1-21. https://doi.org/10.1207/s15326985ep3301_1 Harackiewicz, J. M., & Linnenbrink, E. A. (2005). Multiple achievement goals and multiple pathways for learning: The agenda and impact of Paul R. Pintrich. Educational Psychologist, 40(2), 75-84. https://doi.org/10.1207/s15326985ep4002_2 Harrison, A. G., & Treagust, D. F. (2000). A typology of school science models. International Journal of Science Education, 22(9), 1011-1026. Hattie, J. A. C., Hodis, F. A., & Kang, F. H. K. (2020). Theories of motivation: Integration and ways forward. Contemporary Educational Psychology, 61, 101865. https://doi.org/10.1016/j.cedpsych.2020.101865 Hayenga, A. O., & Corpus, J. H. (2010). Profiles of intrinsic and extrinsic motivations: A person-centered approach to motivation and achievement in middle school. Motivation and Emotion, 34(4), 371-383. https://doi.org/10.1007/s11031-010-9181-x Henderson, C., Beach, A., & Finkelstein, N. (2011). Facilitating change in undergraduate STEM instructional practices. An analytic review of literature. Wiley Online Library. https://doi. org/10.1002/tea.20439 Hernandez, P. R., Schultz, P. W., Estrada, M., Woodcock, A., & Chance, R. C. (2013). Sustaining optimal motivation: a longitudinal analysis of interventions to broaden participation of underrepresented students in STEM. Journal of Educational Psychology, 105(1). Hicks, L. J., Caron, E. E., & Smilek, D. (2021). SARS-CoV-2 and learning: The impact of a global pandemic on undergraduate learning experiences. Scholarship of Teaching and Learning in Psychology. Hong, W., Bernacki, M. L., & Perera, H. N. (2020). A latent profile analysis of undergraduates' achievement motivations and metacognitive behaviors, and their relations to achievement in science. Journal of Educational Psychology, 112, 1409–1430. https://doi.org/10.1037/edu0000445 Hsieh, P., Cho, Y., Liu, M., & Schallert, D. L. (2008). Examining the interplay between middle school student’s achievement motivation goals and self-efficacy in a technology-enhanced learning environment. American Secondary Education, 36(3), 33-50. Huang, C. (2012). Discriminant and criterion-related validity of achievement goals in predicting academic achievement: A meta-analysis. Journal of Educational Psychology, 104(1), 48-73. Huang, C. (2016). Achievement goals and self-efficacy: A meta-analysis. Educational Research Review, 19, 119-137. https://doi.org/10.1016/j.edurev.2016.07.002 58 Hughes, B. (2018). Coming out in STEM: Factors affecting retention of sexual minority STEM students. Science Advances, 4(3), 1-5. Hulleman, C. S., Durik, A. M., Schweiert, S. A., & Harackiewicz, J. M. (2008). Task values, achievement goals, and interest: and integrative analysis. Journal of Educational Psychology, 100(2), 398-416. Hulleman, C. S., Godes, O., Hendricks, B. L., & Harackiewicz, J. M. (2010a). Enhancing interest and performance with a utility value intervention. Journal of Educational Psychology, 102(4), 880–895. https://doi.org/10.1037/a0019506 Hulleman, C. S., Schrager, S. M., Bodmann, S. M., & Harackiewicz, J. M. (2010b). A meta-analytic review of achievement goal measures: Different labels for the same constructs or different constructs with similar labels? Psychological Bulletin, 136, 442-449. https://doi.org/10.1037/a0018947 Johnson-Laird, P. N. (1983). Mental models. Cambridge, MA: Harvard University Press. Jonassen, D. H., & Strobel, J. (2006). Modeling for meaningful learning. Engaged Learning with Emerging Technologies, 1-27. Jonassen, D., Strobel, J., & Gottdenker, J. (2005). Model building for conceptual change. Interactive Learning Environments, 13(1-2), 15-37 Jones, B. D., Paretti, M. C., Hein, S. F., & Knott, T. W. (2010). An analysis of motivation constructs with first-year engineering students: Relationships among expectancies, values, achievement, and career plans. Journal of Engineering Education, 99, 319-336. Jowkar, B., Kojuri, J., Kohoulat, N., & Hayat, A. A. (2014). Academic resilience in education: the role of achievement goal orientations. Journal of Advances in Medical Education and Professionalism, 2(1), 33-38. Justi, S. R., & Gilbert, K. J. (2002). Modelling teachers’ views on the nature of modelling and the implications for the education of modellers. International Journal of Science Education, 24(4), 369-387. https://doi.org/10.1080/09500690110110142 Kalman, R., Macias Esparza, M., & Weston, C. (2020). Student views of the online learning process during the COVID-19 pandemic: A comparison of upper-level and entry-level undergraduate perspectives. Journal of Chemical Education, 97(9), 3353-3357. Kanfer, R. (2015). Motivation. Wiley Encyclopedia of Management, 11, 1-8. Kaplan, A. (2014). Section commentary: Theory and research on teachers’ motivation: Mapping an emerging conceptual terrain. In P. W. Richardson, S. A. Karabenick, & H. M. G. Watt (Eds.), Teacher motivation: Theory and practice (pp. 52–66). Routledge. Kaplan, D., & Keller, B. (2011). A note on cluster effect in latent class analysis. Structural Equation Modeling, 18, 525-536. 59 Kennedy, B., Fry, R., & Funk, C. (April, 2021). “6 facts about America’s STEM workforce and those training for it.” PEW Research Center. Retrieved February 10, 2022 from: https://www.pewresearch.org/fact-tank/2021/04/14/6-facts-about-americas-stem- workforce-and-those-training-for-it/ Kubsch, M., Fortus, D., Neumann, K., Nordine, J., & Krajcik, J. (2023). The interplay between students’ motivational profiles and science learning. Journal of Research in Science Teaching, 60, 3-25. https://doi.org/10.1002/tea.21789 Lauermann, F., Chow, A., & Eccles, J. S. (2015). Differential effects of adolescents’ expectancy and value beliefs about math and english on math/science-related and human-services-related career plans. International Journal of Gender, Science and Technology, 7, 205-228. Lau, K., & Lee, J. (2008). Validation of a Chinese achievement goal orientation questionnaire. British Journal of Educational Psychology, 78(2), 331-353. https://doi.org/10.1111/j.2044-8279.2008.tb00486.x Laverty, J. T., Underwood, S. M., Matz, R. L., Posey, L. A., Carmel, J. H. Caballero, M. D., et al. (2016). Characterizing college science assessments: the three-dimensional learning assessment protocol. PLoS ONE, 11. https://doi.org/10.1371/journal.pone.0162333 Lazowski, R. A., & Hulleman, C. S. (2016). Motivation interventions in education: A meta-analytic review. Review of Educational Research, 86(2), 602–640. Lazarides, R., Dietrich, J., & Taskinen, P. H. (2019). Stability and change in students’ motivational profiles in mathematics classrooms: The role of perceived teaching. Teaching and Teacher Education, 79, 165-175. Lazarides, R., Viljaranta, J., Aunola, K., & Nurmi, J. E. (2018). Teacher ability evaluation and changes in elementary student profiles of motivation and performance in mathematics. Learning and Individual Differences, 67, 245-258. Lazarides, R., Viljaranta, J., Aunola, K., Pesu, L., & Nurmi, J. E. (2016). The role of parental expectation and students’ motivational profiles for educational aspirations. Learning and Individual Differences, 51, 29-36. Linnenbrink, E. A. (2005). The dilemma of performance-approach goals: the use of multiple goal contexts to promote students’ motivation and learning. Journal of Educational Psychology, 97(2), 197-213. Linnenbrink-Garcia, L., & Patall, E. A. (2016). Motivation. In Corno, L., & Anderman, E. M. (Eds.). Handbook of Educational Psychology (3rd ed., pp. 91–103). Routledge/Taylor & Francis Group. https//doi.org/10.1016/j.cedpsych.2007.08.001 Linnenbrink-Garcia, L., & Pintrich, P. R. (2003). The role of self-efficacy, task value, and achievement goals in predicting learning strategies, task disengagement, peer 60 relationship, and achievement outcome. Contemporary Educational Psychology, 33, 486-512. Linnenbrink-Garcia, L., & Wormington, S. V. (2019). An integrative perspective for studying motivation in relation to engagement and learning. In K. A. Renninger, & S. Hidi (Eds.), Cambridge handbooks in psychology. The Cambridge handbook of motivation and learning (pp. 739–758). Cambridge University Press. https://doi.org/ 10.1017/9781316823279.031 Linnenbrink-Garcia, L. & Wormington, S. V., Snyder, K. E., Riggsbee, J., Perez, T., Ben-Eliyahu, A., et al. (2018). Multiple pathways to success: An examination of integrative motivational profiles among upper elementary and college students. Journal of Educational Psychology, 110, 1026-1048. https://doi.org/10.1037/edu0000245 Luttrell, V. R., Callen, B. W., Allen, C. S., Wood, M. D., Deeds, D. G., & Richard, D. C. (2010). The mathematics value inventory for general education students: Development and initial validation. Educational and Psychological Measurement, 70(1), 142–160. Luo, W., Paris, S., Hogan, D., & Luo, Z. (2011). Do performance goals promote learning? A pattern analysis of Singapore students’ achievement goals. Contemporary Educational Psychology, 36(2), 165-176. https://doi.org/10.1016/j.cedpsych.2011.02.003 Maehr, M. L. (1984). Meaning and motivation: Toward a theory of personal investment. In R. E. Ames & C. Ames (Vol. Eds.), Motivation in Education: Student Motivation: Vol. 1, (pp. 115-1114). New York: Academic Press. Maehr, M. L., & Midgley, C. (1991). Enhancing student motivation: A schoolwide approach. Educational Psychologist, 26(3-4), 399-427. https://doi.org/10.1207/s15326985ep2603&4_9 Maehr, M. L., & Midgley, C. (1996). Transforming school cultures. Westview Press. Magnusson, D. (2015). Individual development from an interactional perspective: A longitudinal study. Psychology Press. https://doi.org/10.4324/9781315722221 Manthey, S., & Brewe, E. (2013). Toward university modeling instruction - Biology: Adapting curricular frameworks from physics to biology. CBE-Life Sciences Education, 12, 206– 214. Marler, E. K., Bruce, M. J., Abaoud, A., Henrichsen, C., Suksatan, W., Homvisetvongsa, S., & Matsuo, H. (2021). The impact of COVID-19 on university students’ academic motivation, social connection, and psychological well-being. Scholarship of Teaching and Learning in Psychology. Masyn, K. E. (2013). Latent class analysis and finite mixture modeling. In T. D. Little (Ed.), The Oxford handbook of quantitative methods. Oxford University Press. 61 Matz, R. L., Fata-Hartley, C. L, Posey, L. A., Laverty, J. T., Underwood, S. M., Carmel, J. H., et al. (2018). Evaluating the extent of large-scale transformation in gateway science courses. Science Advances, 4(10). https://doi.org/10.1126/sciadv.aau0554 Mayers, T., Mathis, B. J., Ho, C. K., Morikawa, K., Maki, N., & Hisatake, K. (2022). Factors affecting undergraduate medical science students’ motivation to study during the COVID-19 pandemic. Education Sciences, 12(9), 628. McCutcheon, A. L. (1987). Latent class analysis. Newbury Park, CA: Sage. McDonald, J. (2015). The next generation science standards: the impact on college science teaching. Journal of College Science Teaching, 45(1), 13-14. McGregor, H. A. & Elliot, A. J. (2002). Achievement goals as predictors of achievement-relevant processes prior to task engagement. Journal of Educational Psychology, 94, 381-395. https://doi.org/10.1037/0022-0663.94.2.381 McLachlan, G.J., Peel, D. (2000). Finite mixture models. New York: Wiley. Meece, J. L., Anderman, E. M., & Anderman, L. H. (2006). Classroom goal structure, student motivation, and academic achievement. Annual Review of Psychology, 57, 487-503. https://doi.org/10.1146/annurev.psych.56.091103.070258 Minner, D.D.; Levy, A.J.; Century, J. (2010). Inquiry-based science instruction: what is it and does it matter? Results from a research synthesis years 1984 to 2002. Journal of Research on Science Teaching, 47(4), 474− 49. Momsen, J. L., Long, T. M., Wyse, S. A., Ebert-May, D. (2010). Just the facts? Introductory undergraduate biology courses focus on low-level cognitive skills. CBE-Life Sciences Education, 9(4), 435-440. Momsen, J. L., Bray Speth, E., Wyse, S. , & Long, T. (2022). Using systems and systems thinking to unify biology education. CBE-Life Sciences Education, 21(2), 1-11. https://doi.org/10.1187/cbe.21-05-0118 Morin, A. J. S., & Wang, C. K. J. (2016). A gentle introduction to mixture modeling using physical fitness performance data. In N. Ntoumanis & N. Myers (Eds.), An introduction to intermediate and advanced statistical analyses for sport and exercise scientists. (pp. 195–220). Chichester, UK: Wiley. Morrison, C., & Williams, L. E. (1993). Minority engineering programs: A case for institutional support. NACME Research Letter, 4(1), 1–11. Mouratidis, A., Michou, A., Demicioğlu, A. N., & Sayil, M. (2018). Different goals, different pathways to success: Performance-approach goals as direct and mastery-approach goals as indirect predictors of grades in mathematics. Learning and Individual Differences, 61, 127-135. 62 Murayama, K., Elliot, A. J., & Yamagata, S. (2011). Separation of performance-approach and performance-avoidance goals. A broader analysis. Journal of Educational Psychology, 103, 238-256. National Academies of Sciences, Engineering, and Medicine. (2016). Supporting students’ college success: Assessment of intrapersonal and interpersonal competencies. Washington, DC: The National Academies Press. https://doi.org/10.17226/24697 National Academies of Sciences, Engineering, and Medicine. (2018). How people learn II: learners, contexts, and cultures. Washington, DC: The National Academies Press. National Science Foundation. (2012). Science and engineering indicators 2012. Washington, DC: National Science Board. National Research Council. (2010). Rising above the gathering storm, revisited: Rapidly approaching category 5. Washington, DC: The National Academies Press. National Research Council. (2012a). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. Washington, DC: National Academies Press. National Research Council. (2012b). Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering. Washington, DC: National Academies Press. Nicholls, J. G. (1984). Achievement motivation: Conceptions of ability, subjective experience, task choice, and performance. Psychological Review, 91, 328-346. https://doi.org/10.1037/0033-295X.91.3.328 Nylund, K. L., Asparouhov, T., & Muthén, B. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling: A Multi-disciplinary Journal, 14(4), 535–569. https://doi.org/10.1080/10705510701575396 Olson, S. and Riordan, D.G., 2012. Engage to excel: producing one million additional college graduates with degrees in science, technology, engineering, and mathematics. Report to the president. Executive Office of the President. Parker, S. W., Hansen, M. A., & Bernadowski, C. (2021). COVID-19 campus closures in the United States: American student perceptions of forced transition to remote learning. Social sciences, 10(2), 62. Pastor, D. A., Barron, K. E., Miller, B. J., & Davis, S. L. (2007). A latent profile analysis of college students’ achievement goal orientation. Educational Psychology, 32, 8-47. Pedraza, L., & Chen, R. (2022). Examining motivator factors of STEM undergraduate persistence through two-factor theory. Journal of Higher Education, 93(4), 532-558. https://doi.org/10.1080/00221546.2021.1999722 63 Perez, T., Cromley, J., & Kaplan, A. (2014). The role of identity development, values, and costs in college STEM retention. Journal of Educational Psychology, 106, 315-329. http://dx.doi.org/10.1037/a0034027 Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95(4), 667-686. Pintrich, P. R. (2000). Multiple goals, multiple pathways: The role of goal orientation in learning and achievement. Journal of Educational Psychology, 92(3), 544-555. https://doi.org/10.1037/0022-0663.92.3.544 President’s Council of Advisors on Science and Technology [PCAST]. (2012, February). Engage to excel Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics. Washington, DC: US: Government Office of Science and Technology (Report to the President), Retrieved from: https://eric.ed.gov/?id=ED541511 Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93, 223-231. R Development Core Team. (2008). R: A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.R-project.org Ranellucci, J., Hall, N. C., & Goetz, T. (2015). Achievement goals, emotions, learning, and performance: A process model. Motivation Science, 1(2), 98-120. https://doi.org/10.1037/mot0000014 Renninger, K. A., & Hidi, S. (2002). Student interest and achievement: Developmental issues raised by a case study. In A. Wigfield & J. S. Eccles (Eds.), Development of achievement motivation (pp. 173-195). San Diego, CA: Academic Press. Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130(2), 261–288. Roeser, R. W., Midgley, C., & Urdan, T. C. (1996). Perceptions of the school psychological environment and early adolescent’s psychological and behavioral functioning in school: the mediating role of goals and belonging. Journal of Educational Psychology, 88, 408-422. Rosenberg, J., Beymer, P., Anderson, D., van Lissa, C. j., & Schmidt, J. (2018). tidyLPA: An R package to easily carry out latent profile analysis (LPA) using open-source or commercial software. Journal of Open Source Software, 3(30), 978. https://doi.org/10.21105/joss.00978 64 Schmidt, J. A., Rosenberg, J. M., & Beymer, P. N. (2018). A person-in-context approach to student engagement in science: Examining learning activities and choice. Journal of Research in Science Teaching, 55(1), 19–43. https://doi.org/10.1002/tea.21409 Schnettler, T., Bobe, J., Schuenemann, A., Fries, S., Grunschel, C. (2020). Motivation and Emotion, 44, 491-507. https://doi.org/10.1007/s11031-020-09822-w Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. Schwarz, C. V., Reiser, B. J., Davis, E. A., Kenyon, L., Archér, A., Fortus, D., Shwartz, Y., Hug, B., & Krajcik, J. (2009). Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners. Journal of Research in Science Teaching, 46(6), 632-654. Senko, C., & Dawson, B. (2017a). Do normative-based and appearance-based performance goals have different effects? A meta-analysis. Journal of Educational Psychology, 109, 574-598. Senko, C., & Dawson, B. (2017b). Performance-approach goal effects depend on how they are defined: Meta-analytic evidence from multiple educational outcomes. Journal of Educational Psychology, 109(4), 574-598. Senko, C., Hulleman, C. S., & Harackiewicz, J. M. (2011). Achievement goal theory at the crossroads: Old contro-versies, current challenges, and new directions. Educational Psychologist, 46(1), 26–47. Senko, C., & Tropiano, K. (2016). Comparing three models of achievement goals: Goal orientations, goal standards, and goal complexes. Journal of Educational Psychology, 108, 1178-1192. https://doi.org/10.1037/edu0000114 Seymour, E., Hunter, A. B., Weston, T. J. (2019) Why We Are Still Talking About Leaving. In: Seymour, E., Hunter, A. B. (Eds.), Talking about Leaving Revisited. Springer, Cham. https://doi-org.proxy1.cl.msu.edu/10.1007/978-3-030-25304-2_1 Silva, E., & White, T. (2013). Pathways to improvement: Using psychological strategies to help college students master developmental math. Carnegie Foundation for the Advancement of Teaching. Retrieved from www.carnegiefoundation.org/sites/default/files/pathways_to_improvement. Sungur, S., & Senler, B. (2008). An analysis of Turkish high school students’ metacognition and motivation. Educational Research and Evaluation, 15(1), 45-62. https://doi.org/10.1080/13803610802591667 Suresh, R. (2007). The relationship between barrier courses and persistence in engineering. Journal of College Student Retention: Research, Theory, and Practice, 8(2), 215-239. 65 Trautwein, U., Nagengast, B., Marsh, H. W., Gaspard, H., Dicke, A.-L., Lüdtke, O., & Jonkmann, K. (2013). Expectancy-value theory revisited: From expectancy-value theory to expectancy-values theory? In D. M. McInerney, H. W. Marsh, R. G. Craven, & F. Guay (Eds.). International advances in self-research. Theory driving research: New wave perspectives on self-processes and human development (pp. 233–249). Charlotte, NC: IAP Information Age Publishing. Tuominen-Soini, H., Salmela-Aro, K., & Niemivirta, M. (2011). Stability and change in achievement goal orientations: A person-centered approach. Contemporary Educational Psychology, 36(2), 82–100. https://doi.org/10.1016/j.cedpsych.2010.08.002 Umarji, O., McPartlan, P., & Eccles, J. (2018). Patterns of math and english self-concepts as motivation for college major selection. Contemporary Educational Psychology, 53, 146- 158. https://doi.org/10.1016/j.cedpsych.2018.03.004 Urdan, T. C., & Mestas, M. (2006). The goals behind performance goals. Journal of Educational Psychology, 98, 354-365. Usher, E. L., Golding, J. M., Han, J., Griffiths, C. S., McGavran, M. B., Brown, C. S., & Sheehan, E. A. (2021). Psychology students’ motivation and learning in response to the shift to remote instruction during COVID-19. Scholarship of Teaching and Learning in Psychology. Vansteenkiste, M., Sierens E., Soenens B., Luyckx K., Lens W. (2009). Motivational profiles from a self-determination perspective: the quality of motivation matters. Journal of Educational Psychology, 101,671–88. Vermunt, J. K., & Magidson, J. (2002). Latent class cluster analysis. Applied Latent Class Analysis, 11(89-106), 60. Viljaranta, J., Aunola, K., & Hirvonen, R. (2016). Motivation and academic performance among first-graders: A person-oriented approach. Learning and Individual Differences, 49, 336- 372. https://doi.org/10.1016/j.lindif.2016.06.00 Watt, H. M. G., Shapka, J. D., Morris, Z. A., Durik, A. M., Keating, D. P., & Eccles, J. S. (2012). Gendered motivational processes affecting high school mathematics participation, educational aspirations, and career plans: A comparison of samples from Australia, Canada, and the United States. Developmental Psychology, 48(6), 1594–1611. https://doi.org/10.1037/a0027838 Weston, T. J., Seymour, E., Koch, A. K., & Drake, B. M. (2019). Week-out classes and their consequences. In:Seymour E., Hunter AB. (Eds.). Talking about Leaving Revisited. Springer, Cham. Pg. 197-243. https://doi-org.proxy1.cl.msu.edu/10.1007/978-3-030-25304-2_1 66 Wigfield, A., & Cambria, J. (2010). Students' achievement values, goal orientations, and interest: Definitions, development, and relations to achievement outcomes. Developmental Review, 30, 1–35. https://doi.org/10.1016/j.dr.2009.12.001 Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation. Contemporary Educational Psychology, 25(1), 68–81. https://doi.org/10.1006/ceps. 1999.1015 Wigfield, A., & Eccles, J. S. (2002). The development of competence beliefs, expectancies for success, and achievement values from childhood through adolescence. In A. Wigfield & J. Eccles (Eds.), Development of achievement motivation (pp. 91-120). San Diego, CA: Academic Press. Wigfield, A., Tonks, S., & Klauda, S. L. (2009). Expectancy-value theory. In K. R. Wenzel, & A. Wigfield (Eds.). Educational Psychology Handbook Series. Handbook of Motivation at School (pp. 55–75). New York: Routledge/Taylor & Francis Group. Wigfield, A., Tonks, S. M., & Klauda, S. L. (2016). Expectancy-value theory. In K. R. Wentzel, D.B. Miele (Eds.), Handbook of Motivation at School. New York, NY: Routledge. Wiggins, B. L., Eddy, S. L., Grunspan, D. Z., & Crowe, A. J. (2017). The ICAP active learning framework predicts the learning gains observed in intensely active classroom experiences. AERA Open, 3(2). https://doi.org/10.1177/2332858417708567 Wilbert, L., Elliot, A. J., & Murayama, K. (2012). Perceived competence moderates the relationship between performance-approach and performance-avoidance goals. Journal of Educational Psychology, 104(3), 806-819. Wilson, K. J., Long, T. M., Momsen, J. L., & Bray Speth, E. (2019). Modeling in the classroom: making relationships and systems visible. CBE-Life Sciences Education, 19(fe1), 1-5. Wormington, S. V., & Linnenbrink-Garcia, L. (2017). A new look at multiple goal pursuit: The promise of a person-centered approach. Educational Psychology Review, 29, 407-445. https://doi.org/10.1007/s10648-016-9358-2 Zusho, A., Pintrich, P. R., & Coppola, B. (2003). Skill and will: The role of motivation and cognition in the learning of college chemistry. International Journal of Science Education, 25, 1081-1094. http://dx.doi.org/10.1080/0950069032000052207 67 APPENDIX Table S1.1. Survey items within their corresponding motivational variable. Contextualized text is underlined. Item # Motivational variable Not true at all Not so true Somewhat true True Very true Science Academic Perceived Competence 9 I am certain I can master the skills taught in science classes. 26 I’m certain I can figure out how to do the most difficult class work in science. 16 18 20 I can do almost all the work in science classes if I don’t give up. Even if the work in science is hard, I can learn it. I can do even the hardest work in science if I try. Achievement Goal: Mastery 4 It’s important to me that I learn a lot of new biological concepts this year. 32 One of my goals in class is to learn as much about biology as I can. 28 One of my goals is to master a lot of new biological skills this year. 1 It’s important to me that I thoroughly understand my biology class work. 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 68 Table S1.1 (cont’d). 34 It’s important to me that I improve my biology skills this year. 1 2 3 4 5 Achievement Goal: Performance Approach 2 It’s important to me that other students in my class think I am good at my biology class work. 1 2 3 4 5 31 One of my goals is to show others that I’m good at my biology class work. 1 2 3 4 5 11 One of my goals is to show others that biology class work is easy for me. 1 2 3 4 5 7 One of my goals is to look smart in comparison to other students in my 1 2 3 4 5 biology class. 25 It’s important to me that I look smart compared to others in my biology class. 1 2 3 4 5 Achievement Goal: Performance Avoidance 13 It’s important to me that I don’t look stupid in biology class. 1 2 3 4 5 21 One of my goals is to keep others from thinking I’m not smart in biology 1 2 3 4 5 class. 17 It’s important to me that my teacher doesn’t think that I know less about biology than others in my class. 1 2 3 4 5 15 One of my goals in class is to avoid looking like I have trouble doing the 1 2 3 4 5 biology class work. Task Value: Intrinsic Value 22 I enjoy the subject of biological systems. 1 2 3 4 5 8 I enjoy the scientific practice of modeling biological systems. 1 2 3 4 5 5 Modeling biological systems is exciting to me. 3 I am fascinated by modeling biological systems. 23 I like modeling biological systems. Task Value: Attainment Value 69 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 Table S1.1 (cont’d). 29 It is important for me to be a person who reasons with a systems perspective. 1 2 3 4 5 6 Thinking with a systems perspective is an important part of who I am. 1 2 3 4 5 14 Being someone who is good at modeling biological systems is important to 1 2 3 4 5 me. 24 It is important for me to be someone who is good at modeling biological systems. 1 2 3 4 5 33 Being good at modeling biological systems is an important part of who I am. 1 2 3 4 5 Task Value: Utility Value 30 Modeling biological systems is valuable because they will help me in the 1 2 3 4 5 future. 12 Modeling biological systems will be useful for me later in life. 1 2 3 4 5 27 Modeling biological systems is practical for me to know. 1 2 3 4 5 10 Modeling biological systems helps me in my daily life outside of school. 1 2 3 4 5 19 Being good at modeling biological systems will be important for my future 1 2 3 4 5 (like when I get a job or go to graduate school). 70 Table S1.2. Correlations between all motivational variables for Time 1(a) and Time 2(b). Table S1.3. Elbow plots of Bayesian information criterion (BIC) for Time 1(a) and Time 2(b). 71 Measuring student cognitive engagement in modeling (CEM): development and application of a CHAPTER TWO: INTRODUCTION CEM framework Attracting and retaining diverse individuals in science, technology, engineering, and mathematics (STEM) has been a persistent problem for decades. In the United States (U.S.), calls for programs and practices aimed at amplifying the number and diversity of STEM professionals (President’s Council of Advisors on Science and Technology [PCAST], 2012; National Research Council [NRC], 2010) have fallen short in meeting workforce demands (e.g., Lytle et al., 2021; National Science Board [NSB], 2020; National Center for Education Statistics (NCES), 2022; Seymour et al., 2019). Despite the growing need for STEM workers, attrition, meaning switching to a non-STEM pathway or leaving college altogether, remains high among STEM undergraduates (Chen, 2015; Lytle et al., 2021; National Science Foundation [NSF], 2012). As a result, attrition and retention in STEM has been one of the most widely researched areas in higher education over the past few decades (e.g., The American Association for The Advancement of Science (AAAS), 2011; Braxton & Hirschy, 2005; Meeuwisse et al., 2010; Seymour et al., 2019; Tinto, 1975; 1993; 2006; Xu, 2016). In “Talking about Leaving Revisited”, Hunter (2019) re-implemented a survey conducted over two decades earlier (Seymour & Hewitt, 1997) aimed at measuring concerns contributing to STEM-switching. They found that strikingly similar patterns continue to persist. For over half of students surveyed, three reasons were attributed to their switching out of STEM: (1) poor quality of teaching; (2) issues with curricular design, such as content overload, pace of delivery, and poor alignment between content taught and assessed; and (3) trouble with conceptual 72 understanding (Hunter, 2019). These findings echo a persistent theme that emerges from the collective of research on STEM attrition - that is, if we are to increase STEM retention, the quality of pedagogy must improve (e.g., AAAS, 2015; Cooper et al., 2015; Dagley, et al., 2015; Seymour et al., 2019; Sithole, et al., 2017; Xu, 2016). Specifically, learning experiences must be designed to engage learners - both in terms of their interests and in ways that promote their active construction of knowledge. Unlike some educational variables (e.g., socioeconomic status), engagement, or students’ investment in their learning, can be influenced by the way we teach (Appleton et al., 2008). Indeed, The Framework for Science Education (National Research Council [NRC], 2012) is built upon the goal of integrating an understanding of big content ideas in science with engagement in practices of science. “Active engagement” is encouraged throughout the framework in multiple scientific and engineering practices. One of these practices, which is a particular focus for this study, is the development and use of models (NRC, 2012, p. 42). Scientific models can be defined as specialized representations scientists use to depict a concept, process, or natural phenomenon (Constantinou et al., 2019; Halloun, 2007; Lee et al., 2017; Osbeck & Nersessian, 2006). Scientists use models to illustrate and evaluate thinking, develop explanations, make predictions, and communicate science (Gilbert, 2004; Halloun, 2007; Long, et al., 2014; Passmore et al, 2014; Schwarz et al., 2009). Engaging students in modeling has long been advocated as a way to make teaching and learning science more consistent with the way science is practiced (e.g., AAAS, 2011; Bray Speth et al., 2014; Clement, 2000, 2008; Gilbert, 1991; Gobert & Buckley, 2000; Justi & Gilbert, 2002a, 2002b; Long et al., 2014; Schwarz et al., 2009; Wilson et al., 2020). 73 Modeling-based instruction (MBI) is an evidence-based pedagogical approach that engages students in the construction, interpretation, revision, and evaluation of scientific models (Clement, 2000; Gilbert & Justi, 2016; Justi & Gilbert, 2002a, 2002b; Long et al., 2014; Louca & Zacharia, 2012; Schwarz et al., 2009). MBI has been associated with significant gains in student understanding of unobservable phenomena in science (Kahn, 2011) and promoting more scientific habits of mind (Gilbert & Justi, 2016). Research on teaching and learning through MBI in biology has shown that building models of biological systems can promote students’ ecological literacy and system thinking skills and can help students identify concepts and relationships within a system (Dauer et al., 2013; Hmelo-Silver & Pfeffer, 2004; Hmelo-Silver et al., 2007; Jordan et.al, 2013; Long et al., 2014; Tripto et al., 2013; Vattam et al., 2011). Evidence from some studies suggest MBI may have an additional benefit in reducing achievement gaps, particularly for students traditionally underrepresented in science and those that typically underachieve on standard or rote assessments (Bierema et al., 2017; Brewe et al., 2010; Manthey & Brewe, 2013; Reinagel & Bray Speth, 2016; Verhoeff et al., 2008). Of particular interest for this research, four related MBI studies suggest that prior academic achievement is a poor predictor of modeling-based performance and there may be additional benefits for students from lower achievement groups (Bennett et al., 2020; Dauer et al., 2013; Dauer & Long, 2015; de Lima, 2020). Indeed, a better understanding of how students are learning through MBI within STEM courses could inform targeted interventions that could have a large numeric impact on increasing STEM retention rates and make progress toward fulfilling STEM workforce goals. 74 In this study, we explore cognitive engagement as a potential mechanism for explaining performance differences in MBI contexts. Additionally, this work aims to advance MBI research by moving beyond examining outcomes through simple performance measures such as grades and focusing on students’ engagement in specific learning strategies utilized by students. Multidimensional framework for academic engagement In this study, we operationalize cognitive engagement as a framework for exploring learning strategies employed during modeling-based tasks. This study builds upon work within a modeling-based introductory biology course that explored associations between MBI and motivation (Furqueron & Long, in preparation). Although motivation and engagement are used interchangeably in some literature, scholars have identified them as fundamentally different components of the learning process (Finn & Zimmer, 2012; Fredricks & McColskey, 2012; Järvelä and Renninger, 2014; The National Academies of Sciences, Engineering, and Medicine [NASEM], 2018; Martin et al., 2017). Motivation refers to the private, internal processes that explain how and why a student is involved with an academic task while engagement represents the external, observable manifestation of that motivation (Connell & Wellborn, 1991; Eccles & Wang, 2012; Fredricks & McColskey, 2012; Finn & Zimmer, 2012; Maehr & Meyer, 1997; Schunk & Mullen, 2012; Skinner et al., 2009; Wang & Degol, 2014). Although theoretically distinct, researchers broadly agree that motivation is an antecedent of engagement (Anderman & Midgley, 1997; Anderman and Patrick, 2012; Dweck, 1986; Finn & Zimmer, 2012; Martin et al., 2017; Reeve, 2013) and that engagement is a mediator that links student motivational beliefs and contextual features (i.e., nature of the learning task, environment, etc.) to learning outcomes (Anderman and Patrick, 2012; Finn & Zimmer, 2012; Wang et al., 2019). 75 Despite substantial variation in how the construct of engagement is defined and measured (see Alrashidi et al., 2016 for review), most overlap in explicitly linking student engagement with academic tasks and activities. For example, Newmann et al. (1992) define engagement as “... [a] student’s psychological (cognitive, emotional) investment in and effort (behaviors) directed toward learning, understanding, or mastering the knowledge, skills, or crafts that academic work is intended to promote” (p.12). Engagement has long been conceptualized as a multidimensional construct (Archambault & Dupéré, 2017; Fredricks et al., 2004; Patall et al., 2016). Both two- and four-dimensional models of engagement have been proposed (Finn, 1989; Skinner et al., 2009; Appleton et al., 2006; Reschly & Christenson, 2006), but the Fredricks et al. (2004) three-dimensional model has become widely adopted in studies of engagement and gained much empirical support (see Alrashidi et al., 2016 for review). In Fredricks’ (2004) model, academic engagement is conceptualized in three dimensions: cognitive, behavioral, and emotional. In this model, each dimension is recognized as being separate, yet overlapping (Bae & DeBusk-Lane, 2019; Reschly & Christenson., 2012; Fredricks et al., 2004; Wang et al., 2019). Cognitive engagement can be thought of as students’ mental investment in learning (Corno & Mandinach, 1983; Fredricks et al., 2004; Meece et al., 1988; Wehlage & Smith, 1992) and is reflected in students asking questions for clarification, persisting in difficult activities, and applying flexible approaches to problem solving (Finn & Zimmer, 2012; Fredricks et al., 2004). Behavioral engagement is defined as physical participation in learning and academic-related tasks, including displays of effort, persistence, discussion contribution, and purposely seeking out information without prompting or assistance (Buhs & Ladd, 2001; Finn, 1989; Fredricks et al., 2004; Nguyen et al., 2016). Emotional engagement concerns students’ emotional reactions, 76 including boredom, happiness, sadness, anxiety, and levels of interest related to academic tasks and settings which engage them in learning (Mih & Mih, 2013; Pekrun & Linnenbrink-Garcia, 2012). In this study, we focus on cognitive engagement and build upon existing conceptualizations of this dimension to measure student cognitive engagement in modeling- based activities. Cognitive engagement Cognitive engagement is focused on students’ psychological investment in learning, including internal efforts that promote understanding and mastering knowledge and/or skills (Cooper, 2014; Chi et al, 2018; Fredricks et al., 2004; Nguyen et al., 2016; Shernoff, 2013; Wehlage & Smith, 1992; Yazzie-Mintz & McCormick, 2012). When students are cognitively engaged, they invest significant effort in understanding a topic and succeeding on a task (Rotgans & Schmidt, 2011). Through the lens of self-regulated learning theory, cognitive engagement is a continuous cycle between strategizing about a learning task and reflecting on how best to learn and progress towards one’s learning goals (Corno & Mandinach, 1983; Greene, 2015; Richardson & Newby, 2006; Winne & Nesbit, 2010). Cognitive engagement is often conceptualized as the use of learning strategies (Corno & Mandinach, 1983; Chi et al., 2018; Greene, 2015; Greene et al., 2004; Helme & Clarke, 2001; Pintrich, 2000; Pintrich & Degroot, 1990; Winne, 2010). According to Greene (2015), cognitive engagement is the primary construct, of which, specific components include the strategies used to think about what one is learning or being asked to do, reflections about how best to proceed through the task, and the mental effort exerted to regulate the strategies. 77 Much literature exists on different categories of cognitive learning strategies and how they can be identified (e.g., Weinstein & Meyer, 1991; Weinstein, et al., 2000). For example, learning strategies can be categorized as deep and surface, with their use being identified through indicators such as organizing notes around themes, creating concept maps, asking questions (deep), or copying exact statements and memorizing (surface), (e.g., Bingham & Okagaki, 2012; Borkowski et al., 1987; Green & Miller, 1996; Deekens et al., 2018; Miller et al, 1996; Sedaghat et al., 2011). Cognitive learning strategies can also be categorized more broadly as metacognitive (e.g., Bennett et al., 2020; Kisac & Budak, 2014; Shannon, 2008; Weinstein, et al., 2000), generative (e.g., Bennett et al., 2020; Brod, 2021; Fiorella & Mayer, 2015, 2016; Wittrock, 1985) or retrieval learning strategies (e.g., Grimaldi & Karpicke, 2014; Karpicke & Grimaldi, 2012; Roediger et al., 2011). Additionally, cognitive strategy use varies by the task and individual. Cognitive engagement is often measured through surveys or self-report measures of self- regulated learning strategies (e.g., Ben-Eliyahu et al., 2018; Meece et al., 1988; Pandero, 2017). However, cognitive engagement can also be reliably recognized through specific behavioral and linguistic indicators of strategy use (e.g., Barlow & Brown, 2019; Chi et al., 2018; Helme & Clarke, 2001). In this study, we measure cognitive engagement in modeling-based tasks by observing students’ behavioral and linguistic indicators of three categories of cognitive learning strategies: metacognitive, generative, and retrieval (Fig. 2.1). 78 Figure 2.1. A multidimensional framework for engagement consists of behavioral, emotional, and cognitive dimensions. Cognitive engagement is reflected in students’ use of metacognitive, generative, and retrieval learning strategies. Metacognition Since the concept of metacognition was introduced (Flavell, 1976; 1979; Brown, 1987), many efforts have been made to organize the theory and research within the field. Researchers generally agree that metacognition has two key elements: metacognitive knowledge is the understanding and awareness of our own thinking and learning processes (Brown, 1978; Jacobs & Paris, 1987) while metacognitive regulation refers to the actual actions taken in order to facilitate learning (Sandi-Urena et al., 2011). Metacognitive knowledge is generally measured through questionnaires that assess students’ knowledge of learning strategies, how they implement them, and when and why they should be used (Stanton, et al., 2015; Stephanou & Mpiontini, 2017). Research in science education has found that students’ metacognitive knowledge contributed to meaningful understanding of biology concepts such as genetics and 79 ecology, and improved scientific inquiry skills (Eilam & Reiter, 2014; Martin et al., 2000; Zion et al., 2005). Metacognitive regulation is not a single overt behavior and can be challenging to measure (Akturk et al., 2011; Chi et al., 2018; Desoete, 2008; Fredricks et al., 2004), but external indicators, such as verbalizing internal cognitive processes, can provide evidence of metacognitive-regulation strategy use in students (e.g., Bannert & Mengelkamp, 2008; Berardi- Coletta et al., 1995; Fox et al., 2011; Georghiades, 2004; NRC, 2000). Research on metacognitive regulation in science disciplines suggests a relationship between metacognition and students’ ability to transfer scientific concepts between contexts, adapt their learning to new tasks, monitor reading of scientific texts, and show improved scientific reading comprehension (e.g., Bransford et al., 2000; Michalsky, 2013; Norris & Phillips, 2012; Palincsar & Brown, 1984; Scardamalia et al., 1984; Schoenfield, 1991; Wang and Chen, 2014; Wang & Degol, 2014). Although research recognizes the need for instruction that can help science learners develop all metacognitive abilities (e.g., Avargil et al., 2018; NRC, 2000, 2007, 2012), our study focuses specifically on metacognitive regulation strategies. Generative Learning Generative learning is defined as a cognitive process in which new information is mentally reorganized and integrated with existing knowledge; thus, enabling the learner to develop an understanding of the material and apply it in new situations (Fiorella & Mayer, 2015, 2016; Gunawan et al., 2019; Parong & Mayer, 2018; Wittrock, 1974, 1985, 1992). Generative learning strategies are grounded in the constructivist view of learning in that learning involves creating meaning from to-be-learned information by mentally reorganizing it and integrating it with existing knowledge (Fiorella & Mayer, 2016; von Glaserfeld, 1983; Wittrock, 1985). 80 Fiorella and Mayer (2015) state, “In short, generative learning is transforming incoming information (e.g., words and pictures) into usable knowledge (e.g., mental models)” (pp. 1). Generative learning strategies have been defined as activities that prompt learners to produce meaningful information that goes beyond information provided by an instructor or what is within the instructional content (Brod, 2021; Chi et al., 2018). Research in science education recognizes the importance of students being able to use science knowledge generatively in order to solve problems and construct meaningful explanations of phenomena (Duncan, 2007; NRC, 2000). Retrieval Retrieval is the cognitive strategy of remembering previously learned concepts or events (Roediger & Guynn, 1996) and considers the interaction between retrieval cues in the present with knowledge pieces of the past (e.g., Grimaldi & Karpicke, 2014; Karpicke & Grimaldi, 2012; Roediger & Karpicke, 2006; Roediger & Guynn, 1996). Retrieval includes both recognition and recall (See Moreira, et al., 2019 for review; Vorhölter et al., 2019) where recognition is an awareness triggered by an external cue that information has been seen before, and recall involves a mental search for information (e.g., Cleary, 2019; Kintsch, 1970). Both recall and recognition can be used in the measurement of cognitive engagement, as they imply active involvement in the task (Finn & Zimmer, 2012; Pintrich, 2004). Retrieval processes are used in all situations in which the learners convey knowledge. In disciplinary contexts, such as biology, students are often asked to express their knowledge through tasks that require both content and procedural knowledge - e.g., constructing a model, explaining a concept, making an inference, evaluating one’s work, and applying knowledge to a new problem. Therefore, explicit statements about recognition or recall of either content or 81 procedural knowledge can provide evidence of retrieval as students plan for, monitor, and evaluate modeling tasks. Measuring engagement Tools that measure engagement (see Fredricks et al., 2011 for review) traditionally fit into two categories. Survey instruments have been used to document an individual’s own self- reflection. For example, the Student Engagement Instrument (SEI; Appleton et al., 2006) and Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich & De Groot, 1990) have been used to measure students’ reflections on their use of self-regulatory and goal-setting strategies. Interview studies, in contrast, record objectively observed behaviors, but are less frequent in the literature. This may be due, in part, to the fact that observational studies are limited in their ability to reveal aspects of engagement that are internal and unobservable in nature (Li, 2021). Both surveys and observational techniques come with strengths and weaknesses (see Fredricks & McColsky, 2012 for in-depth discussion). Self-report measures are less likely to disrupt an individual’s normal learning process versus other methods, however, challenges include prospection (i.e., data gathered before learning events), retrospection (i.e., data gathered after learning events), and self-report bias (e.g, Veenman, 2005; Schelling & Van Hout-Wolters, 2011). In-situ observation and interview methods that capture learning as it is occurring in the classroom or in an interview setting can eliminate issues of prospection and retrospection but are not free from the possibility of classroom distractions and observer bias. Because of this, research on engagement in science learning recognizes that a combination of measures that triangulate quantitative self-report and qualitative observational data are preferred over the use of a single instrument (Fredricks & McColskey, 2012; Greene, 2015; Sinatra et al., 2015). 82 In addition to the many modes of assessment, there is considerable variation in the grain size, or level of specificity, at which engagement is conceptualized and measured (Appleton et al., 2006; Reeve, 2013; Sinatra et al., 2015). For example, at a coarse grain size, engagement can be measured in school generally through attendance or participation in extracurricular activities (e.g., Appleton et al., 2006; Furlong & Christenson, 2008). Within the classroom, engagement can be assessed at a smaller grain size through measures such as homework completion and hand-raising (Bӧheim, et al., 2020). At a fine-grained or task-based level, engagement can be inferred through use of learning strategies (e.g., Dent & Koenka, 2016), time on task (e.g., Helme & Clarke, 2001), or eye-tracking (e.g., Antonietti et al., 2015; D’Mello et al., 2017). Research Objectives Student engagement in science practices is key for fostering students’ understanding of science content, developing their science-related skills, and facilitating long-term learning. However, little is known about students’ cognitive engagement during their participation in science practices, such as modeling. This study serves two research objectives: (1) Develop a Cognitive Engagement in Modeling (CEM) framework for characterizing and measuring student cognitive engagement during modeling tasks. The CEM measures evidence of cognitive engagement through the use of metacognitive, generative, and retrieval learning strategies as students perform model-construction tasks. (2) Apply the framework to conduct original research about students’ cognitive engagement during model construction. For this, we applied the CEM to characterize cognitive engagement when students were asked to construct two types of models: a repeat model that asked students to reconstruct a model they had built previously for an exam, and a novel model that asked students to model a phenomenon based on familiar content but that was presented in a novel context. Previous research suggests that metacognitive 83 strategy-use becomes progressively more important as task complexity increases (e.g., Hattie et al., 1996; Mokos & Kafoussi, 2013). We therefore hypothesized that construction of the novel model would elicit greater levels of cognitive engagement, particularly through the use of more and/or more diverse metacognitive strategies compared to the repeat model. METHODS Course Description Interviewees (N=10) were undergraduate students at a large, Midwestern university with very high research activity (The Carnegie Classification of Institutions of Higher Education) that had successfully completed the second of a two-course introductory biology sequence required for life science majors. The first course focuses on cellular and molecular biology, whereas the second course provides instruction on genetics and inheritance, evolution, and ecology through MBI. Enrollment is open to students at any level, but the majority are in their sophomore year and all have completed at least one semester of general chemistry. Tests, homeworks, and in-class activities provided students multiple opportunities to engage in model-based learning (MBL). The course theme centered on biological variation and domain-specific concepts were introduced through this lens as three discrete but interrelated modules. Module 1 (Genetics and Inheritance) focused on the origin and expression of genetic variation, including the role of mutation and environment on gene-to-phenotype processes. Module 2 (Evolution) considered the consequences of phenotypic variation for species’ fitness and persistence in variable environments. Module 3 (Ecology) focused on the role of variation in the biotic and abiotic environments in predicting the structure and dynamics of populations, communities, and ecosystems. Collaborative modeling exercises were a central feature of in- class activities and teams provided support and feedback during modeling tasks. In-class 84 activities were typically followed by whole-class discussions, during which, subsets of students’ models were shared for peer and instructor feedback and opportunities were provided for model revision. Online rubrics were provided for higher-stakes assessments, such as homeworks and exams. Rubrics specified essential components and processes that should be represented in students’ models, but also emphasized that there was no single “correct” model and variation among models was both normal and desirable. Participants Theoretical sampling ensures that specific groups of participants who may possess certain characteristics are included in a study (Glaser & Strauss, 1967). In this case, we sought to ensure diversity in students’ academic performance (i.e., grades). Students were binned into tertiles according to their first-exam score approximately 4 weeks into the semester. From these tertiles, thirty students were recruited (10 per tertile) for in-person interviews. In-person interviews were conducted using an electronic SmartBoard that recorded students’ modeling activities and video- and audio-recorded. Eleven interviews were completed prior to the university’s shift to online instruction due to the COVID-19 pandemic. A technical problem resulted in one interview being unusable. Of the ten student participants, 8 were female, 8 white (non-Hispanic), 2 first-generation college students, and 8 sophomores (Table 2.1). Students are identified by the first letter(s) of a chosen pseudonym. Achievement levels are based on the first-exam grade, used for interview recruitment, and their grade earned in the course (used for post-interview analysis). 85 Table 2.1. Interviewee demographics. Interviewees are identified by a pseudonym. Achievement levels were determined by binning students into tertiles at two time points: first exam and final course grade. Additional demographic data, including self-identified gender, ethnicity, first-generation college student status, class rank, and declared major, were derived from university registrar data. Interview Design We used a semi-structured, think-aloud interview adapted from the 3P-SIT protocol described by Schӧnborn (2005). The interview was organized around a set of open-ended questions, specific probes intended to elicit conceptual understanding and/or interviewee perceptions or feelings, and additional questions that emerged from the dialogue between the interviewer and interviewee. Think-aloud protocols are commonly used in education and psychology research and are considered a valid tool for accessing cognitive activities (Ericsson, 2006; Ericsson & Simon, 1998). Unlike structured interviews, the semi-structured format enables researchers to ask unplanned questions that can clarify interpretation of observed behaviors and emotions and co-create understanding with interviewees regarding strategy-use during modeling tasks (Flick, 2006; Lindlof & Taylor, 2002). According to Megaldi and Berler (2020), semi- 86 structured interviews are an exploratory approach that enable the researcher to probe for deep discovery. Interview tasks and probes were designed to elicit data about cognitive, behavioral, and emotional dimensions of engagement, however, this paper describes cognitive engagement only. Interview Process Interview procedures were determined exempt by a university institutional review board (#00003353). All interviews took place in a research lab designed to facilitate in-person, clinical interview studies. The space was large, had a table with ample room and chairs, and video and audio equipment. Interviews lasted approximately 1-1.5 hours and were moderated by two interviewers. One interviewer acted as the primary discussant, the second assisted with note- taking, logistics, and occasional questioning. Our interview protocol consisted of three phases: Consent, Orientation, and Tasks. Consent & Orientation Upon student arrival, interviewers gave a brief overview of the study and purpose for the interview. Students were provided a consent form, which had previously been emailed to them for review. To protect anonymity of participants, unique pseudonyms were used to identify students during the interviews. Video and audio recording began only after students provided consent. To begin the interview, students were asked to reflect on their time and experience in the course. Questions during this phase included: How did you like the course? What was the main goal you had for yourself in the course? How would you rank your effort compared to other classes you were taking at the time? And, did you ever seek out help (during or outside of class) by asking questions to the instructor, teaching assistant, or undergraduate learning assistants? To 87 familiarize students with the technology that would be used for the interview, students were asked to make a simple drawing on the SmartBoard using the stylus and applying different pen colors, shape inserts, line sizes, etc. Modeling Construction Tasks Tasks were designed to elicit students’ thinking and modes of engagement in relation to two model construction tasks: Repeat Model (CFTR): Students were asked to construct a model in response to a prompt that was previously administered on their first exam. The prompt was designed to assess students’ understanding of information flow using the context of the genetic disease, cystic fibrosis. Specifically, students were asked to construct a model that explains how genetic variation originates at the CFTR gene and ultimately results in the expression or non-expression of the cystic fibrosis phenotype. A minimal list of potential model components was provided (e.g., gene, protein, etc.) and students were encouraged to add additional concepts as they saw fit (Appendix, page 141). Novel Model (Carbon Cycle): In this task, students were asked to construct a model that explained carbon cycling in an aquatic ecosystem. Carbon cycling was a subject in the Ecology module of the course and although students had modeled carbon cycling for a variety of systems, the context of the aquatic ecosystem was novel. Background information was provided in order to re-familiarize students with the concepts but a key components list was not initially provided. Instead, students were first prompted to identify and list concepts they believed would be needed to create a model that would describe the cycling of carbon in a simple aquatic system (Appendix, page 141). Once the student informed researchers they were done reading the background information and had 88 identified key concepts, students were then provided a list of key components and prompted to construct a model using the provided words and any other model elements necessary to explain the model function. (Appendix, page 142). In this way, we were able to first elicit students’ conceptions of key concepts, but also ensure that all students had an equivalent baseline of key concepts for the model construction task. As part of the think-aloud protocol, interviewers asked probing questions as students worked through each task to encourage discussion of strategies employed. Prompts such as, “Please keep thinking aloud for us,” and “Can you keep talking us through what you are doing?” were frequently used. Also, if a student began to erase a component of their model, interviewers asked, “Can you explain what you are doing?”, or “Why did you decide to erase/change that?”. Data indicating students’ cognitive engagement were derived from both observable behaviors (e.g., a student erasing work) and from dialogue that arose between interviewers and interviewees. Following completion of the model-building tasks, all students were probed with procedural questions to elicit understanding of cognitive strategies they employed. These questions included: • Why did you start your model with [component]? • Do you have any particular strategy or approach you use that helps you to get started? • Is there any particular approach or strategy you use to progress through the model building phase after you’ve gotten started? • What helps you put the components and relationships together? and, • How do you know when you are finished? 89 Coding Protocols Metacognition and metacognitive regulation are often measured in relation to three distinct phases associated with progression through a learning task: planning, monitoring, and evaluating (e.g., Fogarty, 1994; Jacobs & Paris, 1987; Pintrich, 2002; Sandi-Urena et al., 2011; Schraw & Moshman, 1995; Silver, 1979; Winne & Nesbit, 2010). Because this three-phase distinction used when measuring metacognition is generally applicable to the overall model- construction process, we used them to demarcate distinct model-construction phases (described below). While we acknowledge that students could conceivably iterate among phases (e.g., one might evaluate their plan before progressing through a task), we define specific start and end points for each phase for the purpose of simplifying our coding approach and guiding our analyses. (1) Planning refers to the development of a plan before approaching a learning task. These activities include predicting, brainstorming, determining time and effort allocation, strategy selection, and setting goals (Brown, 1987; Karpicke, 2009; Schraw & Moshman, 1995; Stefanou et al., 2002). Metacognitive planning strategies are essential in the problem-solving process for students to generate ideas for approaching a problem (Lesh & Zawojewski, 2007) and can improve outcomes regardless of context and content of the task (Schraw & Moshman, 1995). We define planning as the time from which the student was provided the background information for a prompt until they began the task. (2) Monitoring encompasses self-assessment during a learning situation in order for the learner to be successful on the task (Schraw & Moshman, 1995; Stanton et al., 2015). This phase specifically includes self-regulating activities concerning the need for help, error detection, and consideration of whether one’s selected strategy is working and 90 making appropriate adjustments (e.g., Carter et al., 1998; Perry, 2013; Zimmerman, 2002). Researchers are particularly interested in metacognitive monitoring because student self-awareness and subsequent application of monitoring activities can improve content understanding and problem-solving ability (Metcalfe, 2009; Schraw & Moshman, 1995; Stephanou & Mpiontini, 2017). Monitoring begins when the student starts the task and ends when they declare they are finished. (3) Evaluating refers to one’s appraisal of the results after completing the task or a component of the task (Schraw et al., 2006). Metacognitive evaluation comes in response and is complementary to the monitoring phase (Kim et al., 2013). For example, if one’s monitoring reveals a lack of progress towards a solution, evaluative processes may reveal the need to try an alternative problem-solving strategy. Tanner (2012) additionally notes that evaluation is closely related to the planning phase of metacognitive regulation because as someone evaluates their learning they may also be considering a different approach or strategy if they were to complete the task again. However, for the purposes of our study, evaluating considers the time from task completion until the interviewers finish with probing questions. Each modeling phase was independently coded by two raters for evidence of indicators of each CEM dimension (i.e., metacognition, generative learning, and retrieval). Analyses began with interviewers writing a post-interview memo (Glaser, 1978) describing any key interview moments and initial thoughts on the students’ level of engagement. Once all interviews were complete, we adapted a qualitative content analysis approach (Morgan, 1993; Mayring, 2000) to code interview transcripts and video data to identify and categorize behavioral and linguistic indicators of cognitive engagement. Both raters had expertise and familiarity with the literature 91 on metacognition, generative learning, and retrieval, and were therefore aware of plausible indicators reflective of each dimension. The initial phase of coding process included regular conversations to establish clear definitions for a priori codes. In addition, raters retained an open coding approach, in which relevant novel behaviors or strategies were noted, even when their identity or nomenclature was unknown. These were organized into meaningful categories and emergent themes were identified. Due to the length of the interviews and human resources, two researchers coded the video and transcript data concurrently. Intercoder reliability (ICR) measures agreement between two or more coders regarding how the same data should be coded (O’Connor & Joffe, 2020). Whereas interrater reliability (IRR) is reported for data rated on an ordinal or interval scale (e.g., scale of low to high engagement), ICR is appropriate for categorizing data at a nominal level (e.g., presence or absence of a behavior) (Cheung & Tai, 2021; O’Connor & Joffe, 2020). In cases of non-agreement, researchers discussed and came to a consensus decision, reaching an ICR of 1.0. A clear coding frame was developed (Table 2.2) to reduce, classify, and synthesize the data (Gaskell, 2000). RESULTS A Framework for Measuring Cognitive Engagement During Model-Construction Our analyses of students’ statements and behaviors during model construction revealed a total of 14 unique indicators distributed across three dimensions of learning (metacognition, generative learning, and retrieval) and three phases of task completion (planning, monitoring, and evaluating) (28 indicators overall). Below, we provide definitions and examples of key indicators for each CEM dimension that derive from literature review (Table 2.2). In addition, we note the phase(s) in which each indicator appeared (Table 2.3). 92 CEM Dimension 1: Metacognitive Strategies Nine unique indicators of metacognitive strategy use were identified during interviews (Table 2.2). Each indicator was consistent with metacognitive indicators referenced in cognitive engagement literature, though in some cases, we modified the indicator name to better reflect the unique context of modeling. Metacognitive strategies were observed in all phases of model construction, though not all indicators were observed in all phases. (1) Task organization considers students’ verbal or behavioral indicators that explain how they are combining different pieces of information together in order to complete a task or solve a problem. According to Morin (2014), metacognition begins when a student thinks about the steps and strategies they will use to complete a task. Butler (1998) refers to this type of metacognitive activity as “interpreting task requirements”, which is considered in some research as a deep learning strategy (e.g., Appleton et al., 2006; Fredricks et al., 2004; Chi et al., 2018; Greene, 2015; Miller et al., 1996; Schnitzler et al., 2020; Veenman et al., 2006). Students’ use of task organization was only observed during the planning phase. (2 & 3) Identifying key components and relationships is critical for constructing system models, such as the ones students were tasked with in this study. Students used a box- and-arrow framework for developing system models in which structures (physical components of a system) are in boxes, and relationships (the mechanisms connecting structures) are on connecting arrows (Goel & Stroulia, 1996; Dauer et al., 2013). Together, the structures and relationships (boxes and arrows) illustrate how a system functions. As students identify key components and relationships, they are engaging in a metacognitive strategy of “unpacking the task” and deciding what is or is not important 93 to include (Flavell, 1976; Fogarty, 1994). In other words, as students identify components and relationships that will go into their model, they must consider both what is necessary for representing the system as well as explaining the specified model function (Momsen et al., 2022). This is consistent with Meijer et al.’s (2006) metacognitive planning category of “looking for particular information in text.” Students identified key components and relationships only during the planning phase. (4) Self-questioning is a metacognitive process that enables learners to gain a better understanding and organize their thinking before, during, and after the task at hand (King, 1991; Kramarski & Mevarech, 2003; Meijer et al., 2006; Schoenfeld, 1992; Weinstein, et al., 2000; Williamson, 1996). Self-questioning can help students focus their attention and interact more deeply with the presented information (Kramarski & Dudai, 2009). One study found that students who self-questioned before a challenging task (i.e., “What do I need to do first?”) performed significantly better than students who made declarative statements, such as, “I will do this first” (Senay et al., 2010). Questions during the planning phase referred to preparation of the problem-solving process, whereas questions during the monitoring phase were directed toward the problem- solving process itself. No self-questioning was observed during the evaluation phase. (5) Error detection, sometimes referred to as error monitoring, is considered a metacognitive skill in which students find and reflect upon errors, leading to deeper learning and more correct conceptions (e.g., Borasi, 1994; Grosse & Renkl, 2004; Kruger & Dunning, 1999; Meijer et al., 2006; Melis et al., 2010; Ohlsson, 1996; Weinstein et al., 2000; Yeung & Summerfield, 2012). Several researchers consider error detection an “expert- like” skill (e.g., Aleven & Koedinger, 2002; Bielaczyc et al., 1995; Lewis, 1989; Masson 94 et al., 2014). In modeling, error detection can include verbal and non-verbal indicators of students noticing something missing or incorrect in their model (Bennett et al., 2020; Dauer et al., 2024). Error detection emerged during the monitoring phase as students worked through the model-based task, and during the evaluation phase as students reviewed their work and were probed by interviewers on their problem-solving process. (6) Error correction is a metacognitive strategy in which one revises an element of a model or explanation in order to correct an error (e.g., Bennett et al., 2020; Chin & Brown, 2000a; Fernandez-Duque et al., 2000; Meijer et al., 2006; NRC, 2000). While error detection always precedes error correction, error correction does not always follow from error detection. Students engaged in error correction during the metacognitive monitoring and evaluating phases. (7) Progress toward a solution is indicated when students verbalize the steps they are taking to solve a problem while engaged in the problem-solving process (Veenman et al., 2006). Evidence of progress towards a solution was made during the monitoring phase as students talked the interviewers through their mental processes while trying to understand and complete the task at hand. (8) Acknowledging uncertainty, limitations in one’s ideas, or a lack of knowledge are considered productive metacognitive strategies (Chin & Brown, 2000a; Meijer et al., 2006). Uncertainty is common in academic settings as students may struggle to learn and utilize new knowledge and skills and come to new understandings (Jordan, 2010), yet the experience of uncertainty can push students toward reorganization of their thinking; thus, leading to learning (e.g., Jonassen & Land, 2012). Students’ acknowledgement of 95 uncertainty occurred during the monitoring phase as they worked through completion of the task. (9) Rechecking is a metacognitive strategy in which the student monitors one’s own comprehension of text or images by deliberately pausing and going back to the provided text or image (Meijer et al., 2006; Huff & Nietfeld, 2009). Rechecking was identified during the monitoring phase with combined behavioral and verbal cues reflecting students’ comprehension of the task or their own work. CEM Dimension 2: Generative Learning Strategies Indicators of generative learning were derived from two existing frameworks. Fiorella & Mayer’s (2016) generative learning framework identifies summarizing and self-explaining as indicators of generative learning, while Bennett et al’s (2020) ‘Approach to Modeling’ framework contributes self-generated analogies as an additional indicator of generative learning strategy use. All three indicators were measured during planning, monitoring, and evaluating phases. (1) Summarizing entails actively selecting main ideas and translating them into one’s own words (Brod, 2021; Fiorella & Mayer, 2015, 2016). This can include giving a brief verbal overview of a discussion, argument, or passage, or taking notes during a lecture (e.g., Peper & Mayer, 1986; Ross & Kirby, 1976). Brod (2021) characterizes summarizing as a student enriching the provided information with additional content beyond only paraphrasing or condensing the given information. The act of summarizing encourages learners to select what they believe to be the most relevant information and integrate it into existing knowledge (Fiorella & Mayer, 2015, 2016). 96 (2) Self-explaining differs slightly from summarizing by involving further elaboration upon the material. Self-explaining draws upon more active use of relevant prior knowledge to reorganize the new information into a more meaningful mental representation (Chi et al., 1994; Fiorella & Mayer, 2016). For example, while reading through background information for a modeling-based task, a student could verbally explain how the new material integrates with existing knowledge or areas where they are unfamiliar with the content. (3) Self-generated analogies indicate generative learning as a learner creates meaning from the a by relating it to other ideas or concepts (Bennett et al., 2020; Chin & Brown, 2000a, 2000a, Postareff et al., 2015; Fiorella & Mayer, 2015; Wittrock & Alesandrini, 1990). Research suggests that the use of analogies is a key component of the process of modeling (Chin & Brown, 2000a, 2000b; Louca & Zacharia, 2012), and that the generation of analogies between new and existing knowledge can lead to deeper levels of learning (Mayer, 2010; Wittrock, 1994) and better conceptual understanding in science (Wong, 1993a, 1993b). In our study, students generated analogies regarding both content and procedural information. CEM Dimension 3: Retrieval Strategies For students to be successful on tasks, including modeling-based tasks, they must be able to draw from previously learned information and apply it to the present context. For our study, we considered retrieval in relation to both procedural knowledge about modeling and biological content knowledge. Statements of previously learned content and procedural knowledge indicate an activation of prior knowledge and can inform researchers about what information has been stored in long-term memory and whether that information exists in a meaningful, retrievable 97 form (Moreira et al., 2019). Retrieval strategies were observed for both content and procedural knowledge in all model-construction phases. (1) Retrieval of procedural knowledge was measured through statements reflecting recall or recognition of model-based practices or techniques. For example, students could make statements about specific model-building techniques, such as how to start or where to start, or statements on how to analyze and find a solution to data integration or reasoning problems. (2) Retrieval of content knowledge was measured through statements reflecting recall or recognition of previously learned biological concepts. Statements could reflect content specific to the introductory-biology course or general content from any other course. For example, carbon cycle model tasks may have elicited additional content from prior molecular biology or chemistry courses. 98 Table 2.2. Cognitive Engagement in Modeling (CEM) Framework Indicators. Definitions and examples of CEM indicators for Metacognitive, Generative Learning, and Retrieval Dimensions. 99 Table 2.3. Cognitive Engagement in Modeling (CEM) Indicators by Modeling Phase. Appearance of indicators for Metacognitive, Generative and Retrieval dimensions during planning, monitoring, and evaluation phases of model-construction. Applying the CEM Framework to Characterize Students’ Cognitive Engagement During Modeling We applied our CEM framework to interview transcript and video data to explore two research questions: (1) How does cognitive engagement vary across phases of model construction?; (2) How does cognitive engagement compare when students are constructing a novel model versus a model that had been previously constructed (i.e., repeat model)?; and, (3) How does academic achievement (i.e., grades) relate to students’ cognitive engagement in model-construction tasks? RQ1. Cognitive engagement in model-construction phases For this study, we applied the CEM framework to identify the presence of indicators in each modeling phase for novel and repeat model-construction tasks. Overall, both model construction 100 tasks elicited a variety of learning strategies and, occasionally, a large number of them. Data in Tables 2.4 and 2.5 suggest that all phases of modeling construction tasks elicit a fair amount of cognitive engagement, but trends differ by modeling phase. Planning. Students evidenced a fair number of strategies in the planning phase. Of these, metacognitive strategies were most prevalent; particularly, task organization (10 students), and identification of key components (10 students) and relationships (10 students). Within students, identification of key components was the most frequently used strategy overall. Of the generative learning strategies, only two students generated an analogy, whereas all 10 evidenced self- explanation. Nine students indicated retrieval strategy use; particularly, eight students evidenced retrieval of procedural knowledge and five content knowledge. Monitoring. Students exhibited the most strategies and the greatest prevalence of strategy-use during monitoring. Six indicators of metacognitive strategy use were recorded, and of those, error detection (9 students), error correction (9 students), progress toward a solution (10 students), and rechecking (10 students) were used most frequently. Within students, rechecking was used most frequently. Of the generative learning strategies, only one student generated an analogy, seven engaged in summarizing, and all 10 evidenced self-explanation. The majority of students evidenced both indicators of retrieval strategy use, with seven demonstrating retrieval of procedural knowledge and eight demonstrating retrieval of content knowledge. Evaluation. Students exhibited the fewest strategies during evaluation. Only three indicators of metacognitive strategy use were evidenced, including rechecking (10 students), error detection (5 students), and error correction (5 students). Compared to the planning and monitoring phases, generative learning strategies were evidenced the least during the evaluation phase. Of the generative learning strategies, however, self-explanation was used by all 10 101 students. Within the evaluation phase, retrieval strategies were the least observed, with six students evidencing retrieval of procedural knowledge and five content knowledge. RQ2. Cognitive Engagement by Task Context Based on previous research, we hypothesized that students would exhibit greater levels of cognitive engagement in the Novel Model due to the increase in task complexity. Table 2.6 shows the difference in the frequency of indicators between the Novel and Repeat Model. Overall student cognitive engagement totals presented in Table 2.6 do not necessarily support our hypothesis, as while there is evidence of greater cognitive engagement in the Novel context (positive total), there is also equal use in both contexts (zero), and more cognitive engagement in the Repeat context (negative total). At the phase level, data indicates a greater frequency of strategy use during Planning in the Novel context (i.e., more red), whereas there is greater strategy use during Evaluation in the Repeat context (i.e., more blue). Students appear to use strategies fairly equally in both contexts during Monitoring. Students varied greatly in their use of metacognitive strategy use, for example, students engaged in more task organization and identification of key components and relationships with the Novel Model, whereas self-questioning was the only metacognitive strategy students used more frequently with the Repeat Model. Both models elicited a small number of distinct generative learning strategies. Analogy was rarely used, with only two students using it during the Planning Phase of the Repeat Model and one student using it during Evaluation of the Novel Model. Self-explanation, however, was used by all students and more frequently in the Novel Model. Interestingly, summarizing was only used by one student during Evaluation, but appeared in both Planning and Monitoring Phases of both model types. Retrieval strategies were mixed 102 across model types, but there was a tendency to engage in more content recall for the Repeat Model. Some students, such as June and Ember, demonstrated trends we hypothesized and used more strategies for the Novel Model (Table 2.5) than the Repeat Model (Table 2.4). For example, although June was the least cognitively engaged of the ten students for the Repeat Model, she was among the most cognitively engaged for the Novel Model. In the Planning Phase of the Repeat Model, June used no generative strategies and only a single instance of a metacognitive learning strategy (task organization) stating, “I’m trying to think about the type of model to show this. I think it may be a DNA helix model.” But when Planning for the Novel Model, June indicated four instances of two generative strategies (summarizing, self-explaining) and eight instances of four metacognitive strategies (self-questioning, identifying key components and relationships, and task organization). Her greater use of task organization was illustrated by her stating, for example, “It says there are two different pathways, so it probably branches off” and “It says they are coupled, so I think that means they will interact somehow.” During Monitoring, June used the same number of strategies overall between tasks, but the specific strategies and frequencies of use differed. For example, June only used error detection and correction for the Repeat Model but invested more cognitive engagement into making progress towards a solution and acknowledging uncertainty for the Novel Model. Despite being considered a middle-achieving student, Ember was the most cognitively engaged in both model-construction contexts. She used more learning strategies thank any of her peers, particularly in the Novel Model task (52 total strategies vs. 13-36 for all other students). Unlike June, Ember remained fairly consistent in which strategies she used during the Planning Phase but engaged in them more frequently in the Novel context. She became more specific in 103 the Novel context as well. For example, in the Repeat context, she made two general statements about identifying key concepts: “I’m just underling key information,” and, “I’m circling words that will go into the model.” On the other hand, her statements in the Novel context included, “I’m circling what carbon is controlled through […]”, “I think this is the key takeaway – that carbon is transformed,” and, “I circled ‘concentration of carbon’ because I think that’s a huge component to understanding all of this.” Ember’s cognitive engagement in the Novel Model is notable for her acknowledgement of uncertainty. Although absent in the Repeat Model, she indicated multiple points of confusion in the Novel context through statements such as, “Oh, I was getting confused on which one [component] is putting it [carbon] out and which one is putting it in […] and now I feel like I’m missing a lot of stuff,” and, “This just doesn’t feel right, I think I’m missing some things like photosynthesis, so I think I need to add more boxes.” In contrast to students like June and Ember, some students executed fewer learning strategies in the Novel Model. For example, Ashley used 26 strategies for the Repeat Model and only 1e for the Novel Model. In the Repeat Model, Ashley used no learning strategies for Planning and only one (rechecking) in the Evaluation Phase. Her primary strategy use occurred during the Monitoring Phase, as she engaged in repetitive use of trying to find a solution and rechecking and was centered on trying to determine appropriate relationships for her model. For example, she stated, “So I'm going to start with the alleles and then I need an action word […] Okay, so I will have the defective and normal protein, but now I need to figure out the relationship […] "I'm trying to figure out what to put on the action arrow there [between normal protein and no cystic fibrosis]”. Ashley expressed more ‘self-questioning’ than any other student, which included explicitly trying to recall the exam-model she had previously constructed, with, “How did I start this before?”. Ashley was the least-cognitively-engaged student in the Novel 104 Model where she shifted her investment to the Planning Phase (five strategies) but only two during Monitoring, and was the only student to use no strategies for Evaluation. Table 2.4. Cognitive Engagement During a Repeat Model Task. Heat map reflecting instances of metacognitive, generative learning, and retrieval strategy-use during repeat-model construction according to total number of indicators recorded. Interviewees are identified by their pseudonym. Colors represent varying counts of each indicator: the lighter the color, the lower the count; the darker the color, the higher the count. 105 Table 2.5. Cognitive Engagement During a Novel Model Task. Heat map reflecting instances of metacognitive, generative learning, and retrieval strategy-use during novel-model construction according to total number of indicators recorded. Interviewees are identified by their pseudonym. Colors represent varying counts of each indicator: the lighter the color, the lower the count; the darker the color, the higher the count. 106 Table 2.6. Difference Map of Cognitive Engagement for Novel and Repeat Models. Interviewees are identified by their pseudonym. Scores reflect differences in the frequency of indicators between the novel and previously-constructed model (i.e., [frequency of indicator during novel model construction] - [frequency of indicator during repeat model construction]. Positive scores (red) reflect a higher frequency of an indicator during novel model construction. Negative scores (blue) reflect a higher frequency of an indicator during repeat model construction. Zero values (white) indicate no difference between the two models in the frequency of an indicator. RQ3. Cognitive engagement across student achievement levels We predicted that higher achieving students might show the greatest use and diversity of learning strategies, and that the reverse would be true for lower achieving students. This prediction was not born out by the data. Tables 2.4 and 2.5 show that students of all achievement levels are cognitively engaged during model construction tasks and that learning-strategy use appears unrelated to achievement level. Indeed, no clear trends appear to emerge in relation to achievement level. When considering overall strategy use for both the Repeat and Novel tasks, high and middle achieving students appear at both the highest and lowest frequencies of strategy use. When considering differences in approaches between task types (Tables 2.6), high and 107 middle achievers similarly appear at both extremes. Interestingly, students classified as lower achieving are consistently in the middle across all comparisons. However, some differences to emerge when considering interactions among achievement level, task type, and specific strategy use. Considering metacognitive strategy use, higher-achieving students were more engaged in error detection and correction during Monitoring of the Repeat Model (i.e., greater frequency of blue) compared to their middle- and lower-achieving peers who indicated more error detection and correction with the Novel Model (i.e., greater frequency of red). Across high- and middle- achievement groups, error detection and correction in both contexts consisted mostly of students’ statements and behaviors of detecting and then correcting incorrect components or relationships in their model. Neither Ryan nor Hope, the two students considered low achieving, engaged in this type of error detection and correction. Instead, their indicators error detection were concerned with physical model construction. For example, in the Repeat Model, Hope drew a single arrow from her first ‘Chromosome 7’ component, but then erased it stating, “Oh wait, not this.” Hope ended up not including a relationship from Chromosome 7 in her model, and, when prompted by interviewers to talk through her thinking she stated, “This model would need some verbal explanation. I have to mentally form arrows on this model.” Similarly, in the Novel Model, Ryan began by drawing a single, long, curved arrow. He then stated, “Wait, this needs components,” and then erased the large arrow and re-drew smaller, curved arrows connected by boxes. Overall, differences in error detection and correction that emphasized content versus physical model structure warrants further exploration. 108 Trends for other metacognitive indicators (e.g., progress towards solution, acknowledges uncertainty, rechecking), suggest higher-achieving students engaged in these strategies more during the Novel Model, whereas middle-and low-achieving students indicated more use during the Repeat Model. Generative learning strategy use was generally infrequent across achievement levels and across contexts, but some gaps were noteworthy. No low-achieving students used generative strategies in the Planning Phase of the Repeat Model, and no high-achieving students used them in the Evaluation Phase of the Novel Model. Middle-achieving students remained fairly consistent with generative learning strategy use across contexts and phases. Overall, retrieval strategies were not common in the Novel context, but middle-achieving students accounted for the majority of them used. Interestingly, these middle-achieving students (specifically, Anna, David, and Ember) retrieved content from other biology courses in relation to the carbon cycle (Novel Context). For example, Anna specifically stated, “I haven’t thought about carbon cycles since the cellular and molecular course.” David recalled, “I know from previous classes that CO2 is stored in the atmosphere.” And Ember stated, “I remember learning about the carbon cycle from the cellular course, so I’m trying to remember the big ideas from back then.” DISCUSSION Attrition from STEM majors has been positively linked to pedagogical practices that fail to engage learners in ways that reflect their interests and promote active construction of knowledge (Hunter, 2019). MBI is as an evidence-based pedagogical approach in which, students construct, interpret, revise, and evaluate scientific models (Clement, 2000; Gilbert & Justi, 2016; Justi & Gilbert, 2002a, 2002b; Long et al., 2014; Louca & Zacharia, 2012; Schwarz et al., 2009). MBI shows promise as an instructional approach that reduces achievement gaps 109 and promotes more equitable outcomes compared with traditional performance measures (Bierema et al., 2017; Dauer et al., 2013; Manthey & Brewe, 2013; Reinagel & Bray Speth, 2016; Verhoeff et al., 2008). However, research to date has not explicitly addressed whether modeling specifically promotes engagement, nor what behavioral or cognitive indicators can provide evidence of engagement when students are performing model-based tasks. Our work draws from existing theory about cognitive engagement and in-situ observations of students actively constructing models to propose and test a Cognitive Engagement in Modeling (i.e., CEM) framework for characterizing how students are cognitively engaged in model-based tasks. Framework Elements We identified 14 unique linguistic and behavioral indicators distributed across three dimensions, where dimensions reflect a distinct category of strategy use: Metacognition, Generative Learning, and Retrieval. All 14 indicators were used by more than one student, and many appeared in more than one phase of modeling, suggesting that our proposed indicators are generally relevant for inclusion in the framework and not unique to any individual. Analogy, a generative learning strategy, was the least frequently used indicator overall, but was still reflected in the responses of three of the ten students interviewed. We hypothesized that students’ strategy use might differ at different times during a model-construction task based on their specific goals at any given moment. We therefore included three Modeling Phases (planning, monitoring, evaluation) as elements in our CEM and sought to characterize differences in strategy use by phase. Our data affirm that students’ strategy use differs across modeling phases, and therefore, provides support for our decision to include Phase as a CEM element. Metacognition, for example, has the largest number of indicators (9 overall) but none of these was observed across all three modeling phases despite 110 being used by a majority of students. Identifying key components (used by all 10 students), identifying key relationships (9 students), and task organization (10 students) were unique to the planning phase, while progress toward a solution (10 students) and acknowledging uncertainty (7 students) were observed exclusively in the monitoring phase. Self-questioning occurred in both planning and monitoring phases and was used by eight and six students, respectively. Rechecking was used by all students in both monitoring and evaluation phases, but error detection and error correction were both more likely to be used during monitoring (9 students) than evaluation (5 students). Indicators of Generative Learning and Retrieval appeared in all three phases of modeling, and therefore may be indicative of more generalized strategies for model-based learning not aligned with any specific phase. Additionally, because Generative Learning and Retrieval were each observed through a smaller number of indicators (3 and 2, respectively; Table 2.3) compared to Metacognition (9 indicators), we caution against the potential interpretation that the importance of a framework dimension in being an effective and engaged modeler should be measured through the number (or frequencies of instances) of any particular indicator. Framework Application We applied our framework to examine relationships between cognitive engagement, task phases, task type, and academic achievement. Nature of the problems or task can result in differences in student interest and engagement (e.g., Mitchell & Carbone, 2011). In line with this, previous model-based research found that prompt construction influenced students’ depth of engagement needed to construct a correct model (Bennett et al., 2020). Our data further suggest the type of model, previously-constructed or novel model, influences which types of cognitive learning strategies students’ use and when they are used during the model-constructed process. 111 During the Planning phase, for example, we find students use more metacognitive strategies, particularly task organization and identification of key concepts and relationships, while constructing a novel-model compared to a Repeat Model. This finding is supported in previous metacognitive research that suggests metacognitive strategy-use becomes progressively more important as task complexity increases (e.g., Hattie et al., 1996; Mokos & Kafoussi, 2013). When compared to the Repeat Model, Planning for the Novel Model required students to interpret key components and relationships, and actively generate a simplified representation of the phenomenon of familiar content, but apply it to a novel context. Our data is mixed, however, during the Monitoring phase as students indicated a varied use of metacognitive strategies across both model contexts. Finally, our data recorded during the Evaluation phase contends prior research, as students utilized greater metacognitive strategy use for the Repeat Model. One key component to the generative learning theory is the idea of integration (i.e., connecting textual, verbal, or pictorial representations with each other and with relevant prior knowledge) (Fiorella & Mayer, 2015, 2016; Gunawan, et al., 2019; Parong & Mayer, 2018; Wittrock, 1974, 1992). We hypothesized that construction of the novel model would inherently require greater integration with learners’ existing knowledge structure as they generate a mental and physical representation of familiar content in a novel context. Our data for this is mixed, as that all or majority of students indicated greater use during the Planning and Monitoring phases of the novel model, however, there is greater use of generative learning strategies during the Evaluation phase for the repeat model. Research on retrieval informs researchers on not only what students know, but also what students don’t know. Our study investigates this over the long-term and which concepts are being transferred to new contexts. The knowledge a person expresses can vary greatly depending 112 on the retrieval cues present in a particular context (Grimaldi & Karpicke, 2014; Karpicke & Grimaldi, 2012). Some research suggests that successful learners might be developing better procedural knowledge and establishing a better repertoire of strategies for how to learn in the domain (e.g., Alexander & Judy, 1988; Anderson, 1996; Greene, 2015). Examining engagement during construction of a repeat model from the course, followed by construction of a novel model can provide in-depth information on content and skills not only retained from the class, but also students’ ability to transfer these to new contexts. Understanding retrieval is essential for understanding learning (Grimaldi & Karpicke, 2014; Karpicke & Grimaldi, 2012) helps us figure out what are the specific retrieval cues that students pick up on - can inform better ways of providing retrieval cues for students that allow them to reconstruct their knowledge. Studies that give student practice retrieving, which can promote meaningful learning, in which students are better able to organize and integrate new information into mental models for which can then be used to apply knowledge. Previous model-based research suggested that achievement is a poor predictor of modeling-based performance and more research was needed to get a better understanding of potential mechanisms to explain performance differences in MBI contexts (Bennett et al., 2020; Dauer et al., 2013; Dauer & Long, 2015; de Lima, 2020). In our study, students’ achievement level did not predict level of cognitive engagement during model-construction and the level of engagement varied across task types. We found that high-, middle-, and low-achieving students employed a large and diverse number of learning strategies when completing model-construction tasks. With future applications of the CEM framework, we can further investigate how students of lower achievement groups are cognitively engaging, specifically, the type of learning strategies they are, or are not, utilizing in other types of practice-based tasks. With this 113 knowledge, we can inform targeted interventions across STEM courses that can have a large numeric impact on STEM retention rates. Surely, many factors play a role in our results, however, it is possible that the different levels of cognitive engagement and differences in learning-strategy use in model construction are related to motivational factors, such as students’ learning goals during the course. The present study builds from research on students’ motivational profiles in a modeling-based introductory biology course, in which motivation is considered an antecedent to engagement in modeling (Furqueron & Long, in preparation). Development of the CEM framework now allows for relationships to be explored between students’ motivational profiles and use of cognitive engagement learning strategies in modeling-based biology courses. LIMITATIONS The CEM framework was developed for modeling generally, but was tested only in the context of introductory biology students’ construction of biological system models. As a science practice, modeling includes additional processes, such as using models to reason and make predictions, evaluating model-based information, and revising models that incorporate new information or feedback (Krell et al., 2013). Additional research will be necessary to determine the generalizability of our findings to other model-based tasks and disciplinary contexts. While observational protocols are designed to overcome issues of self-report bias, we acknowledge a possibility of bias in that the observer may be attuned to noticing what they are looking for and missing what they are not (Minner et al., 2010; Sinatra et al., 2015). Observational studies are further limited by the need to interpret or infer constructs that may not be explicitly presented (Van Hout-Wolters, 2000). For example, Meijer et al. (2006) indicate that metacognitive activities can be very hard to distinguish, and thoughts and actions inferred by 114 researchers from specific behaviors may not always be accurate. In many instances within our study, the learning strategy used as evidence of cognitive engagement was necessarily inferred from verbal or behavioral indicators, and therefore may be limited to those that were most easily identifiable. We aimed to address these limitations by including a second interviewer and discussing codes and indicators among the larger research team. Finally, qualitative work that measures frequencies of statements or indicators risks overestimation of constructs in students that talk more or are most comfortable externalizing their thoughts (Meijer et al., 2006). We therefore analyzed indicators as both presence/absence and frequency. In addition, we further acknowledge that interview studies are inherently limited due to being in non-natural settings and potential bias in student responses due to researcher presence (Creswell & Creswell, 2018). Although researchers attempted to create a relaxing environment for students by offering refreshments and generating welcoming small talk, a few students expressed feelings of nervousness being in front of the interviewers, which may have limited students’ task performance. Our study included interviews from ten participants from a second-semester introductory biology course. Our intended design of 30 participants was unachievable on account of Covid-19 restrictions that went into effect after the study was underway. We intentionally sampled across achievement levels in order to capture diversity in the student population and explore the influence of prior academic achievement, but acknowledge that our sample sizes are small and limit our ability to make claims about the influence of prior academic achievement on trends in engagement. Although women are over-represented in our study, our sample otherwise approximates the diversity of the course in which it was conducted. 115 IMPLICATIONS FOR INSTRUCTION AND CONCLUSION Our study addresses a gap identified by earlier researchers (Christenson et al., 2012) by contributing a framework for measuring student cognitive engagement in modeling (CEM). The CEM framework identifies specific cognitive processes and learning strategies students use during model-based tasks. Strategies are classified by type (dimension) and organized in relation to distinct phases during task completion. The CEM advances research on both cognitive engagement and model-based learning by establishing a framework for posing and testing hypotheses about students’ thinking as they are actively engaged in the work of completing a learning task. Model-based instruction (MBI) is used in multiple domains (e.g., biology, physics, chemistry) and is one example of an instructional approach rooted in authentic scientific practice. Our study uses the CEM to provide direct evidence about the nature of cognitive engagement during modeling, but additional research would be necessary to determine if CEM components translate to other practice-based learning tasks, such as scientific argumentation, explanation, data analysis, etc. Generalized frameworks about cognitive engagement could be especially useful in supporting students in the transfer of learning between disciplinary contexts and across task types; the CEM could be a useful tool for advancing such research. Previous model-based research suggested that achievement is a poor predictor of modeling-based performance, and more research was needed to get a better understanding of potential mechanisms to explain performance differences in MBI contexts (Bennett et al., 2020; Dauer et al., 2013; Dauer & Long, 2015; de Lima, 2020). With future applications of the CEM framework, we can further investigate how students of lower achievement groups are cognitively engaging, specifically, the type of learning strategies they are, or are not, utilizing in other types of practice-based tasks. Indeed, recent work suggests that the “existing literature provides little 116 insight into whether passing a given science course relates to student engagement in intellectual work authentic to the practice of science” (Ralph et al., 2022, pp.843). With this knowledge, we can inform targeted interventions across STEM courses that can have a large numeric impact on STEM retention rates. Researchers, policymakers, and educators are increasingly focused on student engagement as a means to enhance student learning and promote retention in academic programs and STEM fields particularly (e.g., Fredricks et al., 2004; Hofkens & Ruzek, 2019; Reschly & Christenson, 2012; Sinatra et al., 2015; Wang et al., 2019). Cognitive engagement cannot be undervalued in educational settings, as it has been tied to improved educational outcomes for many years (e.g., Chi et al., 2018; Fredricks, 2011; Fredricks et al., 2004; Greene, 2015; Fredricks & McColskey, 2012; Martin et al., 2017). Fostering engagement in learning tasks is therefore not only an end-goal in itself, but a means toward achieving positive academic outcomes, including retaining students at-risk of leaving. Importantly, the CEM was derived by considering student voices (Christenson et al., 2012) from a range of achievement levels to better inform how and when diverse students are engaging in task-specific learning strategies. Our data clearly show that students differ in the ways in which they manifest cognitive engagement during task completion, and offers additional support for non-traditional and practice-based instructional approaches to be more inclusive of diverse students and effective in reducing achievement gaps. Finally, our study revealed a range of distinct learning strategies that students used to navigate their way through two model-based learning tasks. Students who were able to leverage a broader range of strategies were generally more successful in their ability to progress toward a solution. We therefore call for instruction that explicitly trains students about a range of alternative learning strategies in order to empower them with a robust and flexible toolkit for solving 117 diverse problems. Students who are aware of different strategies for learning, thinking, and problem solving are more likely to use them (Pintrich, 2002). This, however, requires that students know about these strategies and have been trained how to use them. 118 REFERENCES Akturk, A. O., & Sahin, I. (2011). Literature review on metacognition and its measurement. Procedia-Social and Behavioral Sciences, 15, 3731-3736. https://doi.org/10.1016/j.sbspro.2011.04.364 Aleven, V., & Koedinger, K. R., (2002). An effective metacognitive strategy: learning by doing and explaining with a computer-based cognitive tutor. Cognitive Science, 26, 147-179. https://doi.org/10.1207/s15516709cog2602_1 Alexander, P. A., & Judy, J. E. (1988). The interaction of domain-specific and strategic knowledge in academic performance. Review of Educational Research, 58, 375-404. https://doi.org/10.3102/00346543058004375 Alrashidi, O., Phan, H. P., & Ngu, B. H. (2016). Academic engagement: an overview of its definitions, dimensions, and major conceptualisations. International Education Studies, 9(12), 41-52. The American Association for The Advancement of Science (AAAS). (2011). Vision and change in undergraduate biology education: A call to action. AAAS: Washington, DC. Retrieved December 12, 2018, from https://www.aaas.org/sites/default/files/content_files/VC_report.pdf The American Association for The Advancement of Science (AAAS). (2015). Vision and Change in Undergraduate Biology Education: Chronicling change, inspiring the future. AAAS: Washington, DC. Anderman, E. M., & Midgley, C., (1997). Changes in achievement goal orientations, perceived academic competence, and grades across the transition to middle-level schools. Contemporary Educational Psychology, 22(3), 269-298. https://doi.org/10.1006/ceps.1996.0926 Anderman, E. M., & Patrick, H. (2012). Achievement goal theory, conceptualization of ability/intelligence, and classroom climate. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 173–191). Springer Science + Business Media. https://doi.org/10.1007/978-1-4614-2018-7_8 Anderson, J. R. (1996). ACT: A simple theory of complex cognition. American Psychologist, 51, 355-363. http://dx.doi.org/10.1037/0003- 066X.51.4.355 Antonietti, A., Colombo, B., & Di Nuzzo, C. (2015). Metacognition in self-regulated multimedia learning: Integrating behavioural, psychophysiological and introspective measures. Learning, Media and Technology, 40(2), 187-209. https://doi.org/10.1080/17439884.2014.933112 119 Appleton, J. J., Christenson, S. L., & Furlong, M. J. (2008). Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in the Schools, 45(5), 369–386. https://doi.org/10.1002/pits.20303 Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: validation of the Student Engagement Instrument. Journal of School Psychology, 44, 427-445. https://doi.org/10.1016/j.jsp.2006.04.002 Archambault, V., Dupéré, V., (2017). Joint trajectories of behavioral, affective, and cognitive engagement in elementary school. The Journal of Educational Psychology, 110, 188-198. https://doi.org/10.1080/00220671.2015.1060931 Avargil, S., Lavi, R., & Dori, Y. J. (2018). Students’ metacognition and metacognitive strategies in science education. In Y. J. Dori, Z. Mevareach, & D. Baker (Eds.), Cognition, metacognition and culture in STEM education (pp. 33–64). Cham: Springer. Bae, C. L., De-Busk-Lane, M. (2019). Middle school engagement profiles: Implications for motivation and achievement in science. Learning and Individual Differences, 74(2):101753. https://doi.org/10.1016/j.lindif.2019.101753 Bannert, M., & Mengelkamp, C. (2008). Assessment of metacognitive skills by means of instruction to think aloud and reflect when prompted. Does the verbalisation method affect learning? Metacognition and Learning, 3, 39-58. https://doi.org/10.1007/s11409-007-9009-6 Barlow, A. J., & Brown, S. A. (2019, June). Work in progress: Measuring student cognitive engagement using the ICAP framework in and outside of the classroom. In 2019 ASEE Annual Conference & Exposition. Ben-Eliyahu, A., Moore, D., Dorph, R., & Schunn, C. D. (2018). Investigating the multidimensionality of engagement: AFfective, behavioral, and cognitive engagement across science activities and contexts. Contemporary Educational Psychology, 53, 87-105. https://doi.org/10.1016/j.cedpsych.2018.01.002 Bennett, S. W., Gotwals, A. W., & Long, T. M. (2020). Assessing students’ approaches to modeling in undergraduate biology. International Journal of Science Education, 42(10), 1697-1714. https://doi.org/10.1080/09500693.2020.1777343 Berardi-Coletta, B., Buyer, L. S., Dominowski, R. L., & Rellinger, E. R. (1995). Metacognition and problem solving: A process-oriented approach. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 205-223. https://doi.org/10.1037/0278-7393.21.1.205 Bielaczyc, K., Pirolli, P. L., & Brown, A. (1995). Training in self-explaining and self-regulation strategies: investigating the effects of knowledge acquisition activities on problem 120 solving. Cognition and Instruction, 13(2), 221-252. https://doi.org/10.1207/s1532690xci1302_3 Bierema, A. M. -K., Schwarz, C. V., Stoltzfus, J. R. (2017). Engaging undergraduate biology students in scientific modeling: analysis of group interactions, sense-making, and justification. CBE - Life Sciences Education, 16(68), 1-16. https://doi.org/10.1187/cbe.17-01-0023 Bingham, G. E., & Okagaki, L. (2012). Ethnicity and Student engagement. In S. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement. (pp.65-95). Springer. Bӧheim, R., Urdan, T., Knogler, M., Seidel, T. (2020). Student hand-raising as an indicator of behavioral engagement and its role in classroom learning. Contemporary Educational Psychology, 62(1). https://doi.org/10.1016/j.cedpsych.2020.101894 Borasi, R. (1994). Capitalizing on errors as “springboards for inquiry”: A teaching experiment. Journal for research in mathematics education, 25(2), 166-208. https://doi.org/10.5951/jresematheduc.25.2.0166 Borkowski, J., Carr, M., & Pressely, M. (1987). “Spontaneous” strategy use: Perspectives from metacognitive theory. Intelligence, 11, 61–75. https://doi.org/10.1016/0160-2896(87)90027-4 Bransford, J. D., Brown, A. L. & Cocking, R. R. (2000). How people learn: Brain, mind, experience, and school. Washington, D. C.: National Academy Press. Bray Speth, E., Long. T. M., Pennock, R. T., & Ebert-May, D. (2009). Using Avida-ED for Teaching and Learning about evolution in undergraduate introductory biology courses. Evolution: Education and Outreach, 2(3), 415-428. https://doi.org/10.1007/s12052-009-0154-z Braxton, J. M., & Hirschy, A. S. (2005). Theoretical developments in the study of college student departure. In A. Seidman (Ed.), College student retention: Formula for student success, (pp. 61-87). Westport, CT: ACE/Praeger. Brewe, E., Sawtelle, V., Kramer, L. H., O’Brien, G. E., Rodriguez, I, & Pamelá, P. (2010). Toward equity through participation in modeling instruction in introductory university physics. Physical Review ST Physics Education Research, 6(1). https://doi.org/10.1103/PhysRevSTPER.6.010106 Brod, G. (2021). Generative learning: Which strategies for what age? Educational Psychology Review. 33, 1295-1318. https://doi.org/10.1007/s10648-020-09571-9 121 Brown, A. L. (1978). Knowing when, where, and how to remember: a problem of metacognition. In R. Glaser (Ed.), Advances in Instructional Psychology, Vol 1 (pp. 77-165). Hillsdale, NJ: Erlbaum. Brown, A. L. (1987). Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In F. W. R. Kluwe (Ed.), Metacognition, Motivation, and Understanding (pp. 65-116). Hillsdale, NJ: Erlbaum. Buhs, E. S., & Ladd, G. W. (2001). Peer rejection as antecedent of young children’s school adjustment: An examination of mediating processes. Developmental Psychology, 37(4), 550-560. https://doi.org/10.1037/0012-1649.37.4.550 Butler, D. L. (1998). The strategic content learning approach to promoting self-regulated learning: A report of three studies. Journal of Educational Psychology, 90(4), 682-697. https://doi.org/10.1037/0022-0663.90.4.682 The Carnegie Classification of Institutions of Higher Education (n.d.). About Carnegie Classification. Retrieved (March 11, 2024) from https://carnegieclassifications.acenet.edu/ Carter, C. S., Braver, T. S., Barch, D. M., Botvinick, M. M., Noll, D., & Cohen, J. D. (1998). Anterior cingulate cortex, error detection, and the online monitoring of performance. Science, 280, 747-749. https://doi.org/10.1126/science.280.5364.747 Chen, X. (2015). STEM attrition among high-performing college students in the United States: Scope and potential causes. Journal of Technology and Science Education, 5(1), 41–59. https://doi.org/10.3926/jotse.136 Cheung, K. K., & Tai, K. W. (2021). The use of intercoder reliability in qualitative interview data analysis in science education. Research in Science and Technological Education, 41(3), 1155-1175. https://doi.org/10.1080/02635143.2021.1993179 Chi, M. T. H., de Leeuw, N. K., Chiu, M. -H. , & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18, 439-477. https://doi.org/10.1016/0364-0213(94)90016-7 Chi, M. T. H., Adams, J., Bogusch, E. B., Bruchok, C., Kang, S., Lancaster, M., Levy, R., Li, N., McEldoon, K. L., Stump, G. S., Wylie, R., Xu, D., & Yaghmourian, D. L. (2018). Translating the ICAP theory of cognitive engagement into practice. Cognitive Science, 42, 1777-1832. https://doi.org/10.1111/cogs.12626 Chick, N. (2013). Metacognition. Vanderbilt University Center for Teaching. Retrieved April 27, 2022 from https://cft.vanderbilt.edu/guides-sub-pages/metacognition/ 122 Chin, C., & Brown, D. E. (2000a). Learning in Science: A comparison of deep and surface approaches. Journal of Research in Science Teaching, 37(2), 109-138. https://doi.org/10.1002/(SICI)1098-2736(200002)37:2<109::AID-TEA3>3.0.CO;2-7 Chin, C., & Brown, D. E. (2000b). Learning deeply in science: An analysis and reintegration of deep approaches in two case studies of grade 8 students. Research in Science Education, 30(2). https://doi.org/10.1007/BF02461627 Christenson S. L., Reschly L. A., Wylie C. (Eds.). (2012). Handbook of research on student engagement. New York, NY: Springer. Cleary, A. M. (2019). Dependent measures in memory research: From free recall to recognition In H. Otani & B. L. Schwartz (Eds.), Handbook of research methods in human memory (1st ed., pp. 19-35). Routledge. Clement, J. J. (2000). Model based learning as a key research area for science education. International Journal of Science Education, 22(9), 1041-1053. https://doi.org/10.1080/095006900416901 Clement, J. J. (2008). Model based learning and instruction in science. In J. J. Clement & M. A. Rae-Ramirez (Eds.), Models and modeling in science education (pp. 1-9). Springer. Connell, J. P., & Wellborn, J. G. (1991). Competence, autonomy, and relatedness: A motivational analysis of self-system processes. In M. R. Funnar & L. A. Sroufe (Eds.), Self-processes and development: Minnesota symposium on child psychology (Vol. 23, pp. 43-77). Chicago: University of Chicago Press. Constantinou, C. P., Nicolaou, C. t., & Papaevripidou, M. (2019). A Framework for Modeling-Based Learning, Teaching and Assessment. In A. Upmeier zu Belzen, J. van Driel, & D. Krüger (Eds.), Towards a Competence-Based View on Models and Modeling in Science Education (Vol. 12, pp. 39-58). Springer, Cham. Cooper, K. S., (2014). Eliciting engagement in the high school classroom: a mixed-methods examination of teaching practices. American Educational Research Journal, 51(2), 363-402. https://doi.org/10.3102/0002831213507973 Cooper, M. M., Caballero, M. D., Ebert-May, D., Fata-Hatrley, C. L. Jardeleza, S. E., Krajcik, J. S., Laverty, J. T., Matz, R. L., Posey, L. A., & Underwood, S. M. (2015). Challenge faculty to transform STEM learning: Focus on core ideas, crosscutting concepts and scientific practices. Science, 350(6258), 281-282. Corno, L., & Mandinach, E. (1983). The role of cognitive engagement in classroom learning and motivation. Educational Psychologist, 18, 88-108. https://doi.org/10.1080/00461528309529266 123 Creswell, J. W., Creswell, J. D. (2018). Research design: qualitative, quantitative, and mixed methods approaches (5th ed.). Sage. Dagley, M, Georgiopoulos, M., Reece, A., Young, C. 2015. Increasing retention and graduation rates through a STEM learning community. Journal of College Student Retention: Research, Theory & Practice, 18(2), 167-182. https://doi.org/10.1177/1521025115584746 D’Mello, S., Dieterle, E., & Duckworth, A. (2017). Advanced, analytic, automated (AAA) measurement of engagement during learning. Educational Psychologist, 52(2), 104-123. https://doi.org/10.1080/00461520.2017.1281747 Dauer, J., Behrendt, M. G., Elliott, M., Gettings, B., Long, T., & Clark, C. (2024). Individual variation in undergraduate student metacognitive monitoring and error detection during biology model evaluation. Frontiers in Education. 9:1356626. https://doi.org/10.3389/feduc.2024.1356626 Dauer, J. T., & Long, T. M. (2015). Long-term conceptual retrieval by college biology majors following model-based instruction. Journal of Research in Science Teaching, 52(8), 1188-1206. https://doi.org/10.1002/tea.21258 Dauer, J. T., Momsen, J. L., Bray Speth, E., Mokohon-Moore, S. C., & Long, T. M. (2013). Analyzing change in students’ gene-to-evolution models in college-level introductory biology. Journal of Research in Science Teaching, 50(6), 639-659. https://doi.org/10.1002/tea.21094 de Lima, J. (2020). Contextual Influences on Undergraduate Biology Students’ Reasoning and Representations of Evolutionary Concepts. [Doctoral dissertation]. Michigan State University. Deekens, V. M., Greene, J. A., & Lobczowski, N. G. (2018). Monitoring and depth of strategy use in computer-based learning environments for science and history. British Journal of Educational Psychology, 88(1), 63-79. https://doi.org/10.1111/bjep.12174 Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis. Educational Psychology Review, 28(3), 425-474. https://doi.org/10.1007/s10648-015-9320-8 Desoete, A. (2008). Multi-method assessment of metacognitive skills in elementary school children: How you test is what you get. Metacognition and Learning, 3(3), 189-206. https://doi.org/10.1007/s11409-008-9026-0 Duncan, R. G. (2007). The role of domain-specific knowledge in generative reasoning about complicated multileveled phenomena. Cognition and Instruction, 25(4), 271-336. https://doi.org/10.1080/07370000701632355 124 Dweck, C. S. (1986). Motivational processes affecting learning. American psychologist, 41(10), 1040. https://doi.org/10.1037/0003-066X.41.10.1040 Eccles, J., & Wang, M. T. (2012). Part I commentary: So what is student engagement anyway? In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 133-145). New York, NY: Springer. Eilam, B., & Reiter, S. (2014). Long-term self-regulation of biology learning using standard junior high school science curriculum. Science Education, 98(4), 705-737. https://doi.org/10.1002/sce.21124 Ericsson, K. A. (2006). Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts’ performance on representative tasks. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 223-242). Cambridge University Press. Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture, and Activity, 5(3), 178-186. https://doi.org/10.1207/s15327884mca0503_3 Fernandez-Duque, D., Bair, J. A., & Posner, M. I. (2000). Executive attention and metacognitive regulation. Consciousness and Cognition, 9, 299-307. https://doi.org/10.1006/ccog.2000.0447 Finn, J. D. (1989). Withdrawing from school. Review of Educational Research, 59, 117-142. https://doi.org/10.3102/00346543059002117 Finn, J. D., & Zimmer, K. S. (2012). Student engagement: What is it? Why does it matter? In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 97-131). New York, NY: Springer. Fiorella, L., & Mayer, R. E. (2015). Learning as a generative activity: eight learning strategies that promote understanding. New York: Cambridge University Press. Fiorella, L., & Mayer, R. E. (2016). Eight ways to promote generative learning. Educational Psychology Review, 28(4), 717–741. https://doi.org/10.1007/s10648-015-9348-9 Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. Resnick (Ed.) The Nature of Intelligence (pp. 231-366). Hillsdale, NJ: Erlbaum. Flavell, J. H. (1979). Metacognition and cognitive monitoring: a new era of cognitive-development inquiry. American Psychologist, 34, 906-911. https://doi.org/10.1037/0003-066X.34.10.906 Flick, U. (2006). An Introduction to Qualitative Research (3rd Ed.). London: Sage Publication, Ltd. 125 Fogarty, R. (1994). How to teach for metacognition. Palatine, IL: IRI/Skylight Publishing. Fredricks, J. A. (2011). Engagement in school and out-of-school contexts: A multidimensional view of engagement. Theory Into Practice, 50(4), 327-335. http://dx.doi.org/10.1080/00405841.2011.607401 Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59-109. https://doi.org/10.3102/003465430740010 Fredricks, J. A., & McColskey, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 763-782). New York, NY: Springer. Fox, M. C., Ericsson, K. A., & Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137, 316-344. https://doi.org/10.1037/a0021663 Furlong, M. J., & Christenson, S. L. (2008). Engaging students at school with learning: A relevant construct for all students. Psychology in the Schools, 45(5), 365-368. https://doi.org/10.1002/pits.20302 Gaskell, G. (2000). Individual and group interviewing. In Bauer, M. W., Gaskell, G. (Eds.), Qualitative researching with text, image, and sound: A practical handbook (pp. 38-56). Sage Publications. Georghiades, P. (2004). From the general to the situated: Three decades of metacognition. International Journal of Science Education, 26(3), 365-383. https://doi.org/10.1080/0950069032000119401 Furqueron, B., & Long, T. M. (In preparation). Exploring motivational profiles in in a model- based undergraduate biology course. Gilbert, J. K. (1991). Model building and definition of Science. Journal of Research in Science Teaching, 28(1), 73-79. Gilbert, J. K., (2004). Models and Modelling: routes to more authentic science education. International Journal of Science and Mathematics Education, 2(2), 115-130. https://doi.org/10.1007/s10763-004-3186-4 Gilbert, J. K., & Justi, R. (2016). Modelling-based Teaching in Science Education. Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-319-29039-3 Glaser, B. G. (1978). Theoretical sensitivity: Advances in the methodology of grounded theory. MillValley, CA: Sociology Press. 126 Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory. Chicago, IL: Aldine. Gobert, J. D., & Buckley, B. C. (2000). Introduction to model-based teaching and learning in science education. International Journal of Science Education, 22(9), 891-894. https://doi.org/10.1080/095006900416839 Greene, B. A., & Miller, R. B. (1996). Influences on achievement: Goals, perceived ability, and cognitive engagement. Contemporary Educational Psychology, 21(2), 181-192. Greene, B. A. (2015). Measuring cognitive engagement with self-report scales: Reflections from over 20 years of research. Educational Psychologist, 5(1), 14-30. https://doi.org/10.1080/00461520.2014.989230 Greene, B. A., Miller, R. B., Crowson, H. M., Duke, B. L., & Akey, K. L. (2004). Predicting high school student’s cognitive engagement and achievement: Contributions of classroom perceptions and motivation. Contemporary Educational Psychology, 29(4), 462-482. https://doi.org/10.1016/j.cedpsych.2004.01.006 Grimaldi, P. J., & Karpicke, J. D. (2014). Guided retrieval practice of educational materials using automatic scoring. Journal of Educational Psychology, 106, 58-68. https://doi.org/10.1037/a0033208 Goel, A., & Stroulia, E. (1996). Functional device models and model-based diagnosis in adaptive design. Artificial Intelligence for Engineering, Design, Analysis and Manufacturing, 10(4), 355-370. Grosse, C., & Renkl, A. (2004). Learning from worked examples: What happens if errors are included? In P. Gerjets, J. Elen, R. Joiner, & P. Kirschner (Eds), Instructional design for effective and enjoyable computer-supported learning. Tuebingen: Knowledge Media Research Center, pp. 356-364. Gunawan, G., Harjono, A., Susilawati, S., & ShintaMultiara, D. (2019). Generative learning models assisted by virtual laboratories to improve students’ creativity in physics. Journal of Advanced Research in Dynamical and Control Systems, 11(7), 403-411. Halloun, I. A., (2007). Mediated modeling in science education. Science and Education, 16(7-8), 653-697. https://doi.org/10.1007/s11191-006-9004-3 Hattie, J. A., C., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of Educational Research, 66(2), 99-136. https://doi.org/10.3102/00346543066002099 Helme, S., & Clarke, D. (2001). Identifying cognitive engagement in the mathematics classroom. Mathematics Education Research Journal, 13(2), 133-153. https://doi.org/10.1007/BF03217103 127 Hmelo-Silver, C. E. (2007). Fish swim, rock sit, and lungs breathe: Expert-novice understanding of complex systems. Journal of the Learning Sciences, 16(3), 307-331. https://doi.org/10.1080/10508400701413401 Hmelo-Silver, C. E., & Pfeffer, M. G. (2004). Comparing expert novice understanding of complex system from the perspective of structures, behaviors, and functions. Cognitive Science, 28(1), 127-138. https://doi.org/10.1207/s15516709cog2801_7 Hofkens, T. L., & Ruzek, E. (2019). Measuring student engagement to inform effective interventions in schools. In J. A. Fredricks, A. L. Reschly, S. L. Christenson, (Eds.). Handbook of Student Engagement Interventions (pp. 309-324). https://doi.org/10.1016/B978-0-12-813413-9.00021-8 Huff, J.D. & Nietfeld, J. L. (2009). Using strategy instruction and confidence judgments to improve metacognitive monitoring. Retrieved November 21, 2022 from http://findarticles.com/p/articles/mi_m1310/is_1990_Dec/ai_9339028/ Hunter, A-B. (2019). Why undergraduates leave stem majors: changes over the last two decades. In E. Seymour & A-B Hunter (Eds.) Talking about leaving revisited: persistence, relocation and loss in undergraduate stem education, pp. 87-114. https://doi.org/10.1007/978-3-030-25304-2_3 Jacobs, J. E., & Paris, S. G., (1987). Childrens’ metacognition about reading- issues in definition, measurement, and instruction. Educational Psychology, 22, 255-278. https://doi.org/10.1080/00461520.1987.9653052 Järvelä, S., & Renninger, K. A. (2014). Designing for learning: Interest, motivation, and engagement. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (2nd ed., pp. 668–685). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.040 Jonassen, D.H., & Land, S.M. (Eds.). (2012). Theoretical foundations of learning environments. New York: Routledge Jordan, M. E. (2010). Collaborative robotics design projects: Managing uncertainty in multimodal literacy practice. Yearbook of the National Reading Conference. Jordan, R. C., Hmelo-Silver, C., Liu, L., & Gray, S. A. (2013). Fostering reasoning about complex systems: using the aquarium to teach systems thinking. Applied Environmental Education & Communication, 12(1), 55-64. https://doi.org/10.1080/1533015X.2013.797860 Justi, R. S., & Gilbert, J. K. (2002a). Science teachers’ knowledge about and attitudes toward the use of models and modelling in learning science. International Journal of Science Education, 24(12), 1273-1292. https://doi.org/10.1080/09500690210163198 128 Justi, R. S., & Gilbert, J. K. (2002b). Models and modelling in chemical education. In: Gilbert, J. K., De Jong, O., Justi, R., Treagust, D. F., Van Driel, J. H. (eds). Chemical Education: Towards Research-based practice. Science & Technology Education Library (Vol. 17). Springer: Dordrecht. https://doi.org/10.1007/0-306-47977-X_3 Kahn, S. (2011). What’s missing in model-based teaching. Journal of Science Teacher Education, 22(6), 535-560. https://doi.org/10.1007/s10972-011-9248-x Karpicke, J. D. (2009). Metacognitive control and strategy selection: deciding to practice retrieval during learning. Journal of Experimental Psychology General, 138, 469-486. https://doi.org/10.1037/a0017341 Karpicke, J. D., & Grimaldi, P. J. (2012). Retrieval-based learning: a perspective for enhancing meaningful learning. Educational Psychology Review, 24, 401-418. https://doi.org/10.1007/s10648-012-9202-2 Kim, Y. R., Park, M. S., Moore, T. J., & Varma, S. (2013). Multiple levels of metacognition and their elicitation through complex problem-solving tasks. The Journal of Mathematical Behavior, 32(3), 377-396. https://doi.org/10.1016/j.jmathb.2013.04.002 Kintsch, W. (1970). Models for free recall and recognition. In D. A. Norman (Ed.), Models of human memory (pp. 331-373). https://doi.org/10.1016/b978-0-12-521350-9.50016-4 Kisac, I., & Budak, Y. (2014). Metacognitive strategies of the university students with respect to their perceived self-confidence levels about learning. Procedia-Social and Behavioral Sciences, 116, 3336-3339. Kramarski, B., & Dudai, V. (2009). Group-metacognitive support for online inquiry in mathematics differential self-questioning. Journal of Educational Computing Research, 40(4), 377-404. http://dx.doi.org/10.2190/EC.40.4.a Kramarski, B., & Mevarech, Z. R. (2003). Enhancing mathematical reasoning in the classroom: The effects of cooperative learning and metacognitive training. American Educational Research Journal, 40(1), 281-310. https://doi.org/10.3102/00028312040001281 Krell, M., Upmeier zu Belzen, A., and Krüger, D. (2013). Students’ understanding of the purpose of models in different biological contexts. International Journal of Biology Education, 3(1a), 1-34. Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134. https://doi.org/10.1037/0022-3514.77.6.1121 King, A. (1991). Improving lecture comprehension: Effects of a metacognitive strategy. Applied Cognitive Psychology, 5(4), 331-346. https://doi.org/10.1002/acp.2350050404 129 Lee. S. W. Y., Chang, H. Y., & Wu, H. K. (2017). Students’ views of scientific models and modeling: do representational characteristics of models and students’ educational levels matter? Research in Science Education, 47(2), 305-328. https://doi.org/10.1007/s11165-015-9502-x Lei, H., Cui, Y., & Zhou, W. (2018). Relationships between student engagement and academic achievement: A meta-analysis. Social Behavior and Personality: an International Journal, 46(3), 517-528. https://doi.org/10.2224/sbp.7054 Lesh, R., & Zawojewski, J. S. (2007). Problem solving and modeling. In F. K. Lester, Jr. (Ed.), Second handbook of research on mathematics teaching and learning (pp. 763–804). Reston, VA: National Council of Teachers of Mathematics. Lewis, A. B. (1989). Training students to represent arithmetic word problems. Journal of Educational Psychology, 81(4), 521-531. https://doi.org/10.1037/0022-0663.81.4.521 Li, S. (2021). Measuring cognitive engagement: An overview of measurement instruments and techniques. International Journal of Psychology and Educational Studies, 8(3), 63-76. https://dx.doi.org/10.52380/ijpes.2021.8.3.239 Lindlof, T. R., & Taylor, B. C. (2002). Qualitative Research Communication Methods. (2nd Ed.).California: Sage Publication. Long, T. M., Dauer, J. T., Kostelnik, K. M., Momsen, J. L., Wyse, S. A., Bray Speth, E., & Ebert-May, D. (2014). Fostering ecoliteracy through model-based instruction. Frontiers in Ecology and the Environment, 12(2), 138-139. https://doi.org/10.1890/1540-9295-12.2.138 Louca, L. T., & Zacharia, Z. C. (2012). Modeling-based learning in science education: Cognitive, metacognitive, social, material and epistemological contributions. Educational Review, 64(4), 471-492. https://doi.org/10.1080/00131911.2011.628748 Lytle, A., & De Rosa, A. J., & Fisher, F. T. (2021, November), A Review of Psychosocial Factors Associated with Undergraduate Engagement and Retention in STEM. Paper presented at 2021 Fall ASEE Middle Atlantic Section Meeting, Virtually Hosted by the section. Retrieved from: https://peer.asee.org/a-review-of-psychosocial-factors- associated-with-undergraduate-engagement-and-retention-in-stem Maehr, M. L., & Meyer, H. A. (1997). Understanding motivation and schooling: Where we've been, where we are, and where we need to go. Educational psychology review, 9, 371-409. https://doi.org/10.1023/A:1024750807365 Manthey, S., Brewe, E. (2013). Toward university modeling instruction- biology: adaption curricular frameworks from physics to biology. CBE- Life Sciences Education, 12, 206-214. https://doi.org/10.1187/cbe.12-08-0136 130 Martin, A. J., Ginns, P., Papworth, B. (2017). Motivation and engagement: Same or different? Does it matter? Learning and Individual Differences, 55, 150-162. https://doi.org/10.1016/j.lindif.2017.03.013 Martin, B. L., Mintzes, J. J., & Clavijo, I. E. (2000). Restructuring knowledge in biology: Cognitive processes and metacognitive reflections. International Journal of Science Education, 22(3), 303-323. https://doi.org/10.1080/095006900289895 Masson, S., Potvin, P., Riopel, M., & Brault Foisy, L.M. (2014). Differences in brain activation between novices and experts in science during a task involving a common misconception in electricity. Mind, Brain, and Education, 8 (1), 44 – 55. https://doi-org.proxy2.cl.msu.edu/10.1111/mbe.12043 Mayer, R. E. (2010). Merlin C. Wittrock’s enduring contributions to the science of learning. Educational Psychologist, 45(1), 46-50. https://doi.org/10.1080/00461520903433547 Mayring, P. (2000). ‘Qualitative content analysis’. Forum: Qualitative Social Research, 1, 1-10. Meece, J., Blumenfeld, P. C., & Hoyle, R. H. (1988). Students’ goal orientation and cognitive engagement in classroom activities. Journal of Educational Psychology, 80, 514-523. https://doi.org/10.1037/0022-0663.80.4.514 Meeuwisse, M., Severiens, S. E., & Born, M. Ph. (2010). Learning environment, interaction, sense of belonging and study success in ethnically diverse student groups. Research in Higher Education, 51, 528-545. http://dx.doi.org/10.1007/s11162-010-9168-1 Megaldi, D., & Berler, M. (2020). Semi-structured interviews. In. V. Zeigler-Hill & T.K. Shackelford (Eds.) Encyclopedia of Personality and Individual Differences. Springer, Cham. https://doi.org/10.1007/978-3-319-24612-3_857 Melis, E., Sander, A., & Tsovaltzi, D. (2010, November). How to support meta-cognitive skills for finding and correcting errors? In Paper presented at the 2010 AAAI Fall Symposium. Arlington: Cognitive and Metacognitive Educational Systems. Meijer, J., Veenman, M. V. J., & van Hout-Wolters, B. (2006). Metacognitive activities in text-studying and problem-solving: Development of a taxonomy. Educational Research and Evaluation, 12(3), 209-237. https://doi.org/10.1080/13803610500479991 Metcalfe, J. (2009). Metacognitive judgements and control of study. Current Directions in Psychological Science, 18, 159-163. https://doi.org/10.1111/j.1467-8721.2009.01628.x Michalsky, (2013). Integrating skills and wills instruction in self-regulated science text reading for secondary students. International Journal of Science Education, 35(11), 1846-1873. https://doi.org/10.1080/09500693.2013.805890 131 Mih, V., & Mih, C. (2013). Perceived autonomy-supportive teaching, academic self-perceptions and engagement in learning: toward a process model of academic achievement. Cognition, Brain, Behavior, An Interdisciplinary Journal, 17(1), 35-60. Miller, R. B., Greene, B. A., Montalvo, G. P., Ravindran, B., & Nichols, J. D. (1996). Engagement in academic work: The role of learning goals, future consequences, pleasing others and perceived ability. Contemporary Educational Psychology, 21, 388-422. http://dx.doi.org/10.1006/ ceps.1996.0028 Minner, D. D., Levy, A. J., & Century, J. (2010). Inquiry-based science instruction- what is it and does it matter? Results from a research synthesis years 1984-2002. Journal for Research in Science Teaching, 47, 474-496. https://doi.org/10.1002/tea.20347 Mitchell, I., & Carbone, A. (2011). Typology of task characteristics and their effects on student engagement. International Journal of Educational Research, 50(5-6), 257-270. https://doi.org/10.1016/j.ijer.2011.05.001 Mokos, E., & Kafoussi, S. (2013). Elementary students’ spontaneous metacognitive functions in different types of mathematical problems. Journal of Research in Mathematics Education, 2(2), 242-267. https://doi.org/10.4471/redimat.2013.29 Momsen, J., Offerdahl, E., Kryjevskaia, M., Montplaisir, L., Anderson, E., & Grosz, N. (2013). Using assessments to investigate and compare the nature of learning in undergraduate science courses. CBE- Life Sciences Education, 12(2), 239-249. https://doi.org/10.1187/cbe.12-08-0130 Moreira, B. F. T., Pinto, T. S. S., Starling, D. S. V., & Jaeger, A. (2019). Retrieval Practice in classroom settings: A review of applied research. Frontiers in Education, 4(5). https://doi.org/10.3389/feduc.2019.00005 Morgan, D. L. (1993). Qualitative content analysis: A guide to paths not taken. Qualitative Health Research, 3, 112-121. https://doi.org/10.1177/104973239300300107 Morin, E. (2014). Complex thinking for a complex world - About reductionism, disjunction and Systemism. Systema: Connecting Matter, Life, Culture and Technology, 2(1), 14-22. National Academies of Sciences Engineering and Medicine. (2018). How people learn II. Washington, DC: National Academies Press. National Center for Education Statistics. (2022). Undergraduate Degree Fields. Condition of Education. U.S. Department of Education, Institute of Education Sciences. Retrieved May 25, 2022, from https://nces.ed.gov/programs/coe/indicator/cta National Research Council. (2010). Expanding underrepresented minority participation: America’s science and technology talent at the crossroads. Washington, DC: NRC. 132 National Research Council. (2012). Discipline-based education research: Understanding and improving learning in undergraduate science and engineering. Washington, DC: The National Academy Press. National Science Board (2020). The State of U. S. Science and Engineering 2020. Alexandria, VA: National Science Foundation. Retrieved May 25, 2022, from: https://ncses.nsf.gov/pubs/nsb20201/u-s-and-global-education Newmann, F.M., Wehlage, G.G. and Lamborn, S.D. (1992). The Significance and Sources of Student Engagement. In: F.M. Newman (Ed.), Student Engagement and Achievement in American Secondary Schools, Teachers College Press, New York, 11-39. Nguyen, T. D., Cannata, M., & Miller, J. (2016). Understanding student behavioral engagement: Importance of student interaction with peers and teachers. The Journal of Educational Research, 111(2), 163-174. https://doi.org/10.1080/00220671.2016.1220359 Norris, & Phillips, (2012). Reading science: How a naive view of reading hinders so much else. In A. Zohar & Y. J. Dori (Eds.), Metacognition in science education (pp. 37-56). Dordrecht: Springer. O’Connor, C., and Joffe, H. (2020). Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines. International Journal of Qualitative Methods 19, 1–13. https://doi.org/10.1177/1609406919899220 Ohlsson, S. (1996). Learning from performance errors. Psychological Review, 103, 241-262. https://doi.org/10.1037/0033-295X.103.2.241 Osbeck. L. M., & Nersessian, N. J. (2006). The distribution of representation. Journal for the Theory of Social Behaviour, 32(2), 141-160. Palincsar, A. S. & Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 1(2), 117-175. https://doi.org/10.1207/s1532690xci0102_1 Pandero, E. (2017). A review of self-regulated learning: six models and four directions for research. Frontiers in Educational Psychology, 8(422), 1-28. https://doi.org/10.3389/fpsyg.2017.00422 Parong, J., & Mayer, R. E. (2018). Learning science in immersive virtual reality. Journal of Educational Psychology, 110(6), 785. https://dx.doi.org/10.1037/edu0000241 Passmore, C., Gouvea, J. S., & Giere, R. (2014). Models in science and in learning science: focusing scientific practice on sense-making. In M. Matthews (Ed.), International handbook of research in history, philosophy and science teaching (pp. 1171-1202). Dordrecht: Springer Netherlands. https://doi.org/10.1007/978-94-007-7654-8_36 133 Patall, E. A., Vasquez, A. C., Steingut, R. R., Trimble, S. S., & Pituch, K. A. (2016). Daily interest, engagement, and autonomy support in the high school science classroom. Contemporary Educational Psychology, 46, 180-194. https://doi.org/10.1016/j.cedpsych.2016.06.002 Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 259-282). New York, NY: Springer. https://doi.org/10.1007/978-1-4614-2018-7_12 Peper, R. J., & Mayer, R. E. (1986). Generative effects of note-taking during science lectures. Journal of Education and Psychology, 78(1), 34-38. https://doi.org/10.1037/0022-0663.78.1.34 Perry, B. (2013). Impact of a metacognitive intervention in a pre-nursing course. Ph. D Thesis, Baton Rouge, Louisiana: Louisiana State University. Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching and assessing. Theory into Practice, 41(4), 219-225. https://doi.org/10.1207/s15430421tip4104_3 Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16, (385-407). https://doi.org/10.1007/s10648-004-0006-x Pintrich, P. R. & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82, 33-40. http://dx.doi.org/10.1037//0022- 0663.82.1.33 Postareff, L., Parpala, A., & Lindblom-Ylänne, S. (2015). Factors contributing to changes in a deep approach to learning in different learning environments. Learning Environments Research, 18(3), 315-333. https://doi.org/10.1007/s10984-015-9186-1 President’s Council of Advisors on Science and Technology [PCAST]. (2012, February). Engage to excel Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics, Washington, DC: US: Government Office of Science and Technology (Report to the President), Retrieved from: https://eric.ed.gov/?id=ED541511 Ralph, V. R., Scharlott, L. J., Schwarz, C. E., Becker, N. M., & Stowe, R. L. (2022). Beyond instructional practices: Characterizing learning environments that support students in explaiing chemical phenomena. Journal of Research in Science Teaching, 59(5), 841-875. https://doi.org/10.1002/tea.21746 Reeve, J. (2013). How students create motivationally supportive learning environments for themselves: The concept of agentic engagement. Journal of Educational Psychology, 105(3), 579-595. https://doi.org/10.1037/a0032690 134 Reinagel, A., Bray Speth, E. (2016). Beyond the central-dogma: model-based learning of how genes determine phenotype. CBE- Life Sciences Education, 15(1), 1-13. https://doi.org/10.1187/cbe.15-04-0105 Reschly, A. L., & Christenson, S. L. (2006). Prediction of dropout among students with mild disabilities: A case for inclusion of student engagement variables. Remedial and Special Education, 27, 276-292. https://doi.org/10.1177/07419325060270050301 Reschly, A. L., & Christenson, S. L. (2012). Jingle jangle, and conceptual haziness: evolution and future directions of the engagement construct. In S. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp.3-20). Springer. Richardson, J. C., & Newby, T. (2006). The role of students’ cognitive engagement in online learning. American Journal of Distance Education, 20(1), 23-37. https://doi.org/10.1207/s15389286ajde2001_3 Roediger, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences, 15(1), 20-27. https://doi.org/10.1016/j.tics.2010.09.003 Roediger, H. L., & Guynn, M. J. (1996). Retrieval processes. In E. L. Bjork & R. A. Bjork (Eds.), Memory (pp. 197- 236). San Diego: Academic. Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17, 249-255. https://doi.org/10.1111/j.1467-9280.2006.01693.x Roediger, H. L., Agarwal, P. K, McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in the classroom: long-term improvements from quizzing. Journal of Experimental Psychology: Applied, 17(4), 382-395. https://doi.org/10.1037/a0026252 Ross, S. M. & Kirby, F. J. (1976). Oral summary as a review strategy for enhancing recall of textual material. Journal of Education and Psychology, 68(6), 686-695. https://doi.org/10.1037/0022-0663.68.6.689 Rotgans, J. I., & Schmidt, H. G. (2011). Cognitive engagement in the problem-based learning classroom. Advances in Health Sciences Education, 16(4), 465-479. https://doi.org/10.1007/s10459-011-9272-9 Sandi-Urena, S., Cooper, M. M., & Stevens, R. H. (2011). Enhancement of metacognition use and awareness by means of a collaborative intervention. International Journal of Science Education, 33, 323-340. https://doi.org/10.1080/09500690903452922 135 Scardamalia, M., Bereiter, C., & Steinbach, R. (1984). Teachability of reflective processes in written composition. Cognitive Science, 8, 173-190. https://doi.org/10.1016/S0364-0213(84)80016-6 Schelling, G. & Van Hout-Wolters, B. (2011). Measuring strategy use with self-report instruments: Theoretical and empirical considerations. Metacognition and Learning, 6(2), pp. 83-90. https://doi.org/10.1007/s11409-011-9081-9 Schnitzler, K., Holzberger, D., Seidel, T. (2020). All better than being disengaged: Student engagement patterns and their relations to academic self-concept and achievement. European Journal of Psychology and Education, 36, 627-657. https://doi.org/10.1007/s10212-020-00500-6 Schoenfeld, A. H. (1992). Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics. In D. A. Grouws (Ed.), Handbook of research on mathematics teaching and learning (pp. 165-197). New York: Macmillan. Schӧnborn, K. J. (2005). Using student difficulties to identify and model factors influencing the ability to interpret external representations of IgG-antigen binding. Ph.D. thesis, University of KwaZulu-Natal, Pietermaritzburg, South Africa. Schraw, G., Crippen, K. J., & Hartley, K. (2006). Promoting self-regulation in science education: metacognition as part of a broader perspective on learning. Research in Science Education, 36, 111-139. https://doi.org/10.1007/s11165-005-3917-8 Schraw, G., & Moshman, D. (1995). Metacognitive theories. Educational Psychology Review, 7, 351-371. https://doi.org/10.1007/s11165-005-3917-8 Schunk, D. H., & Mullen, C. A. (2012). Self-efficacy as an engaged learner. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 219-235). New York, NY: Springer. Schwarz, C. V., Reiser, B. J., Davis, E. A., Kenyon, L., Archér, A., Fortus, D., Shwartz, Y., Hug, B., & Krajcik, J. (2009). Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners. Journal of Research in Science Teaching, 46(6), 632-654. https://doi.org/10.1002/tea.20311 Sedaghat, M., Abedin, A., Hejazi, E., & Hassanabadi, H. (2011). Motivation, cognitive engagement, and academic achievement. Procedia-Social and Behavioral Sciences, 15, 2406-2410. https://doi.org/10.1016/j.sbspro.2011.04.117 Senay, I., Albarracín, D., & Noguchi, K. (2010). Motivating goal-directed behavior through introspective self-talk: the role of the interrogative form of simple future tense. Psychological Science, 21(4), 499-504. ttps://doi.org/10.1177/0956797610364751 136 Seymour, E., & Hewitt, N. M. (1997). Talking about leaving: Why undergraduates leave the sciences. Boulder, CO: Westview Press. Seymour, E., Hunter, AB., Weston, T.J. (2019). Why We Are Still Talking About Leaving. In: Seymour, E., Hunter, AB. (Eds) Talking about Leaving Revisited. Springer, Cham. https://doi.org/10.1007/978-3-030-25304-2_1 Shannon, S. V. (2008). Using metacognitive strategies and learning styles to create self-directed learners. Institute for Learning Styles Journal, 1(1), 14-28. Shernoff, D. J. (2013). Optimal learning environments to promote student engagement. New York, NY: Springer. Silver, E. A. (1979). Student perceptions of relatedness among mathematical verbal problems. Journal for Research in Mathematics Education, 10(3), 195-210. https://doi.org/10.5951/jresematheduc.10.3.0195 Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). The challenges of defining and measuring engagement in science. Educational Psychologist, 50(1), 1-13. https://doi.org/10.1080/00461520.2014.1002924 Sithole, A., Chiyaka, E.T., McCarthy, P., Mupinga, D. M., Bucklein, B. K., Kibrige, J. (2017). Student attraction, persistence and retention in STEM programs: successes and continuing challenges. Higher Education Studies, 7(1), 46-59. Skinner, E. A., Kindermann, T. A., Connell, J. P., & Wellborn, J. G. (2009). Engagement as an organizational construct in the dynamics of motivational development. In K. Wentzel & A. Wigfield (Eds.), Handbook of motivation at school (pp. 223-245). Malwah, NJ: Erlbaum. Stanton, J. D., Neider, X. N., Gallegos, I. J., & Clark, N. C. (2015). Differences in metacognitive regulation introductory biology students: when prompts are not enough. CBE-Life Sciences Education, 14 (15), 1-12. https://doi.org/10.1187/cbe.14-08-0135 Stephanou, G., & Mpiontini, M. (2017). Metacognitive knowledge and metacognitive regulation in self-regulatory learning style, and its effects on performance expectation and subsequent performance across diverse school subjects. Psychology, 8, 1941-1975. Stefanou, C.R., Salisbury-Glennon, J.D. (2002). Developing Motivation and Cognitive Learning Strategies Through an Undergraduate Learning Community. Learning Environments Research, 5, 77–97. https://doi.org/10.1023/A:1015610606945 Tanner, K. D. (2012). Feature approaches to biology teaching and learning promoting student metacognition. CBE-Life Sciences Education, 11, 113-120. https://doi.org/10.1187/cbe.12-03-0033 137 Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of research research. Review of Educational Research, 45, 89-125. http://dx.doi.org/10.3102/00346543045001089 Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago and London: The University of Chicago Press. Tinto, V. (2006). Research and practice of student retention: What next? Journal of College Student Retention, 8, 1-19. https://doi.org/10.2190/4YNU-4TMB-22DJ-AN4W Tripto, J., Assaraf, O. B-Z., & Amit, M. (2013). Mapping what they know: concept maps as an effective tool for assessing students’ system thinking. American Journal of Operations Research, 3(1), 245-258. http://www.scirp.org/journal/PaperInformation.aspx?PaperID=27564 Van Hout-Wolters, B. (2000). Assessing active self-directed learning. In R. Simons, J. van der Linden, & T. Duffy (Eds.), New Learning (pp. 83-101). Dordrecht: Kluwer. Vattam, S. S., Goel, A. K., Rugaber, S., Hmelo-Silver, C. E., Gray, S., & Sinha, S. (2011). Understanding Complex Natural Systems by Articulating Structure- Behavior- Function Models. Journal of Educational Technology & Society, 14(1), 66–81. Veenman, M. V. J. (2005). The assessment of metacognitive skills: What can be learned from multimethod designs? In C. Artelt and B. Moschner (Eds.), Lernstrategien und Metakognition: Implikationen für Forschung und Praxis (pp. 75-97). Berlin: Waxman. Veenman, M. V. J., Van Hout-Wolters, B. & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1(1), 3-14. https://doi.org/10.1007/s11409-006-6893-0 Verhoeff, R. P., Boersma, K. T., Waarlo, A., J. (2008). Systems modeling and the development of coherent understanding of cell biology. International Journal of Science Education, 30(4), 543-568. https://doi.org/10.1080/09500690701237780 von Glasersfeld, E., (1983). Learning as a constructive activity. In J. C. Bergeron & N. Herscovics (Eds.), Proceedings of the 5th Annual Meeting of the North American Group of Psychology in Mathematics Education (Vol. 1, pp. 41-101). Montreal. Vorhölter, K., Krüger, A., & Wendt, L. (2019). Metacognition in mathematical modeling - an overview. In S. A. Chamberlin, & B. Sriraman (eds.), Affect in Mathematical Modeling, Advances in Mathematics Education. Springer. https://doi.org/10.1007/978-3-030-04432-9_3 Wang, J. R., & Chen, S. F. (2014). Exploring mediating effect of metacognitive awareness on comprehension of science texts through structural equation modeling analysis. Journal of Research on Science Teaching, 51(2), 175-191. https://doi.org/10.1002/tea.21131 138 Wang, M-T., & Degol, J. (2014). Staying engaged: Knowledge and research needs in student engagement. Child development perspectives, 8(3), 137-143. https://doi.org/10.1111/cdep.12073 Wang, M. T., Fredricks, J. A., Ye, F., Hofkens, T., & Schall Linn, J. (2019). Conceptualization and assessment of adolescents’ engagement and disengagement in school. European Journal of psychological Assessment, 35(4), 592-606. https://doi.org/10.1027/1015-5759/a000431 Wehlage, G. G., & Smith, G. A. (1992). Building new programs for students at risk. In F. M. Newmann (Ed.), Student engagement and achievement in American secondary schools (pp. 92-118). New York: Teachers College Press. Weinstein, C. E., Husman, J., Dierking, D. R. (2000). Self-regulation interventions with a focus on learning strategies. In. M. Boekaerts, P. R. Pintrich, & M. Ziedner (Eds.), Handbook of Self-Regulation (pp. 727-783). Academic Press. Weinstein, C. E., & Meyer, D. K. (1991). Cognitive learning strategies and college teaching. New directions for teaching and learning, 45, 15-26. Williamson, R. A. (1996). Self questioning - an aid to metacognition. Reading Horizons: A Journal of Literacy and Language Arts, 37(1), 30-47. Retrieved from https://scholarworks.wmich.edu/reading_horizons/ vol37/iss1/3 Wilson, K. J., Long, T. M., Momsen, J. L., & Bray Speth, E. (2020). Modeling in the classroom: Making relationships and systems visible. CBE—Life Sciences Education, 19(1), fe1. https://doi.org/10.1187/cbe.19-11-0255 Winne, P. H. (2010). Improving measurements of self-regulated learning. Educational Psychologist, 45, 267-276. Winne, P. H. & Nesbit, J. C. (2010). The Psychology of Academic achievement. Annual Review of Psychology, 61, 653-678. https://doi.org/10.1146/annurev.psych.093008.100348 Wittrock, M. C. (1974). Learning as a generative process. Educational Psychologist, 11(2), 87-95. https://doi.org/10.1080/00461527409529129 Wittrock, M. C. (1992). Generative processes of the brain. Educational Psychologist, 27, 531-541. https://doi.org/10.1207/s15326985ep2704_8 Wittrock, M. C. (1985). Teaching learning generative strategies for enhancing reading comprehension. Theory into Practice, 24(2), 123-126. https://doi.org/10.1080/00405848509543158 139 Wittrock, M. C., & Alesandrini, K. (1990). Generation of summaries and analogies and analytic and holistic abilities. American Educational Research Journal, 27(3), 489-502. https://doi.org/10.3102/00028312027003489 Wong, E. D. (1993a). Self-generated analogies as a tool for constructing and evaluating explanations of scientific phenomena. Journal of Research in Science Teaching, 30, 367-380. https://doi.org/10.1002/tea.3660300405 Wong, E. D. (1993b). Understanding the generative capacity of analogies as a tool for explanation. Journal of Research in Science Teaching, 30, 1259-1272. https://doi.org/10.1002/tea.3660301008 Xu, Y. J. (2016). Attention to Retention: Exploring and addressing the needs of college students in STEM majors. Journal of Education and Training Studies, 4(2), 67-76. Yazzie-Mintz, E., & McCormick, K. (2012). Finding the humanity in the data: Understanding, measuring and strengthening student engagement. In S. L. Christenson, A. L. Reschly, & C. Wiley (Eds.), Handbook of research on student engagement (pp. 743-761). New York, NY: Springer. Yeung, N., & Summerfield, C. (2012). Metacognition in human decision-making: confidence and error monitoring. Philosophical Transactions of the Royal Society B: Biological Sciences, 367(1594), 1310-1321. https://doi.org/10.1098/rstb.2011.0416 Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into practice, 41(2), 64-70. https://doi.org/10.1207/s15430421tip4102_2 Zion, M., Michalsky, T., & Mevarech, Z. R. (2005). The effects of metacognitive instruction embedded within an asynchronous flaring network on scientific inquiry skills. International Journal of Science of Science Education, 27(8), 957-983. https://doi.org/10.1080/09500690500068626 140 Repeat Model Prompt: Cystic Fibrosis APPENDIX Construction Background Cystic fibrosis is the most common lethal inherited disease in Caucasian populations. Cystic fibrosis is caused by a defect in the CFTR gene located on the q arm of Chromosome 7. In healthy individuals, the wild-type allele (R) is dominant and contains the information necessary for producing normal CFTR protein. CFTR is a protein that forms a channel in cell membranes that allows the movement of chloride ions out of cells. As chloride leaves cells, water follows and thins the mucus on cell surfaces, allowing it to flow freely. In individuals with cystic fibrosis, the CFTR proteins are defective and block the flow of chloride ions and water out of cells. The inability to regulate chloride and water results in a drier mucus that is thick and sticky and accumulates on cell surfaces in the lungs, pancreas, digestive tract, and other internal organs. Individuals with cystic fibrosis experience frequent and serious bacterial infections, are unable to absorb adequate nutrients, and have chronic respiratory problems. If untreated, children with cystic fibrosis generally die before 5 years of age. However, daily chest pounding to clear mucus, along with heavy doses of antibiotics and other therapies have extended life expectancy for cystic fibrosis patients into their 20’s and 30’s. Using what you know about how genetic information is organized and expressed, construct a system model that shows how key concepts in this CFTR case work together to determine whether cystic fibrosis or a normal phenotype becomes expressed. Your overall model should show how two different outcomes are possible - cystic fibrosis or normal. R allele, r allele, chromosome 7, DNA, gene, nucleotide sequence, cystic fibrosis, normal (no cystic fibrosis), normal CFTR protein, defective CFTR protein Novel Model Prompt: Carbon Cycling Construction Background In most aquatic environments, carbon (both its form and concentration) is controlled through the microbial biofilms that cover all wet surfaces. A biofilm is a collection of microorganisms living together. These biofilm communities are composed of algae and bacteria. Carbon is transformed through a series of redox reactions that drive energy metabolism within their cells. This changes carbon between its inorganic forms (e.g. CO2) and organic forms (e.g. glucose). The autotrophic organisms, like algae, in these communities use photosynthesis to reduce CO2 into glucose while the heterotrophic organisms, like bacteria (and also algae too) oxidize glucose back into CO2 with respiration. These two coupled metabolic pathways are largely what cycle carbon through the environment. 141 Identify the components needed to create a model whose function would describe the cycling of carbon in a simple aquatic ecosystem. Model components Use the following components to build a system model that describes the cycling of carbon in the previously described simple aquatic ecosystem: ● Algae ● Bacteria ● CO2 ● Glucose 142 CHAPTER THREE: Measuring Emotional Engagement in Modeling (EEM): development and application of an emoji-based EEM scale INTRODUCTION As the number of diverse scientists entering the STEM workforce continues to fall short of goals (e.g., Estrada et al., 2016; Kennedy et al., 2021, National Center for Science and Engineering Statistics [NCSES], 2019), it is imperative that we explore all opportunities for attracting and retaining students - especially those who have been traditionally underrepresented in science careers. Research on student affect in practice-based learning has focused on differences among groups of students in terms of learning and performance outcomes and perceptions of various aspects of their learning (e.g., motivation for learning, confidence, etc.). However, students’ emotions during practice-based learning may be a mechanism that has been historically overlooked (Murphy et al., 2019). Emotions can profoundly impact multiple components of educational settings, such as engagement in and motivation for action, performance outcomes, mental health, career decisions, and dropout rates (e.g., see Barroso et al., 2021 for review; Camacho-Morles et al., 2021; Cheng & McCarthy, 2018; Loukidou et al., 2009). To increase participation in science, we must gain a better understanding of student emotions that are present and persistent within science contexts; particularly those that promote sustained interest and retention, and, equally important, those that can be impediments to learning and discourage engagement in science (Sinatra et al., 2014). Emotions Emotions are generally defined as multi-component affective responses that occur in relation to specific objects or situations (e.g., Gray & Watson, 2001; Pekrun, 2006; Rosenberg, 143 1998; Scherer & Moors, 2019). Emotions play a powerful role in cognitive processes and the way individuals interpret events (Damasio, 1994; Lazarus, 1984). Mulligan and Sherer (2012) consider emotions to be an interface between an organism and its environment that is constantly changing between events and social context, and between the individual’s responses and experiences. Within the classroom, students’ emotions are increasingly recognized as a critical component of their learning, motivation, and achievement (e.g., Boekaerts & Pekrun, 2015; Pekrun et al., 2002; Pekrun et al., 2017; Pekrun et al., 2011; Pekrun & Stephens, 2012). Skinner and Pitzer (2012) emphasize that emotional reactions play a critical role in one’s patterns of actions. For example, even different versions of negative emotions (e.g., boredom, sadness, anxiety, or frustration) may cause a student to proceed differently through a task. Gaining a better understanding of the role of emotions in students’ academic engagement will be beneficial in improving the efficacy of practice-based instruction. Emotional Engagement Emotional engagement is a component of a larger meta-construct, academic engagement, which also consists of cognitive and behavioral engagement (Archambault et al., 2009; Fredricks et al., 2004; Sharkey et al., 2008; Zaff et al., 2011). Cognitive engagement considers students’ personal investment in learning activities, including the use of learning strategies, whereas behavioral engagement entails students’ active participation in activities related to school and learning (Fredricks et al., 2004). Emotional engagement centers around students’ affective responses and includes students’ emotional reactions and attitudes related to academic tasks and settings which engage them in learning (Connell & Wellborn, 1991; Fredricks et al., 2004). Cognitive and behavioral engagement have received the greatest attention in prior research, whereas emotional engagement is notably less explored (Fredricks et al., 2004; Sagayadevan & 144 Jeyaraj, 2012). Despite being studied less, research has demonstrated that emotional engagement is a fundamental component in the learning process (e.g., Appleton et al., 2008; Rocca, 2010; Sansone & Thoman, 2005). The quality of education and classroom settings can significantly impact learning through students' emotions (e.g., Bellocchi et al., 2017; Nicolaou et al., 2015; Rodríguez-Muñoz et al., 2021; Schutz, et al., 2009). Educators can support learners’ engagement, persistence, and performance by creating an emotionally supportive learning environment where students feel safe and valued (National Academies of Sciences, Engineering, and Medicine [NASEM], 2018). Positive emotional engagement can influence students’ willingness to do work (Appleton et al., 2008; Connell & Wellborn, 1991; Finn, 1989; King et al., 2015) and promote positive future orientations as students are thinking about and planning for their future (Crespo et al., 2013). Emotional engagement similarly increases confidence (Sinatra et al., 2015; Ritchie & Tobin, 2018), academic engagement (Ketonen et al., 2019; Ouweneel et al., 2011; Robayo-Tamayo et al., 2020), and performance and achievement (Carmona-Halty et al., 2019; Heddy & Sinatra, 2013; Pekrun & Linnenbrink-Garcia, 2012; Rand et al., 2020). On the other hand, students who experience increased anxiety and other negative emotions in their academic life can become disengaged and are at risk of poor academic outcomes, such as decreased persistence and performance (Archambault et al., 2009; Bledsoe & Baskin, 2014; England et al., 2017; Green et al., 2008; Hirschfield and Gasper, 2011) and lower cognitive engagement in academic work (Broughton et al., 2013; Wang & Holcombe, 2010; Wang & Eccles, 2013). Research suggests emotions can have a discipline-specific component (Goetz, et al., 2006), emphasizing the need for a better understanding of their role in student learning within each domain. Our understanding of emotional engagement in STEM remains limited (Murphy et 145 al., 2019), however, it is known that for students to be successful in STEM they must feel a sense of belonging with their school community and develop positive emotions toward schoolwork (Appleton et al., 2008; Green et al., 2008). STEM disciplines, such as engineering, neuroscience and economics, have identified the importance of emotions in student development, retention, diversity and inclusion.(e.g., Davidson et al., 2020; Hess et al., 2020; Kellam et al., 2018; Lönngren et al., 2020; Pekrun & Linnenbrink-Garcia, 2014; Sinatra et al., 2014; Zembylas & Schutz, 2016), yet, despite theoretical advances and calls for more empirical studies across all fields, there continues to be a lack of research on the role of emotions in biology. Measuring emotional engagement Emotional engagement has been assessed at varying scales, including at the level of the whole classroom learning context (i.e., academic emotions; Gonida et al., 2009; Pekrun et al., 2002), at the level of a particular topic within a domain (i.e., topic emotions; Broughton et al., 2013; Pekrun & Stephens, 2012), and at the level of an object (i.e., either activity-achievement or outcome-achievement emotions; Pekrun, 2006; Pekrun et al., 2002). When considering the type of object, emotions are assessed in relation to either the specific activity (activity emotions) or outcome (outcome emotions) (Pekrun et al., 2002). Activity emotions are most relevant to an ongoing achievement activity, whereas outcome emotions are typically related to past or future outcomes resulting from the activity. For example, a student may find the process of taking an exam enjoyable because the challenge itself is rewarding (activity emotion) regardless of whether they believe they will be successful or not (outcome emotion) (Lumby, 2011). Research on activity emotions in STEM is important, particularly in the context of students’ real-time experiences with learning tasks. Existing research suggests that students 146 experience a complex mix of emotions and affective states as they complete STEM-related tasks, such as problem solving or generating responses to questions (Naibert & Barbera, 2022; Naibert et al., 2022; Blobstein et al., 2022). Gaining a better understanding of the emotions experienced while performing diverse types of STEM learning tasks could inform our design of activities and assessments that best promote positive emotional engagement. Model-based tasks and Modeling-Based Instruction (MBI) Modeling is a foundational scientific practice (Gilbert, 1991; National Research Council [NRC], 2012) defined as the process of building and externalizing mental models (Jonassen & Strobel, 2006; Jonassen et al., 2005; Louca & Zacharia, 2012). Modeling-based instruction (MBI) is an evidence based pedagogical approach that actively engages students in model-based tasks, such as using, constructing, revising, and evaluating scientific models (Clement, 2000; Gilbert & Justi, 2016; Justi & Gilbert, 2002; Long et al., 2014; Louca & Zacharia, 2012; Schwarz et al., 2009). The act of modeling elicits multiple indicators of students’ behavioral and cognitive engagement as they work through and successfully complete model-based tasks (Furqueron, de Lima, and Long, 2023). It seems plausible that as students progress through tasks and express different types of cognitive or behavioral engagement, they are concomitantly experiencing a range of emotions. For example, as a student constructs a model, they may discover and correct an error (cognitive engagement indicator) which produces feelings of joy or pride (emotional indicator) in their performance. In this study, we build upon our prior research investigating linguistic and behavioral indicators of students’ cognitive engagement (Furqueron, de Lima, & Long, 2023) by exploring the emotions students experience during model-based tasks. 147 Challenges to measuring emotional engagement Research on emotions presents many challenges, including construct definition, self- report accuracy, and issues of measurement (see Pekrun & Linnenbrink-Garcia, 2014, for review). Observational measures are generally discouraged when evaluating emotions, as the indicators tend to be internal to the student (Appleton et al., 2006). Indeed, emotional engagement is inherently defined as a latent construct that cannot be observed directly, thus requiring a more intentional approach to its measuring (Pekrun & Linnenbrink-Garcia, 2014). Furthermore, emotions are subjective and can be hard to verbalize and characterize at times (De Angeli et al., 2020; Desmet, Overbeeke, & Tax, 2001; Mehrabian, 1995), and emotional states can be immediate and change rapidly (Borod, 2000; Linnenbrink-Garcia & Pekrun, 2011). To measure latent variables, researchers can operationally define the variable in terms of observable indicators or behaviors, which allows for linking the unobservable variable to an observable and measurable one (Byrne, 1998). One increasingly popular data collection method developed to account for these challenges in measuring emotions is the experience sampling method (ESM; Hektner et al., 2007; Scollon et al., 200). Experience Sampling Method Experience sampling methods (ESMs) permit researchers to examine individuals’ experiences in context and closer to the point of occurrence, allowing for more accurate recall (Csikszentmihalyi & Larson, 1987; Csikszentmihalyi & Csikszentmihalyi, 2006; Sinatra et al., 2015; Zirkel et al., 2015). The characteristic feature of ESM is the repeated measure of an individual’s feelings, thoughts, actions, etc., as they go through an experience. In the past decade, there has been an increase in the application of ESMs and an evolution in the mode of measurement for learning about students’ affective states in educational settings. For example, 148 Nett et al. (2011) conducted an ESM study to evaluate students’ boredom-related coping strategies in mathematics classes by applying self-report measures. Shernoff (2010) applied an ESM to investigate the relationship between student experience in after-school programs and academic achievement. In Shernoff’s study, participants wore digital wristwatches that cued them to log aspects of their ‘in-that-moment’ experience, including components of positive and/or negative affect. Over the past decade, research has utilized the accessibility of mobile devices, including phones, which have become a particularly useful technology for ESM studies (e.g., Xie et al., 2019; Xie et al., 2019). Emojis Emojis are becoming increasingly utilized as a means to evaluate emotions in a wide range of contexts and across diverse modes of communication (Novak et al., 2015). Emoji (from the Japanese e [picture] + moji [character]) is defined as a visual representation of facial expressions, abstract concepts, emotions, gestures, plants, animals, objects, etc. (Rodrigues et al., 2017). For instance, emojis are commonly used to express emotions associated with text or as a substitute for words in instant messages and on social media (Boutlet et al., 2021; Kerslake & Wegerif, 2017). Within the area of customer service relations, emojis are commonly used to assess customer satisfaction in contexts such as the food industry (e.g., Jaeger et al., 2017) and airport travel (e.g., Dickinson, 2018). Additionally, emojis are widely used in medical contexts to improve patient communication on matters such as pain, psychological assessment, and pediatric communication (e.g., Szeto et al., 2022), and are becoming increasingly used to capture visitor emotional responses to museum exhibits (e.g., De Angeli et al., 2020). Emoji use in education research is rare but becoming progressively attractive due to widespread recognition and use of emoji in daily communication and ease of implementation. 149 For example, recent research has explored the role of emojis in course online feedback, including correspondence with the instructor (e.g., Marder et al., 2020) and assessment feedback (Moffitt et al., 2020; Padgett, et al., 2021). In addition, Vareberg et al. (2022) investigated the role of teacher emoji use in a course welcome email on students’ perception of teacher credibility, immediacy, and liking. Within science specifically, Blobstein et al. (2022) used emojis to assess student affective states within forum discussions as part of a general biology course. In Blobstein et al.’s (2022) study, students reported the use of emojis enhanced meaning for the information they were trying to convey and allowed students to express emotions they would not otherwise verbally express. Overall, emojis have shown great potential across a range of contexts for assessing emotional responses, however, they have yet to be applied for the purpose of evaluating students’ engagement in science practice-based learning tasks, such as modeling. Although the use of models and modeling in science is a fundamental practice, and becoming increasingly implemented within the classroom, the way students are engaging with model-based tasks is much less understood. This study aims to fill a gap by developing a tool that can be easily implemented by instructors to identify and assess emotional responses and provides students’ emotional responses to variations of a practice-based assessment. Research Objectives Our study explores the potential for using emojis to assess student emotions while engaged in the scientific, practice-based task of modeling. Specifically, we use this preliminary work to meet two research objectives: (1) Develop an Emotional Engagement in Modeling (EEM) scale for capturing and characterizing the types of emotions students experience during 150 model-based tasks. (2) Use the EEM scale as a research tool for assessing and comparing students’ emotional responses during model-construction and model-evaluation tasks. METHODS Course Description Ten undergraduate students (N=10) at a large, Midwestern university with very high research activity (The Carnegie Classification of Institutions of Higher Education) were recruited from the second of a two-course introductory biology course required for life science majors. The first course is based in cellular and molecular biology, followed by the second course which provides instruction on genetics, evolution, and ecology through MBI. Throughout the course, students were provided multiple opportunities to engage in model-based learning (MBL) through a variety of model-based tasks on assessments including collaborative in-class activities, homeworks, and tests. While enrollment is open to students at any level of their college career, the majority are in their sophomore year. Participants Interview recruiters utilized theoretical sampling (Glaser & Strauss, 1967) to ensure achievement diversity (i.e., grades) in the sample population. Specifically, students were binned into tertiles based on their first-exam score and ten students from each tertile were recruited for the interviews (total of 30 recruits). Interviews were conducted 6-10 weeks post-completion of the Fall 2019 semester (mid- January to early March 2020). Due to the university’s full transition to virtual learning in response to the COVID-19 pandemic, the study was terminated early and only a third of the intended interviews were completed. In total, eleven interviews were completed but only ten were usable due to a technical malfunction during one interview. Of the ten participants, 8 were 151 female, 8 caucasian (non-hispanic), 8 were sophomores, and 2 were first-generation college students (Table 3.1). Table 3.1 identifies students by a pseudonym and includes achievement tertile at the time of recruitment (first exam) and their final course grade (used for post-interview analysis). Table 3.1. Interviewee demographics. Interview participants are identified by a pseudonym. Achievement levels were determined by tertiling students at two timepoints: first exam, used for interview recruitment, and final course grade, used for post-interview analysis. University registrar data provided additional demographic data, including self-identified gender, ethnicity, first-generation college student status, class rank, and declared major. Interview design Students performed in-person, semi-structured, think-aloud interviews using an electronic SmartBoard that recorded modeling activities while also being video- and audio-recorded. Interviews lasted approximately 1-1.5 hours and were conducted in a research lab designed to facilitate in-person interview studies. Two interviewers were present for each interview: one acted as the primary interviewer and the second assisted with note taking, logistics, and 152 occasional questioning. The study was determined exempt by the local Institutional Review Board (IRB #00003353). Modeling tasks Students were asked to complete two types of modeling activities, model construction and model evaluation, in two different contexts, repeat and novel (see Furqueron et al., In prep.). Repeat Model (CFTR): Students constructed a model that repeated a prompt that was previously used on an exam. Specifically, the prompt was designed to assess student understanding of information flow in the context of the genetic disease, cystic fibrosis. For this, students were asked to construct a model that explained the origin of genetic variation at the CFTR gene and how it would ultimately result in expression or non- expression of the cystic fibrosis phenotype. The prompt included a small list of potential model components (e.g., gene, protein, etc.) and students were encouraged to make these specific to the CFTR context and add additional concepts as they saw fit (Appendix, page 174). Once construction was completed, students evaluated their CFTR model by being asked to describe and explain any similarities and differences between their interview- constructed model and their exam model (provided to students by researchers). Novel model (Carbon Cycle): Following evaluation of the CFTR model, students constructed a model that explained carbon cycling in a simple aquatic ecosystem. Carbon cycling was a subject covered during the course and although students had modeled carbon cycling for a variety of systems, the context of the aquatic ecosystem was novel. Background information was provided to re-familiarize students with carbon cycling 153 processes. Students were first prompted to identify and list concepts they believed would be necessary for explaining cycling of carbon in a simple aquatic system (Appendix, page 175). Students were then provided a list of key components, just as they were for the CFTR prompt (Appendix, page 175). This ensured that all students had an equivalent baseline of key concepts for the novel context. Once construction was completed, students evaluated their model by comparing it with an expert-drawn model provided to them (Appendix, page 175). ANALYSIS Measuring student emotional engagement We pre-selected nine emojis that reflected a range of emotions we anticipated students might experience during a learning task (Table 3.2; De Angeli et al., 2020). Because emoji are subject to different interpretations, we asked each student to provide a key word or phrase they associated with each of the nine emojis in the set. Students’ emotions experienced during the tasks were assessed retrospectively, immediately after each modeling task using the self-report EEM scale. Students were asked to verify their interpretation of each emoji, whether discrete (a single emoji) or multiple, they selected. Similar to Novak et al.’s (2015) work generating an Emoji Sentiment Map, students’ emotions associated with each emoji were used to generate a scale of positive to negative emotions (Table 3.2). The ordination of emotions affirmed by students in our study is consistent with previous research (e.g., Novak et al., 2015). 154 Table 3.2. Nine emoji and associated emotion on a scale from green (positive), yellow (neutral), to red (negative) in the EEM. 155 RESULTS In total, 29 discrete and 11 multiple-emoji selections were made (Table 3.3). Students reported primarily positive emotions for both constructing and evaluating in both repeat and novel contexts. Feelings of contentment (38%) and happiness (31%) were the most frequently reported emotions across task types and contexts. Feelings of happiness, contentment, and confusion were the only emotions selected in both tasks and in both contexts. Surprisingly, students reported limited negative emotions, as overwhelmed/nervous was only selected twice (7%). Students did not report feeling tired/bored, or sad/discouraged. Frustration was selected by one student in combination with other emojis. In total, students made 11 multiple-emoji selections, representative of mixed emotions and complex affective states (Table 3.4). The data show that constructing and evaluating the novel model, composed of familiar biological concepts in an unfamiliar (novel) context, elicited more mixed emotions (73%) compared to the repeat model (27%), and that across tasks and contexts students expressed a variety of mixed emotional states - from consisting of multiple negative emotions (n=1), multiple positive emotions (n=1), and a combination of negative, neutral, and positive emotions (n=9). 156 Table 3.3. Student Emoji Selection by Task and Context. Single emoji (n=29) represent discrete emotion selection where multiple emoji (n=11) selection reflect mixed emotions experienced during construction and evaluation of repeat and novel models. 157 Table 3.4. Associated Student Response for Multiple Emoji Selection. Students’ 11 multiple emoji selections and associated response experienced during model construction and evaluation of repeat and novel contexts. 158 LIMITATIONS This interview study was conducted with a limited sample of undergraduate students from an MBI-based introductory biology course for life science majors. We recognize that these findings may not be generalizable across domains or more diverse model-based tasks, or to larger, more diverse student populations, including those from upper-level or non-majors’ courses. Additional research and application of the EEM scale in multiple disciplinary contexts is necessary for gaining a better understanding of students’ emotional engagement in modeling. Interview studies are recognized as being innately limited due to their non-natural settings and potential bias in student responses due to interviewer presence (Creswell & Creswell, 2018). Researchers attempted to mitigate this bias by creating a welcoming and relaxing environment for students, however we must consider that students may still experience a hesitation to expose certain emotions in public (Blobstein et al., 2022) and this may account for a larger than expected proportion of positive emotions reported. DISCUSSION AND CONCLUSION This work contributes research on emotional engagement during STEM learning, particularly in the unexamined setting of model-based learning. Our study design is novel and aims to provide additional insights into students’ emotional engagement through the use of an emoji-based EEM scale as a relatable, intentional, and individualistic approach for measuring students’ discrete and mixed-emotions in real-time. We envision the EEM scale to be broadly applicable as a tool for educators to gain real-time feedback about students’ emotional states in diverse contexts, including during or following lessons, activities, or high-stakes assessments. The EEM could be easily adapted for use with technology, such as personal response systems or polling softwares, and for multiple disciplinary contexts. Continuing work on understanding the 159 range of students’ emotions as they learn and perform academic tasks will be useful for informing the design of instruction that promotes meaningful engagement in both the content and competencies expected of aspiring STEM learners. Although research on activity emotions has expanded, research examining more than just a singular emotion (e.g., enjoyment, anger, and boredom) remains limited (e.g., Lichtenfeld et al., 2012; Pekrun et al., 2023). Indeed, students can simultaneously experience a wide range of affective responses in the form of emotions to learning tasks which, in turn, can have a profound effect on their learning (Boekaerts & Pekrun, 2015). Our study finds that students experience a range of discrete and complex emotions from negative (i.e., frustration, feeling overwhelmed) to positive (i.e., happy, relieved, feeling good) while performing model-based tasks. Students in our study made more discrete selections of positive emotions (i.e., happy or contempt) than negative (i.e., frustrated), which can suggest greater levels of interest (Ainley, 2018) and overall be beneficial for learning (Pekrun et al., 2017). Our data also show students experienced a variety of mixed emotional states - from consisting of multiple negative emotions, multiple positive emotions, and a range of negative to positive emotions. This finding is consistent with research that suggests students can experience mixed feelings while engaging in a learning experience (e.g., Jarrell, et al., 2016; Karamarkovich & Rutherford, 2021; Robinson et al., 2017; Robinson et al., 2020). Of particular interest in our results is students’ expression of confusion, either as a discrete feeling or part of a complex affective state, across both tasks and contexts. Literature suggests confusion may serve as an impetus for engagement and positive learning outcomes in complex learning activities (D’Mello et al., 2014), thus exploring the role of confusion in performance outcomes on model-based tasks warrants further investigation. 160 Previous research suggests that students with positive emotions have the highest achievement (e.g., Karamarkovich & Rutherford, 2021; Wigfield et al., 2020), and that students with lower prior achievement may experience greater negative emotions (Karamarkovich & Rutherford, 2021; Pekrun, 2006; Pekrun et al., 2011). In our study, achievement level did not predict emotional states. Our evidence suggests that high- and middle-achieving students were more likely to express a range of emotions (both positive and negative), whereas the two lower- achieving students in our study were more consistently positive. Future applications of the EEM scale could be adapted for larger-scale studies that further explore relationships between achievement level and emotional engagement. Our research contributes to the importance of context effects in assessment design and their impact on student emotions (e.g., Chen & Nieminen, 2024). Although contextual differences were not the explicit goal of this study, our study suggests that constructing or evaluating novel models resulted in a wider range of mixed emotions compared to the repeat model. This work builds from research investigating students’ cognitive engagement in modeling-based tasks (Furqueron et al., in prep) and on students’ motivational profiles in an introductory biology course taught through MBI (Furqueron & Long, in prep). Future research could examine the EEM scale in relation to students’ cognitive engagement (evidenced through cognitive strategy use) and motivational profiles (including students’ learning-oriented goals) to better understand relationships between patterns of emotion on students’ motivation and subsequent action. For example, are positive emotions more likely to promote sustained engagement in tasks and increase students’ use of diverse strategies for problem solving and task completion? Gaining a better understanding of the circumstances that give rise to certain 161 emotions, and the consequences of those emotions on students’ actions, can be useful to researchers and educators interested in improving students’ success in science. Data about students’ emotions may prove another source of evidence that can inform the design of assessments and targeted interventions that promote persistence and diversity among STEM learners. 162 REFERENCES Ainley, M. (2018). Students’ interest and engagement in classroom activities. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 283–302). Springer. https://doi.org/10.1007/978-1-4614-2018-7 Appleton, J. J., Christenson, S. L., Kim, D., Reschly, A. L. (2006). Measuring cognitive and psychological engagement: validation of the student engagement instrument. Journal of School Psychology, 44(5), 427-445. https://doi.org/10.1016/j.jsp.2006.04.002 Appleton, J. J., Christenson, S. L., & Furlong, M. J. (2008). Student engagement with school: critical conceptual and methodological issues of the construct. Psychology in the Schools, 45(5), 369-386. https://doi.org/10.1002/pits.20303 Archambault, I., Janosz, M., Fallu, J., & Pagani, L. S. (2009). Student engagement and its relationship with early high school dropout. Journal of Adolescence, 32, 651–670. https://doi.org/10.1016/j.adolescence.2008.06.007 Barroso, C., Ganley, C. M., McGraw, A. L., Geer, E. A., Hart, S. A., & Daucourt, M. C. (2021). A meta-analysis of the relation between math anxiety and math achievement. Psychological Bulletin, 147(2), 134–168. https://doi.org/10.1037/bul0000307 Bellocchi, A., Quigley, C., & Otrel-Cass, K. (2017). Exploring Emotions, Aesthetics and Wellbeing in Science Education Research. New York, NY: Springer. Bledsoe, T. S., & Baskin, J. J. (2014). Recognizing student fear: The elephant in the classroom. College Teaching, 62, 32-41. https://doi.org/10.1080/87567555.2013.831022 Blobstein, A., Gal, K., Karger, D., Faccoitti, M., Kim, H., Almahmound, J., & Sripathi, K. (2022). #lets-discuss: Analyzing student affect in course forums using emoji. In A. Mitrovic & N. Bosch (Eds.), Proceedings of the 15th International Conference on Educational Data Mining, pp. 339-345. Durham, United Kingdom, July 2022. Boekaerts, M. & Pekrun, R. (2015). Emotions and emotion regulation in academic settings. In Handbook of Educational Psychology. (pp. 90-104). Routledge. Borod, J. C., (2000). The neuropsychology of emotion. Oxford University Press. Boutlet, I., LeBlanc, M., Chamberland, J. A., Collin, C. A. (2021). Emojis influence emotional communication, social attributions, and information processing. Computers in Human Behavior, 119, 106772. https://doi.org/10.1016/j.chb.2021.106722 Broughton, S. H., Sinatra, G. M., & Nussbaum, E. M. (2013). “Pluto has been a planet my whole life!” Emotions, attitudes, and conceptual change in elementary students’ learning about Pluto’s reclassification. Research in Science Education, 43(2), 529-550. https://doi.org/10.1007/s11165-011-9274-x 163 Byrne, B. M. (1998). Structural equation modeling with LISREL, PRELIS. and SIMPLIS. Mahhaw, NJ: Lawrence Erlbaum Associates. Carmona-Halty, M., Salanova, M., Llorens, S., & Schaufeli, W. B. (2019). How psychological capital mediates between study-related positive emotions and academic performance. Journal of Happiness Studies, 20(2), 605-617. https://doi.org/10.1007/s10902-018-9963-5 Camacho-Morles, J., Slemp, G. R., Pekrun, R., Loderer, K., Hou, H., & Oades, L. G. (2021). Activity achievement emotions and academic performance: A meta-analysis. Educational Psychology Review, 33(3), 1051–1095. https://doi.org/10.1007/s10648-020-09585-3 The Carnegie Classification of Institutions of Higher Education (n.d.). About Carnegie Classification. Retrieved (March 11, 2024) from https://carnegieclassifications.acenet.edu/ Chen, S., & Nieminen, J. H. (2024). Towards an ecological understanding of student emotions in feedback: a scoping review. Assessment & Evaluation in Higher Education, 1–18. https://doi.org/10.1080/02602938.2024.2318641 Cheng, B. H., & McCarthy, J. M. (2018). Understanding the dark and bright sides of anxiety: A theory of workplace anxiety. Journal of Applied Psychology, 103(5), 537–560. https://doi.org/10.1037/apl0000266 Clement, J. J. (2000). Model based learning as a key research area for science education. International Journal of Science Education, 22(9), 1041-1053. http://dx.doi.org/10.1080/095006900416901 Connell, J. P., & Wellborn, J. G. (1991). Competence, autonomy, and relatedness: A motivational analysis of self-system processes. In M. R. Gunnar, L. A. Sroufe (Eds.), Self-processes and development. The Minnesota symposia on child psychology, Vol. 23., (pp. 43-77). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Crespo, C., Jose, P. E., Kielpikowski, M., & Pryor, J. (2013). On solid ground; family and school connectedness promotes adolescents’ future orientation. Journal of Adolescence, 36, 993-1002. https://doi.org/10.1016/j.adolescence.2013.08.004 Creswell, J. W., Creswell, J. D. (2018). Research design: qualitative, quantitative, and mixed methods approaches. (5th ed.). Sage. Csikszentmihalyi, M., & Larson, R. (1987). Validity and reliability of the experience-sampling method. Journal of Nervous and Mental Disease, 175(9), 526–536. https://doi.org/10.1097/00005053-198709000-00004 164 Csikszentmihalyi, M., & Csikszentmihalyi, I. S. (Eds.). (2006). A life worth living: Contributions to positive psychology. Oxford University Press. D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153-170. https://doi.org/10.1016/j.learninstruc.2012.05.003 Davidson, S. G., Jaber, L. Z., & Southerland, S. A. (2020). Emotions in the doing of science: Exploring epistemic affect in elementary teachers’ science research experiences. Science Education, 104(6), 1008–1040. https://doi.org/10.1002/sce.21596 Damasio, A. (1994). Descartes’ error: Emotion, reason, and the human brain. New York, NY: Penguin Books. De Angeli, D., Kelly, R. M., & O’Neill, E. (2020). Beyond happy-or-not: Using emoji to capture visitors’ emotional experience. Curator: The Museum Journal, 63(2), 167-191. https://doi.org/10.1111/cura.12352 Desmet, P., Overbeeke, K., Tax, S. (2001). Designing products with added emotional value: Development and application of an approach for research through design. The Design Journal, 4(1), 32-47. https://doi.org/10.2752/146069201789378496 Dickinson, G. (2018). Those smiley feedback buttons do actually work – and they are changing the way we travel. The Telegraph. Accessed March 11, 2024. Retrieved from https://www.telegraph.co.uk/travel/news/happyornot-smiley-buttons-at-airports/ England, B. J., Brigati, J. R., & Schussler, E. E. (2017). Student anxiety in introductory biology classrooms: perceptions about active learning and persistence in the major. PLoS One, 12, e0182506. https://doi.org/10.1371/journal.pone.0182506 Estrada, M., Burnett, M., Campbell, A. G., Campbell, P. B., Denetclaw, W. F., Gutiérrez, C. G., ... & Zavala, M. (2016). Improving underrepresented minority student persistence in STEM. CBE—Life Sciences Education, 15(3), es5. https://doi.org/10.1187/cbe.16-01-0038 Finn, J. D. (1989). Withdrawing from school. Review of Educational Research, 59,117-142 Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59-109. https://doi.org/10.3102/00346543074001059 Furqueron, B., de Lima, J., & Long, T. (In Prep). Measuring student cognitive engagement in modeling (CEM): Development and application of a CEM framework. Furqueron, B., & Long, T. (In Prep). Exploring motivational profiles in a model-based undergraduate introductory biology course. 165 Gilbert, S. W. (1991). Model Building and Definition of Science. Journal of research in science teaching, 28(1), 73-79. Gilbert, J. K., & Justi, R. (2016). Modelling-based Teaching in Science Education. Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-319-29039-3 Goetz, T., Pekrun, R., Hall, N., & Haag, L. (2010). Academic emotions from a social-cognitive perspective: antecedents and domain specificity of students’ affect in the context of Latin instruction. British Journal of Educational Psychology, 76(2), 289-308. https://doi.org/10.1348/000709905X42860 Gonida, E. N., Voulala, K., & Kiosseoglou, G. (2009). Students’ achievement goal orientations and their behavioral and emotional engagement: co-examining the role of perceived school goal structures and parent goals during adolescence. Learning and Individual Differences, 19(1), 53-60. http://dx.doi.org/10.1016/j.lindif.2008.04.002 Gray, E. K., Watson, D. (2001). Emotion, mood, and temperament: Similarities, differences and a synthesis. In R. Rayne, C. Cooper (Eds.), Emotions at Work. pp. 21-43, Chichester: Wiley. Green, G., Rhodes, J., Hirsch, A. H., Suarez-Orozco, C., & Camic, P. M. (2008). Supportive adult relationships and the academic engagement of Latin American immigrant youth. Journal of School Psychology, 46, 393–412. https://doi.org/10.1016/j.jsp.2007.07.001 Heddy, B. C., & Sinatra, G. M. (2013). Transforming misconceptions: Using transformative experience to promote positive affect and conceptual change in students learning about biological evolution. Science Education, 97, 723-744. https://doi.org/10.1002/sce.21072 Hektner, J. M., Schmidt, J. A., & Csikszentmihalyi, M. (2007). Experience sampling method: Measuring the quality of everyday life. Sage. https://doi.org/10.4135/9781412984201 Hess, J. L., Miller, S., Higbee, S., Fore, G. A., & Wallace, J. (2020). Empathy and ethical becoming in biomedical engineering education: A mixed methods study of an animal tissue harvesting laboratory. Australasian Journal of Engineering Education, 26(1), 127-137. https://doi.org/10.1080/22054952.2020.1796045 Hirschfield, P. J., & Gasper, J. (2011). The relationship between school engagement and delinquency in late childhood and early adolescence. Journal of Youth and Adolescence, 40, 3–22. https://doi.org/10.1007/s10964-010-9579-5 Jaeger, S. R., Lee, S. M., Kim, K-O., Chheang, S. L., Jin, D., & Ares, G. (2017). Measurement of product emotions using emoji surveys: Case studies with tasted foods and beverages. Food quality and Preferences, 62, 46-59. https://doi.org/10.1016/j.foodqual.2017.05.016 Jarrell, A., Harley, J. M., & LaJoie, S. P. (2016). The link between achievement emotions, appraisals, and task performance: Pedagogical considerations for emotions in CBLEs. 166 Journal of Computers in Education, 3, 289–307. https://doi.org/10.1007/s40692-016-0064-3 Jonassen, D. H., & Strobel, J. (2006). Modeling for meaningful learning. Engaged learning with emerging technologies, 1-27. https://doi.org/10.1007/1-4020-3669-8_1 Jonassen, D., Strobel, J., & Gottdenker, J. (2005). Model building for conceptual change. Interactive Learning Environments, 13(1-2), 15-37. https://doi.org/10.1080/10494820500173292 Justi, R. S., & Gilbert, J. K. (2002). Science teachers’ knowledge about and attitudes toward the use of models and modelling in learning science. International Journal of Science Education, 24(12), 1273-1292. https://doi.org/10.1080/09500690210163198 Karamarkovich, S. M., & Rutherford, T. (2021). Mixed feelings: Profiles of emotions among elementary mathematics students and how they function within a control-value framework. Contemporary Educational Psychology, 66, 101996. https://doi.org/10.1016/j.cedpsych.2021.101996 Kellam, N., Gerow, K., Wilson, G., Walther, J., & Cruz, J. (2018). Exploring emotional trajectories of engineering students: A narrative research approach. International Journal of Engineering Education, 34(6), 1726-1740. Kennedy, B., Fry, R., & Funk, C. (April, 2021). “6 facts about America’s STEM workforce and those training for it.” PEW Research Center. Retrieved February 10, 2022 from: https://www.pewresearch.org/fact-tank/2021/04/14/6-facts-about-americas-stem- workforce-and-those-training-for-it/ Kerslake, L., & Wegerif, R. (2017). The semiotics of emoji: the rise of visual language in the age of the internet (book review). Media and Communication, 5(4), 75-78. https://doi.org/10.1515/sem-2017-0143 Ketonen, E. E., Malberg, L. E., Salmela-Aro, K., Muukkonen, H., Tuominen, H., & Lonka, K. (2019). The role of study engagement in university students’ daily experiences: A multilevel test of moderation. Learning and Individual Differences, 69, 196-205. https://doi.org/10.1016/j.lindif.2018.11.001 King, R. B., McInerney, D. M., Ganotice Jr, F. A., Villarosa, J. B. (2015). Positive affect catalyzes academic engagement: Cross-sectional, longitudinal and experimental evidence. Learning and Individual Differences, 39, 64-72. https://doi.org/10.1016/j.lindif.2015.03.005 Linnenbrink-Garcia, L., & Pekrun, R. (2011). Students’ emotions and academic engagement: introduction to the special issue. Contemporary Educational Psychology, 36, 1-3. https://doi.org/10.1016/j.cedpsych.2010.11.004 Lazarus, R. S. (1984). On the primacy of cognition. American Psychologist, 39, 124–129. https://doi.org/10.1037/0003-066X.39.2.124 167 Lichtenfeld, S., Pekrun, R., Stupnisky, R. H., Reiss, K., & Murayama, K. (2012). Measuring students’ emotions in the early years: The achievement emotions questionnaire- elementary school (AEQ-ES). Learning and Individual Differences, 22(2), 190–201. https://doi.org/10.1016/j.lindif.2011.04.009 Long, T. M., Dauer, J. T., Kostelnik, K. M., Momsen, J. L., Wyse, S. A., Bray Speth, E., & Ebert-May, D. (2014). Fostering ecoliteracy through model-based instruction. Frontiers in Ecology and the Environment, 12(2), 138-139. https://doi.org/10.1890/1540-9295-12.2.138 Lönngren, J., Adawi, T., Berge, M., Huff, J., Murzi, H., Direito, I., ... & Sultan, U. (2020, October). Emotions in engineering education: Towards a research agenda. In 2020 IEEE frontiers in education conference (FIE) (pp. 1-5). IEEE. Louca, L. T., & Zacharia, Z. C. (2012). Modeling-based learning in science education: Cognitive, metacognitive, social, material and epistemological contributions. Educational Review, 64(4), 471-492. https://doi.org/10.1080/00131911.2011.628748 Loukidou, L., Loan-Clarke, J., & Daniels, K. (2009). Boredom in the workplace: More than monotonous tasks. International Journal of Management Reviews, 11(4), 381–405. https://doi.org/10.1111/j.1468-2370 .2009.00267 Lumby, J. (2011). Enjoyment and learning: policy and secondary school learners’ experience in England. British Educational Research Journal, 37(2), 247–264. https://doi.org/10.1080/10.411920903540680. Marder, B., Houghton, D., Erz, A., Harris, L., & Javornik, A. (2020). Smile(y) - And your students will smile with you? The effects of emoticons on impressions, evaluation, and behaviour in staff-to-student communication. Studies in Higher Education, 45(11), 2274-2286. https://doi.org/10.1080/03075079.2019.1602760 Mehrabian, A. (1995). Framework for a comprehensive description and measurement of emotional states. Genetic, Social, and General Psychology Monographs, 121(3), 339-361. Moffitt, R. L., Padgett, C., & Grieve, R. (2020). Feasibility and emotionality of online assessment feedback: Using emoticons to enhance student perceptions of marker competence and warmth. Computers and Education, 143, 103654. https://doi.org/10.1016/j.compedu.2019.103654 Mulligan, K., & Scherer, K. R. (2012). Toward a working definition of emotion. Emotion Review, 4(4), 4345–57. https://doi.org/10.1177/17540739124458 Murphy, S., Wang, C. A., & Danaia, L. (2019). Towards an understanding of STEM engagement: a review of the literature on motivation and academic emotions. Canadian Journal of Science, Mathematics and Technology Education, 19, 304-320. https://doi.org/10.1007/s42330-019-00054-w 168 Naibert, N., & Barbera, J. (2022). Development and evaluation of a survey to measure student engagement at the activity level in general chemistry. Journal of Chemical Education, 99(3), 1410-1419. https://doi.org/10.1021/acs.jchemed.1c01145 Naibert, N., Vaughan, E. B., Brevick, K, & Barbera, J. (2022). Exploring student perceptions of behavioral, cognitive and emotional engagement at the activity level in general chemistry. Journal of Chemical Education, 99(3), 1358-1367. https://doi.org/10.1021/acs.jchemed.1c01051 Nett, U. E., Goetz, T., & Hall, N. C. (2011). Coping with boredom in school: An experience sampling perspective. Contemporary Educational Psychology, 36(1), 49-59. https://doi.org/10.1016/j.cedpsych.2010.10.003 National Academies of Sciences, Engineering, and Medicine. (2018). How people learn II: learners, contexts, and cultures. Washington, DC: The National Academies Press. National Center for Science and Engineering Statistics. (2019). Women, minorities, and persons with disabilities in science and engineering. Special Report NSF, 19-304. National Research Council. (2012). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. Washington, DC: National Academies Press Nicolaou, C. T.,. Evagorou, M., & Lymbouridou, C. (2015). Elementary school students’ emotions when exploring an authentic socio-scientific issue through the use of models. Science Education International, 26(2), 240–259. Novak, P. K., Smailović, J., Sluban, B., & Mozetič, I. (2015). Sentiment of Emojis. PLoS One, 10(12): e0144296. https://doi.org/10.1371/journal.pone.0144296 Ouweneel, E., Le Blanc, P. M., Schaufeli, W. B. (2011). Flourishing students: A longitudinal study on positive emotions, personal resources, and study engagement. The Journal of Positive Psychology, 6(2), 142-153. https://doi.org/10.1080/17439760.2011.558847 Padgett, C., Moffitt, R. L., & Grieve, R. (2021). More than words: Using digital cues to enhance student perceptions of online assignment feedback. The Internet and Higher Education, 49, 1-10. https://doi.org/10.1016/j.iheduc.2020.100789 Pekrun, R. (2006). The control-value theory of achievement emotions: assumptions, corollaries, and implication for educational research and practice. Educational Psychology Review, 18,(4), 315-341. https://doi.org/10.1007/s10648-006-9029-9 Pekrun, R. (2014). Emotions and learning (Educational Practices Series, Vol. 24.) International Academy of Education (IAE) and International Bureau of Education (IBE) of the United Nations Educational, Scientific and Cultural Organization (UNESCO), Geneva, Switzerland. 169 Pekrun, R., Goetz, T., Frenzel, A. C., Barchfeld, P., & Perry, R. P. (2011). Measuring emotions in students’ learning and performance: The Achievement Emotions Questionnaire (AEQ). Contemporary Educational Psychology, 36(1), 36–48. https://doi.org/10.1016/j.cedpsych.2010.10.002 Pekrun, R., T. Goetz, W. Titz, and R. P. Perry. (2002). Academic Emotions in Students’ Self-regulated Learning and Achievement: A Program of Qualitative and Quantitative Research. Educational Psychologist, 37(2), 91–105. https://doi.org/10.1207/S15326985EP3702_4 Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 259-282). New York, NY: Springer. https://doi.org/10.1007/978-1-4614-2018-7_12 Pekrun, R., & Linnenbrink-Garcia, L. (2014). Introduction to emotions in education. In International handbook of emotions in education (pp. 1-10). Routledge Pekrun, R., Lichtenfeld, S., Marsh, H. W., Murayama, K., & Goetz, T. (2017). Achievement emotions and academic performance: longitudinal models of reciprocal effects. Child Development, 88(5), 1653-1670. https://doi.org/10.1111/cdev.12704 Pekrun, R., Marsh, H. W., Elliot, A. J., Stockinger, K., Perry, R. P., Vogl, E., Goetz, T., van Tilburg, W. A. P., Lüdtke, O., & Vispoel, W. P. (2023). A three-dimensional taxonomy of achievement emotions. Journal of Personality and Social Psychology: Personality Processes and Individual Differences, 124(1), 145-178. https://doi.org/10.1037/pspp0000448 Pekrun, R., Stephens, E. J. (2012). “Academic Emotions.” In K. R. Harris, S. Graham, T. Urdan, S. Graham, J. M. Royer, and M. Zeidner (Eds.), APA Educational Psychology Handbook, Vol. 2, pp. 3-31. Washington, DC: American Psychological Association. Rand, K. L., Shanahan, M. L., Fischer, I. C., & Fortney, S. K. (2020). Hope and optimism as predictors of academic performance and subjective well-being in college students. Learning and Individual Differences, 81(101906). https://doi.org/10.1016/j.lindif.2020.101906 Ritchie, S. M., & Tobin, K. (Eds). (2018). Eventful Learning. Learner Emotions (Vol 61). Leiden, The Netherlands: Brill. Robayo-Tamayo, M., Blanco-Donoso, L. M., Román, F. J., Carmona-Cobo, I., Moreno-Jiménez, B., Garrosa, E. (2020). Academic engagement: A diary study on the mediating role of academic support. Learning and Individual Differences, 80(101887). https://doi.org/10.1016/j.lindif.2020.101887 Robinson, K. A., Beymer, P. N., Ranellucci, J., & Schmidt, J. A. (2020). Momentary emotion profiles in high school science and their relations to control, value, 170 achievement, and science career intentions. Motivation Science, 6(4), 401–412. https://doi.org/10.1037/mot0000174 Robinson, K. A., Ranellucci, J., Lee, Y., Wormington, S. V., Roseth, C. J., & Linnenbrink- Garcia, L. (2017). Affective profiles and academic success in a college science course. Contemporary Educational Psychology, 51, 209–221. https://doi.org/10.1016/j.Cedpsych.2017.08.004 Rocca, K. A. (2010). Student participation in the college classroom: An extended multidisciplinary literature review. Communication Education, 59(2), 185–213. https://doi.org/10.1080/03634520903505936 Rodrigues, D., Prada, M., Gaspar, R., Garrido, M. V., & Lopes, D. (2017). Lisbon Emoji and Emoticon Database (LEED): Norms for emoji and emoticons in seven evaluative dimensions. Behavior Research Methods. https://doi.org/10.3758/s13428-017-0878-6 Rodríguez-Muñoz, A., Antino, M., Ruiz-Zorrilla, P., Ortega, E. (2021). Positive emotions, engagement, and objective academic performance: A weekly diary study. Learning and Individual Differences, 92(102087). https://doi.org/10.1016/j.lindif.2021.102087 Rosenberg, E. L. (1998). Levels of analysis and the organization of affect. Review of General Psychology, 2, 247–270. https://doi.org/10.1037/1089-2680.2.3.247 Sagayadevan, V., & Jeyaraj, S. (2012). The role of emotional engagement in lecturer-student interaction and the impact on academic outcomes of student achievement and learning. Journal of the Scholarship of Teaching and Learning, 12(3), 1-30. Sansone, C., & Thoman, D. B. (2005). Does what we feel affect what we learn? Some answers and new questions. Learning and Instruction, 15, 507–515. Scherer, K. R., & Moors, A. (2019). The emotion process: Event appraisal and component differentiation. Annual Review of Psychology, 70(1), 719– 745. https://doi.org/10.1146/annurev-psych-122216-011854 Schutz, P. A., Aultman, L. P., Williams-Johnson, M. R. (2009). Educational Psychology Perspectives on Teachers’ Emotions. In P. Schutz & M. Zembylas (Eds.), Advances in Teacher Emotion Research, pp. 195-212, Boston, MA: Springer. Schwarz, C. V., Reiser, B. J., Davis, E. A., Kenyon, L., Archér, A., Fortus, D., Shwartz, Y., Hug, B., & Krajcik, J. (2009). Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners. Journal of Research in Science Teaching, 46(6), 632-654. https://doi.org/10.1002/tea.20311 Scollon, C. N., Prieto, C. K., & Diener, E. (2009). Experience sampling: promises and pitfalls, strengths and weaknesses. In Diener, E. (Ed.) Assessing well being (pp. 157-180). Springer. https://doi.org/10.1007/978-90-481-2354-4_8 171 Sharkey, J., Sukkyung, Y., Schnoebelen, K. (2008). Relations among school assets, individual resilience, and student engagement for youth grouped by level of family functioning. Psychology in the Schools, 45(5), 402-418. https://doi.org/10.1002/pits.20305 Shernoff, D. J. (2010). Engagement in after-school programs as a predictor of social competence and academic performance. American Journal of Community Psychology, 45(3-4), 325-337. https://doi.org/10.1007/s10464-010-9314-0 Sinatra, G. M., Broughton, S. H., & Lombardi, D. (2014). Emotions in science education. In International handbook of emotions in education (pp. 415-436). Routledge. Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). The challenges of defining and measuring engagement in science. Educational Psychologist, 50(1), 1-13. https://doi.org/10.1080/00461520.2014.1002924 Skinner, E. A., & Pitzer, J. R. (2012). Developmental dynamics of student engagement, coping, and everyday resilience. In S. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement. (pp. 21-44). Springer. Szeto, M. D., Barber, C., Ranpariya, V. K., Anderson, J., Hatch, J., Ward, J., Aguilera, M. N., Hassan, S., Hamp, A., Coolman, T., Dellavalle, R. P. (2022). Emojis and Emoticons in Health Care and Dermatology Communication: Narrative Review. JMIR Dermatol, 5(3):e33851. Vareberg, K. R., Vogt, O., & Berndt, M. (2022). Putting your best face forward: How instructor emoji use influences students’ impressions of credibility, immediacy, and liking. Education and Information Technologies, 28(5), 6075-6092. https://doi.org/10.1007/s10639-022-11421-w Wang, M., & Eccles, J. (2013). School context, achievement motivation, and academic engagement: A longitudinal study of school engagement using a multidimensional perspective. Learning and Instruction, 28, 12-23. https://doi.org/10.1016/j.learninstruc.2013.04.002 Wang, M., & Holcombe, R. (2010). Adolescents' perceptions of school environment, engagement and academic achievement in middle school. American Educational Research Journal, 47, 633-662. https://doi.org/10.3102/00028312093612 Wigfield, A., Eccles, J. S., & Möller, J. (2020). How dimensional comparisons help to understand linkages between expectancies, values, performance, and choice. Educational Psychology Review, 32(3), 657–680. https://doi.org/10.1007/s10648-020-09524-2 Xie, K., Heddy, B. C., & Greene, B. (2019). Affordances of using mobile technology to support experience-sampling method in examining college students’ engagement. Computers & Education, 128, 183-198. https://doi.org/10.1016/j.compedu.2018.09.020 172 Xie, K., Heddy, B. C., & Vongkulluksn, V. W. (2019). Examining engagement in context using experience-sampling method with mobile technology. Contemporary Educational Psychology, 59, 101788. https://doi.org/10.1016/j.cedpsych.2019.101788 Zaff, J. F., Kawashima-Ginsberg, K., Lin, E. S., Lamb, M., Palsano, A., & Lerner, R. M. (2011). Developmental trajectories of civic engagement across adolescence: Disaggregation of an integrated construct. Journal of Adolescence, 34, 1207-1220. https://doi.org/10.1016/j.adolescence.2011.07.005 Zembylas, M., & Schutz, P. A. (2016). Methodological advances in research on emotion and education. Springer. Zirkel, S., Garcia, J. A., & Murphy, M. C. (2015). Experience-sampling research methods and their potential for education research. Educational Researcher, 44(1), 7–16. https://doi.org/10.3102/0013189X14566879 173 Repeat Model Prompt: Cystic Fibrosis APPENDIX Construction Background Cystic fibrosis is the most common lethal inherited disease in Caucasian populations. Cystic fibrosis is caused by a defect in the CFTR gene located on the q arm of Chromosome 7. In healthy individuals, the wild-type allele (R) is dominant and contains the information necessary for producing normal CFTR protein. CFTR is a protein that forms a channel in cell membranes that allows the movement of chloride ions out of cells. As chloride leaves cells, water follows and thins the mucus on cell surfaces, allowing it to flow freely. In individuals with cystic fibrosis, the CFTR proteins are defective and block the flow of chloride ions and water out of cells. The inability to regulate chloride and water results in a drier mucus that is thick and sticky and accumulates on cell surfaces in the lungs, pancreas, digestive tract, and other internal organs. Individuals with cystic fibrosis experience frequent and serious bacterial infections, are unable to absorb adequate nutrients, and have chronic respiratory problems. If untreated, children with cystic fibrosis generally die before 5 years of age. However, daily chest pounding to clear mucus, along with heavy doses of antibiotics and other therapies have extended life expectancy for cystic fibrosis patients into their 20’s and 30’s. Using what you know about how genetic information is organized and expressed, construct a system model that shows how key concepts in this CFTR case work together to determine whether cystic fibrosis or a normal phenotype becomes expressed. Your overall model should show how two different outcomes are possible - cystic fibrosis or normal. R allele, r allele, chromosome 7, DNA, gene, nucleotide sequence, cystic fibrosis, normal (no cystic fibrosis), normal CFTR protein, defective CFTR protein Novel Model Prompt: Carbon Cycling Construction Background In most aquatic environments, carbon (both its form and concentration) is controlled through the microbial biofilms that cover all wet surfaces. A biofilm is a collection of microorganisms living together. These biofilm communities are composed of algae and bacteria. Carbon is transformed through a series of redox reactions that drive energy metabolism within their cells. This changes carbon between its inorganic forms (e.g. CO2) and organic forms (e.g. glucose). The autotrophic organisms, like algae, in these communities use photosynthesis to reduce CO2 into glucose while the heterotrophic organisms, like bacteria (and also algae too) oxidize glucose back into CO2 with respiration. These two coupled metabolic pathways are largely what cycle carbon through the environment. 174 Identify the components needed to create a model whose function would describe the cycling of carbon in a simple aquatic ecosystem. Use the following components to build a system model that describes the cycling of carbon in the previously described simple aquatic ecosystem: ● Algae ● Bacteria ● CO2 ● Glucose Evaluation Figure 3.1. Novel expert-drawn model provided to students for the Evaluation task. 175 CONCLUSION Instructional reform efforts among Science Technology Engineering and Mathematics (STEM) gateway courses are aimed at increasing diversity and improving retention of students to graduation; particularly those from underrepresented groups (e.g., American Association for the Advancement of Science [AAAS], 2015; Cooper et al., 2015; Dagley, et al., 2015; Hunter, 2019; Seymour et al., 2019; Sithole, et al., 2017; Xu, 2016). My dissertation bridges multiple, interconnecting areas of research to explore potential mechanisms that may account for differences in learning outcomes among students in an introductory biology course taught through model-based instruction (MBI). In Chapter One, I adopted a motivational systems perspective and generated motivational profiles that characterized students according to combinations of seven variables at the beginning and end of an introductory biology course taught through MBI. A Latent Profile Analysis (LPA) revealed four unique motivational profiles; three at each time point. In the beginning of the semester, student motivational profiles were characterized as: highly motivated; motivated, mastery and value driven; and unmotivated and performance driven. The highly motivated and unmotivated and performance driven profiles were present at the end of the semester, along with a new, average motivation profile. The majority of students demonstrated positive shifts in their motivational profile over the semester, regardless of their course grade. This finding suggests that students don’t always maintain the motivation that they enter the class with. Strikingly, all low-achieving students who began the semester characterized as unmotivated and performance driven finished the course as highly motivated, which fosters a speculation that MBI can promote motivation by engaging students in opportunities to learn through non-traditional methods. 176 Chapter Two centers on the development and application of a novel Cognitive Engagement in Modeling (CEM) framework for measuring students’ cognitive engagement during planning, monitoring, and evaluation phases of model-construction tasks. A qualitative content analysis approach (Morgan, 1993; Mayring, 2000) was applied to interview video and transcript data that revealed fourteen unique behavioral and linguistic identifiers distributed across metacognitive, generative learning, and retrieval categories of learning strategies. Application of the CEM framework identified differences in students’ use of learning strategies during different phases of model-construction and across different task types (e.g., when constructing a novel model vs. one that they had previously constructed). During the planning phase, students demonstrated greater use of metacognitive and generative learning strategy use with the novel model, which is consistent with previous research that suggests learners will apply greater strategy-use as task complexity increases (e.g., Hattie et al., 1996; Mokos & Kafoussi, 2013). Students demonstrated mixed use of strategies across contexts during the monitoring phase, suggesting that, regardless of context, modeling construction tasks require students to be cognitively engaged and employ a variety of strategies in order to be successful on them. Finally, students demonstrated greater strategy use during the evaluation phase for a model that they had previously constructed compared to the novel model task. This finding may be a product of students’ second exposure to the model and greater availability of cognitive resources to draw from in their evaluation process, such as prior feedback from the instructor. My CEM framework addresses a gap (Christenson et al., 2012) and advances research on student cognitive engagement, as it aids in identification of specific cognitive processes and learning strategies students employ during model-based tasks. In addition, the framework can be used to identify differences in learning-strategy use across student achievement levels and task 177 types. Overall, data in this study show that learning-strategy use varies across model- construction phases and model contexts. Students’ nature of engagement and the types of strategies they will deploy to complete a modeling task vary depending on what they are asked to do and where they are in the process of task completion. Development and application of an Emotional Engagement in Modeling (EEM) framework was my focus for Chapter Three. I developed the novel emoji-based, EEM framework to enable identification and assessment of what De Angeli et al. (2020) consider a range of students’ emotions during practice-based tasks. I then utilized an experience sampling method (ESM) and applied the framework during interviews to assess and compare students’ emotions during model construction and evaluation for previously-constructed and novel model contexts. Students selected discrete (i.e., single-emoji) or multiple emoji to reflect simple or complex, mixed-emotional states. After each selection, students verified their interpretation of the emotion associated with the emoji, which allowed for the development of an Emoji Sentiment Map (Novak et al., 2015) on a scale of positive to negative emotions. Findings in this study showed that students reported experiencing mostly positive emotions during model- construction and evaluation tasks, and in both previously-constructed and novel contexts. Students most frequently expressed mixed emotions in the novel model context for both construction and evaluation tasks. I additionally examined the relationship between student achievement and reported emotions, and found that students considered as high- and middle- achieving more often expressed mixed emotions, while students considered low-achieving expressed more positive emotions. 178 My EEM framework fills a gap in the traditionally understudied area of student emotional engagement in STEM and in the unexamined context of model-based learning. The findings from this study support previous research that students can simultaneously experience a wide-range of emotions during learning (e.g., Boekaerts & Pekrun, 2015; Jarrell, et al., 2016; Karamarkovich & Rutherford, 2021; Robinson et al., 2017; Robinson et al., 2020), and that these emotions can vary by context and task type, contributing to the importance of contextual influences on student emotions (e.g., Chen & Nieminen, 2024). However, in contrast to previous studies (e.g., Karamarkovich & Rutherford, 2021; Pekrun, 2006; Pekrun et al., 2011; Wigfield et al., 2020), student achievement level in this study did not predict emotional states, suggesting that even those students who are struggling academically can feel positively about practice-based tasks. Although the CEM and EEM frameworks were developed in an MBI and biology context, they can be easily applied to study engagement during other practice-based tasks (e.g.,scientific argumentation, explanation, data analysis) or disciplinary contexts. Application of the frameworks to a broader range of contexts will be beneficial for informing practitioners and researchers about emotional responses during learning more generally, and inform the development of ways we can explicitly train students about different types of learning strategies and when to use them. Findings from this and other research will be beneficial for designing assessments and learning tasks that promote engagement and progressive skill development that can transfer across task types, classroom contexts, and disciplines. Overall, my dissertation posed three research goals aimed at advancing our understanding of how students of all achievement levels are learning in MBI contexts. Evidence-based methods, such as MBI, have been shown to reduce achievement gaps and promote positive long- 179 term outcomes (Bierema et al., 2017; Dauer et al., 2013; Manthey & Brewe, 2013; Reinagel & Bray Speth, 2016; Verhoeff et al., 2008). In MBI contexts specifically, previous findings suggest there may be additional benefits for students most at risk of leaving STEM (Bennett et al., 2020; Dauer et al., 2013; Dauer & Long, 2015; de Lima & Long, 2023), however mechanisms underlying the observed differences are not well understood. My work found that achievement measures (i.e.., grades) failed to predict motivation and engagement, which suggests that motivation and engagement could be contributing factors in explaining how and why MBI and other practice-based instructional methods are successful. However, more research is needed to determine whether improved motivation and engagement translate into long-term outcomes, such as degree completion and retention in STEM. Although students in my dissertation studies demonstrated positive engagement in model-based tasks and improvements in motivation, it is unknown whether these had any impact on degree completion within their STEM program. In conclusion, as a believer in the success of those who navigate life on non-traditional paths, and of those who are traditionally viewed as likely to be unsuccessful, it is my hope that my research will be used as a foundation to promote continued exploration into trends and outcomes for lower-achieving students in practice-based STEM courses. Understanding the mechanisms that explain performance differences can inform the design of targeted interventions that promote persistence and diversity among STEM learners. 180 REFERENCES The American Association for The Advancement of Science (AAAS). (2015). Vision and Change in Undergraduate Biology Education: Chronicling change, inspiring the future. AAAS: Washington, DC. Bennett, S. W., Gotwals, A. W., & Long, T. M. (2020). Assessing students’ approaches to modeling in undergraduate biology. International Journal of Science Education, 42(10), 1697-1714. https://doi.org/10.1080/09500693.2020.1777343 Bierema, A. M. -K., Schwarz, C. V., Stoltzfus, J. R. (2017). Engaging undergraduate biology students in scientific modeling: analysis of group interactions, sense-making, and justification. CBE - Life Sciences Education, 16(68), 1-16. https://doi.org/10.1187/cbe.17-01-0023 Boekaerts, M. & Pekrun, R. (2015). Emotions and emotion regulation in academic settings. In Handbook of Educational Psychology. (pp. 90-104). Routledge. Chen, X. (2015). STEM attrition among high-performing college students in the United States: Scope and potential causes. Journal of Technology and Science Education, 5(1), 41–59. Chen, S., & Nieminen, J. H. (2024). Towards an ecological understanding of student emotions in feedback: a scoping review. Assessment & Evaluation in Higher Education, 1–18. https://doi.org/10.1080/02602938.2024.2318641 Christenson S. L., Reschly L. A., Wylie C. (Eds.). (2012). Handbook of research on student engagement. New York, NY: Springer Cooper, M. M., Caballero, M. D., Ebert-May, D., Fata-Hatrley, C. L. Jardeleza, S. E., Krajcik, J. S., Laverty, J. T., Matz, R. L., Posey, L. A., & Underwood, S. M. (2015). Challenge faculty to transform STEM learning: Focus on core ideas, crosscutting concepts and scientific practices. Science, 350(6258), 281-282. Dagley, M, Georgiopoulos, M., Reece, A., Young, C. 2015. Increasing retention and graduation rates through a STEM learning community. Journal of College Student Retention: Research, Theory & Practice, 18(2), 167-182. https://doi.org/10.1177/1521025115584746 De Angeli, D., Kelly, R. M., & O’Neill, E. (2020). Beyond happy-or-not: Using emoji to capture visitors’ emotional experience. Curator: The Museum Journal, 63(2), 167-191. https://doi.org/10.1111/cura.12352 De Lima, J. & Long, T. M. (2023). Students explain evolution by natural selection differently for humans versus nonhuman animals. CBS-Life Sciences Education, 22(4), ar48. https://doi.org/10.1187/cbe.21-06-0145 181 Dauer, J. T., & Long, T. M. (2015). Long-term conceptual retrieval by college biology majors following model-based instruction. Journal of Research in Science Teaching, 52(8), 1188-1206. https://doi.org/10.1002/tea.21258 Dauer, J. T., Momsen, J. L., Bray Speth, E., Mokohon-Moore, S. C., & Long, T. M. (2013). Analyzing change in students’ gege-to-evolution models in college-level introductory biology. Journal of Research in Science Teaching, 50(6), 639-659. https://doi.org/10.1002/tea.21094 Hattie, J. A., C., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of Educational Research, 66(2), 99-136. https://doi.org/10.3102/00346543066002099 Hunter, A-B. (2019). Why undergraduates leave stem majors: changes over the last two decades. In E. Seymour & A-B Hunter (Eds.) Talking about leaving revisited: persistence, relocation and loss in undergraduate stem education, pp. 87-114. https://doi.org/10.1007/978-3-030-25304-2_3 Lytle, A., & De Rosa, A. J., & Fisher, F. T. (2021, November), A Review of Psychosocial Factors Associated with Undergraduate Engagement and Retention in STEM. Paper presented at 2021 Fall ASEE Middle Atlantic Section Meeting, Virtually Hosted by the section. Retrieved from: https://peer.asee.org/a-review-of-psychosocial-factors- associated-with-undergraduate-engagement-and-retention-in-stem Jarrell, A., Harley, J. M., & LaJoie, S. P. (2016). The link between achievement emotions, appraisals, and task performance: Pedagogical considerations for emotions in CBLEs. Journal of Computers in Education, 3, 289–307. https://doi.org/10.1007/s40692-016- 0064-3 Karamarkovich, S. M., & Rutherford, T. (2021). Mixed feelings: Profiles of emotions among elementary mathematics students and how they function within a control-value framework. Contemporary Educational Psychology, 66, 101996. https://doi.org/10.1016/j.cedpsych.2021.101996 Kennedy, B., Fry, R., & Funk, C. (April, 2021). “6 facts about America’s STEM workforce and those training for it.” PEW Research Center. Retrieved February 10, 2022 from: https://www.pewresearch.org/fact-tank/2021/04/14/6-facts-about-americas-stem- workforce-and-those-training-for-it/ Manthey, S., Brewe, E. (2013). Toward university modeling instruction- biology: adaption curricular frameworks from physics to biology. CBE- Life Sciences Education, 12, 206- 214. https://doi.org/10.1187/cbe.12-08-0136 Mayring, P. (2000). ‘Qualitative content analysis’. Forum: Qualitative Social Research, 1, 1-10. 182 Mokos, E., & Kafoussi, S. (2013). Elementary students’ spontaneous metacognitive functions in different types of mathematical problems. Journal of Research in Mathematics Education, 2(2), 242-267. https://doi.org/10.4471/redimat.2013.29 Morgan, D. L. (1993). Qualitative content analysis: A guide to paths not taken. Qualitative Health Research, 3, 112-121. https://doi.org/10.1177/104973239300300107 National Research Council [NRC]. (2010). Rising above the gathering storm, revisited: Rapidly approaching category 5. Washington, DC: The National Academies Press. National Science Foundation [NSF]. (2012). Science and engineering indicators 2012. Washington, DC: National Science Board. Novak, P. K., Smailović, J., Sluban, B., & Mozetič, I. (2015). Sentiment of Emojis. PLoS One, 10(12): e0144296. https://doi.org/10.1371/journal.pone.0144296 Olson, S. and Riordan, D.G., 2012. Engage to excel: producing one million additional college graduates with degrees in science, technology, engineering, and mathematics. Report to the president. Executive Office of the President. Pedraza, L., & Chen, R. (2022). Examining motivator factors of STEM undergraduate persistence through two-factor theory. Journal of Higher Education, 93(4), 532-558. https://doi.org/10.1080/00221546.2021.1999722 Pekrun, R. (2006). The control-value theory of achievement emotions: assumptions, corollaries, and implication for educational research and practice. Educational Psychology Review, 18,(4), 315-341. https://doi.org/10.1007/s10648-006-9029-9 Pekrun, R., Goetz, T., Frenzel, A. C., Barchfeld, P., & Perry, R. P. (2011). Measuring emotions in students’ learning and performance: The Achievement Emotions Questionnaire (AEQ). Contemporary Educational Psychology, 36(1), 36–48. https://doi.org/10.1016/j.cedpsych.2010.10.002 President’s Council of Advisors on Science and Technology [PCAST]. (2012, February). Engage to excel Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics, Washington, DC: US: Government Office of Science and Technology (Report to the President), Retrieved from: https://eric.ed.gov/?id=ED541511 Reinagel, A., Bray Speth, E. (2016). Beyond the central-dogma: model-based learning of how genes determine phenotype. CBE- Life Sciences Education, 15(1), 1-13. https://doi.org/10.1187/cbe.15-04-0105 Robinson, K. A., Beymer, P. N., Ranellucci, J., & Schmidt, J. A. (2020). Momentary emotion profiles in high school science and their relations to control, value, 183 achievement, and science career intentions. Motivation Science, 6(4), 401–412. https://doi.org/10.1037/mot0000174 Robinson, K. A., Ranellucci, J., Lee, Y., Wormington, S. V., Roseth, C. J., & Linnenbrink- Garcia, L. (2017). Affective profiles and academic success in a college science course. Contemporary Educational Psychology, 51, 209–221. https://doi.org/10.1016/j.Cedpsych.2017.08.004 Seymour, E., Hunter, A. B., Weston, T. J. (2019) Why We Are Still Talking About Leaving. In: Seymour, E., Hunter, A. B. (Eds.), Talking about Leaving Revisited. Springer, Cham. https://doi-org.proxy1.cl.msu.edu/10.1007/978-3-030-25304-2_1 Sithole, A., Chiyaka, E.T., McCarthy, P., Mupinga, D. M., Bucklein, B. K., Kibrige, J. (2017). Student attraction, persistence and retention in STEM programs: successes and continuing challenges. Higher Education Studies, 7(1), 46-59. Verhoeff, R. P., Boersma, K. T., Waarlo, A., J. (2008). Systems modeling and the development of coherent understanding of cell biology. International Journal of Science Education, 30(4), 543-568. https://doi.org/10.1080/09500690701237780 Wigfield, A., Eccles, J. S., & Möller, J. (2020). How dimensional comparisons help to understand linkages between expectancies, values, performance, and choice. Educational Psychology Review, 32(3), 657–680. https://doi.org/10.1007/s10648-020-09524-2 Xu, Y. J. (2016). Attention to Retention: Exploring and addressing the needs of college students in STEM majors. Journal of Education and Training Studies, 4(2), 67-76. 184