A STEP INTO THE UNKNOWN: EXPLORING STUDENTS’ CONSTRUCTION OF MECHANISTIC ARROWS FOR BOTH FAMILIAR AND UNFAMILIAR REACTIONS IN ORGANIC CHEMISTRY By Samantha Houchlei A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Chemistry – Doctor of Philosophy 2022 ABSTRACT Our goal as scientists is to help students make sense of the world by building on their prior knowledge and engaging with what they already understand. Undergraduate students bring with them a whole host of experiences from their prior educations and their prior experiences in the world. To be successful in their coursework and in future courses, students must make connections between their knowledge and the core ideas of a discipline. These connections are necessary for students to be successful in many of their courses as they move from a novice with disconnected understanding to a more integrated expert like understanding. Undergraduate organic chemistry is one such course and is often a prerequisite to many courses that are required for pre professional schools, such as medical and veterinary school. However, previous research shows that students often rely on surface level features and memorization to be successful in organic chemistry. In particular, prior research has found that students struggle with using mechanistic arrows, a tool used to predict the movement of electrons and predict reaction mechanisms. My work seeks to characterize and understand better ways to support students learning mechanistic arrows in organic chemistry. This dissertation focuses on how students use mechanistic arrows in both familiar and unfamiliar reactions. This work was situated within a transformed organic chemistry course that emphasizes students constructing explanations and developing and using models and explicitly making connections to structure property relationships and electrostatic and bonding interactions. However, students can take many combinations of courses throughout their time as undergraduates, and I explored how these different course types and backgrounds affected how students draw mechanistic arrows for both familiar and unfamiliar reactions. The studies in this dissertation used both quantitative and qualitative techniques to examine students use and understanding of reaction mechanisms. Student participants were sampled from both the two-semester transformed organic curriculum and a course that has not undergone a transformation, referred to in this dissertation as a traditional curriculum. By sampling students multiple times throughout both semesters and both course types, this allowed me to investigate the effect various course types and combinations had on students’ responses. Findings suggest that students who took two semesters of the transformed course use their arrows to predict a plausible product more frequently that traditional students do for both a familiar and unfamiliar reaction. Furthermore, when exploring the different course combinations students take for organic chemistry, I found that students’ ability to draw arrows varies depending on their organic chemistry course background. Students who most recently took the transformed curriculum were better able to make plausible predictions with their mechanistic arrows than the traditional students. While students who had the transformed curriculum for the first semester and switched to the traditional curriculum for the second semester did not draw plausible mechanisms as frequently as students who took the transformed curriculum for both semesters. This emphasizes the importance of consistent course environments that support students’ engagement with the core ideas of a discipline. Additionally, I describe the process of developing a task designed to elicit students’ explanations and understandings of an unfamiliar intramolecular reaction. Implications of this work on instructional coherence and future directions will be discussed. I dedicate this dissertation to my best friends, Lindsay Delamarter, Sam Delamarter, Jackie Gipe, Olivia Crandell, and my partner Ngoc Doan. I am forever grateful for the support and love you all have shown me, this would not have been possible without you iv ACKNOWLEDGEMENTS There is not enough room in this chapter to acknowledge and thank all of the people who have helped me grow into the person I am today, but I am going to try. I am still in awe at the support and kindness everyone has shown me over the last 5 years. First and foremost, I would like to thank Dr. Melanie Cooper for her patience and support guiding me through this adventure that is graduate school. I have learned more in the last few years than I ever dreamed possible, and it was with her guidance and care that I was able to grow in my confidence and career. She has always encouraged me to be the best version of myself I can be and given me the tools to do so. I would also like to thank my second reader, Dr. Lynmarie Posey for her support and engaging discussions throughout my time at MSU. I knew I could ask her for support in exploring new ideas and understanding when it comes to higher education. I would also like to thank my graduate committee members, Dr. Karen Draths and Dr. Kevin Walker for their guidance from the first introduction paper I wrote, to writing this dissertation. Their support and knowledge helped shape my graduate career for the better. I would also like to thank the many members of the Cooper group I have grown to know and love over my time at MSU. First and foremost, Dr. Keenan Noyes and Dr. Olivia Crandell, it is exceedingly rare to meet people who have cared for and supported me like you both have. I know I am a better person and academic from knowing you and I cannot imagine where I would be without you two. I am incredibly thankful for the other graduate students in our group that have helped me in both friendship and collaboration: soon to be Dr. Clare Franovic, soon to be Dr. Ryan Bowen, Sewwandi Abeywardana, Kriti Seth, and Veeda Scammahorn. I would like to also thank Robby McKay and Jacob Starkie for their constant support and tenacity when it came to collaborating on projects and thinking outside the box. I have also had the pleasure of working with numerous postdocs and faculty, all of whom supported and shaped my career in more ways than I can express: Dr. Ryan Stowe, Dr. Kinsey Bain, Dr. Elizabeth Day, Dr. Ashling Flaherty, Dr. Justin Carmel, Dr. Ginny Cangelosi, and Dr. Amy Pollock. You all have touched v my life with your friendship and support, and I hope to bring the kindness and care you all showed me into my future endeavors as well. Lastly, I would like to thank Noah and Joshua. Our late-night chats and Saturday adventures got me through the last 5 years with a smile on my face and more appreciation for you both than I could ever say in words. You are the best brothers I could have ever asked for. Finally, I would like to thank my partner Ngoc, I am incredibly grateful for your love and encouragement, I am so blessed to have you in my life. vi TABLE OF CONTENTS LIST OF TABLES ............................................................................................................................................. ix LIST OF FIGURES .......................................................................................................................................... xii CHAPTER I – INTRODUCTION ........................................................................................................................ 1 Study Goals and Research Questions........................................................................................................ 2 REFERENCES .............................................................................................................................................. 5 CHAPTER II – THEORETICAL FRAMEWORKS .................................................................................................. 7 Prior Knowledge and Constructivism ........................................................................................................ 7 From Novice Toward Expert Understanding............................................................................................. 8 Conceptual Change and Prior Knowledge................................................................................................. 8 Summary ................................................................................................................................................. 14 REFERENCES ............................................................................................................................................ 15 CHAPTER III – LITERATURE REVIEW ............................................................................................................ 18 Scaffolding and Evidence Centered Design............................................................................................. 18 Making Sense of the World ..................................................................................................................... 21 Three-Dimensional Learning ................................................................................................................... 22 Prior Literature on Mechanistic Arrows and Explanations in Organic Chemistry................................... 29 Summary ................................................................................................................................................. 35 REFERENCES ............................................................................................................................................ 36 CHAPTER IV - MECHANISMS, MODELS, AND EXPLANATIONS: ANALYZING THE MECHANISTIC PATHS STUDENTS TAKE TO REACH A PRODUCT FOR FAMILIAR AND UNFAMILIAR ORGANIC REACTIONS ........... 40 Preface .................................................................................................................................................... 40 Introduction ............................................................................................................................................ 40 Research Questions: ............................................................................................................................... 49 Methods .................................................................................................................................................. 50 Results ..................................................................................................................................................... 58 Discussion................................................................................................................................................ 67 Implications for Teaching ........................................................................................................................ 70 Limitations............................................................................................................................................... 72 REFERENCES ............................................................................................................................................ 73 APPENDIX A: PERMISSIONS..................................................................................................................... 78 APPENDIX B: STUDIES PARTICIPANT DEMOGRAPHICS AND INTERRATER RELIABILITY .......................... 79 APPENDIX C: INDIVIDUAL CODING MECHANISMS AND INTERRATER RELIABILITY ................................. 80 APPENDIX D: ADDITIONAL RESULTS........................................................................................................ 87 APPENDIX E: ADDITIONAL DEMOGRAPHIC COMPARISON WITH EARLIER STUDY .................................. 90 CHAPTER V – “WHAT ABOUT THE STUDENTS WHO SWITCHED COURSE TYPE?”: AN INVESTIGATION OF INCONSISTENT COURSE .............................................................................................................................. 91 Introduction ............................................................................................................................................ 91 Building on our Prior Work ..................................................................................................................... 96 Research Question guiding this work ..................................................................................................... 97 Methods .................................................................................................................................................. 98 vii Results ................................................................................................................................................... 101 Discussion.............................................................................................................................................. 108 Implications ........................................................................................................................................... 109 Future Directions .................................................................................................................................. 110 Limitations............................................................................................................................................. 111 REFERENCES .......................................................................................................................................... 112 APPENDIX A: STUDIES PARTICIPANT DEMOGRAPHICS ......................................................................... 116 APPENDIX B: CHANGE OVER TIME ........................................................................................................ 117 APPENDIX C: COMPARISON OF OTHER FAMILIAR PROMPTS OVER TIME ............................................ 119 CHAPTER VI –WHAT IS THE RELATIONSHIP BETWEEN STUDENTS’ CAUSAL MECHANISTIC EXPLANATIONS AND STUDENTS USE OF MECHANISTIC ARROWS? ................................................................................... 121 Preface .................................................................................................................................................. 121 Introduction .......................................................................................................................................... 121 Prior Work ............................................................................................................................................. 123 Research Question ................................................................................................................................ 125 Methods ................................................................................................................................................ 126 Results ................................................................................................................................................... 129 Discussion.............................................................................................................................................. 138 Implications and Future directions ....................................................................................................... 139 Limitations............................................................................................................................................. 140 REFERENCES .......................................................................................................................................... 141 APPENDIX A: ALL PLAUSIBLE ARROWS CODING SCHEME AND ANALYSIS ............................................ 144 APPENDIX B: STUDY 2 – CAUSAL MECHANISTIC REASONING ............................................................... 147 CHAPTER VII- THE BEST-LAID PLANS OFT' GO AWRY: HOW CHANGING SCAFFOLDING MAY HAVE UNFORSEEN CONSEQUENCES IN STUDENT RESPONSES .......................................................................... 149 Preface .................................................................................................................................................. 149 Introduction .......................................................................................................................................... 149 Research Questions............................................................................................................................... 151 Methods ................................................................................................................................................ 151 Results and Discussion .......................................................................................................................... 157 Implications and Future Work .............................................................................................................. 184 Limitation .............................................................................................................................................. 185 REFERENCES .......................................................................................................................................... 186 APPENDIX A: INSTRUCTOR PATHS FOR OCHEM 1- OCHEM 2 AND INTERRATER RELIABILITY .............. 190 APPENDIX B: FULL PROMPT DESCRIPTION ........................................................................................... 192 CHAPTER VIII – CONCLUSIONS, IMPLICATIONS, and FUTURE DIRECTIONS .............................................. 193 Conclusions ........................................................................................................................................... 193 Implications ........................................................................................................................................... 195 Future Directions .................................................................................................................................. 197 REFERENCES .......................................................................................................................................... 199 viii LIST OF TABLES Table 2.1. Summary of both sides of the conceptual change fault line...................................................... 13 Table 3.1. Complete list of Disciplinary Core Ideas, Scientific Practices, and Crosscutting concepts.18 ..... 23 Table 4.1. Percent of Students Who Drew a Plausible Product For all chi-square analysis α = 0.01. (α) students in OCLUE outperformed Traditional, (β)students in Traditional outperformed OCLUE. ............. 58 Table 4.2. Time Point One: Percent of Students Who Drew All Plausible Arrows and Predicted a Plausible Product. For all chi-square analysis α = 0.01. ............................................................................................. 60 Table 4.3. Time Point Two: Percent of Students Who Drew All Plausible Arrows and Predicted a Plausible Product. For all chi-square analysis α = 0.01. ............................................................................................. 61 Table 4.4. Prompt A Change in Ranks for Students from Time Point One to Time Point Two. For all chi- square analysis α = 0.01. ............................................................................................................................. 64 Table 4.5. Mann – Whitney Comparison of Traditional and OCLUE students.43,44,50 .................................. 79 Table 4.6. Description of gender of Traditional and OCLUE students.43,44,50 .............................................. 79 Table 4.7. Description of intended major of Traditional and OCLUE students........................................... 79 Table 4.8. Interrater Reliability Prompt A. .................................................................................................. 83 Table 4.9. Interrater Reliability Prompt B. .................................................................................................. 84 Table 4.10. Interrater Reliability Prompt C. ................................................................................................ 84 Table 4.11. Interrater Reliability Prompt D. ................................................................................................ 85 Table 4.12. Interrater Reliability Prompt E. ................................................................................................ 86 Table 4.13. Time Point One: Percent of Students Who Drew All Arrows Correctly and Predicted a Plausible Product. For all chi-square analysis α = 0.01. (α) students in OCLUE out preformed Traditional. ......................................................................................................................... 87 Table 4.14. Percent of Traditional Students in the Current Study Who Drew a Plausible Product Before Drawing Mechanistic Arrows. ..................................................................................................................... 89 Table 4.15. Percent of students who drew the Major product for prompts A-E at Time Point Two. ........ 89 Table 4.16. Description of student from the Previous Studies and Current Study..................................... 90 Table 4.17. How mechanism attempts from the earlier studies compare to the current study. ............... 90 Table 5.1. Differences in students use of arrows for a Familiar Reaction between course type.a ........... 104 ix Table 5.2. Differences in students use of arrows for a unfamiliar Reaction between course type.a ....... 107 Table 5.3. Mann – Whitney Comparison of student participants’ demographics.32,38,39 .......................... 116 Table 5.4. Description of gender and intended major of student participants. Values in the table are the absolute counts for each group of students. ............................................................................................ 116 Table 5.5. Familiar Reaction Change in Ranks for Students from Time Point One to Time Point Two. ... 118 Table 5.6. Differences in students use of arrows for Prompt B: Electrophilic Addition of Water to Alkyne between course type.a .............................................................................................................................. 119 Table 5.7. Differences in students use of arrows for Prompt C: Alkyne Deprotonation Followed by SN2 between course type.a .............................................................................................................................. 119 Table 5.8. Differences in students use of arrows for Prompt D: Nucleophilic Attack at a Carbonyl between course type.a .............................................................................................................................. 120 Table 6.1. Summary of Student responses coded. Traditional is abbreviated Trad for simplicity. .......... 127 Table 6.2. Code groups for student’s use of CMR & Some Plausible Arrows. The colors correspond to the graphs in the results section. .................................................................................................................... 130 Table 6.3. Percent of Students who used CMR and Some Plausible Arrows for both OCLUE-OCLUE and Traditional-Traditional courses. ................................................................................................................ 132 Table 6.4. Comparison of OCLUE-OCLUE students to Traditional-Traditional students use of CMR and drawing some plausible arrows. ............................................................................................................... 133 Table 6.5. Percent of Students who used CMR and Some Plausible Arrows for various course sequences. .................................................................................................................. 136 Table 6.6. Code groups for student’s use of CMR and Plausible Arrows/Products. ................................. 144 Table 6.7. Comparison of OCLUE-OCLUE students to Traditional-Traditional students use of CMR and drawing all plausible arrows and a plausible across both time points. .................................................... 146 Table 6.8. Causal Mechanistic Reasoning Student Examples. .................................................................. 147 Table 7.1. Summary of data collection and response rate. ...................................................................... 155 Table 7.2. Summary of prompt revisions that will be discussed. ............................................................. 159 Table 7.3. Initial coding scheme for where students started or ended their arrows. .............................. 162 Table 7.4. Coding scheme for the products draw by students. ................................................................ 163 Table 7.5. Coding scheme for stundets drawn partial charges................................................................. 164 x Table 7.6. Coding scheme for students written explanations. ................................................................. 168 Table 7.7. Percent of students who drew all arrows correctly and correct product for prompt versions 1-4. ................................................................................................................................. 181 Table 7.8. Instructor paths list OChem 1-OChem 2 (instructor descriptions below in table 7.10)........... 190 Table 7.9. Anonymized Instructor descriptions and course descriptions. ................................................ 191 Table 7.10. Interrater Reliability for students’ mechanism drawing. ....................................................... 191 Table 7.11. Interrater Reliability for students’ drawn charges. ................................................................ 191 Table 7.12. Interrater Reliability for students’ drawn charges. ................................................................ 191 xi LIST OF FIGURES Figure 3.1. Modified Evidence Centered Design process with examples of steps 1 and 2......................... 20 Figure 4.1. The sequence of inferences and connections that students must be able to make to construct a causal mechanistic explanation about acid base reactions. .................................................................... 45 Figure 4.2. Summary of mechanisms administered to participants in the papers.22,23 Highlighted prompt will be discussed in this paper. ................................................................................................................... 49 Figure 4.3. Summary of the data collection. ............................................................................................... 54 Figure 4.4. Example of a student’s completed mechanism in beSocratic. ................................................. 55 Figure 4.5. Overarching code groupings for all reactions. The color scheme shows the colors of the code groupings in the graphs in the results section. ........................................................................................... 57 Figure 4.6. Percent of students who drew all mechanistic steps correctly, students who drew some mechanistic steps correctly, students who drew no mechanistic steps correctly, and students who got the incorrect product based on the courses they took. ............................................................................. 60 Figure 4.7. Prompt A: Percent of students who drew all mechanistic steps correctly, students who drew some mechanistic steps correctly, students who drew no mechanistic steps correctly, and students who got the incorrect product based in the courses they took. ........................................................................ 61 Figure 4.8. Prompt A: Percent of students who drew all mechanistic steps correctly, students who drew some mechanistic steps correctly, students who drew no mechanistic steps correctly, and students who got the incorrect product based in the courses they took. ........................................................................ 63 Figure 4.9. Percent of students who attempted drawing a mechanism but still drew and incorrect product and student who only drew and incorrect product with no arrows. ............................................ 65 Figure 4.10. Screenshot from beSocratic in which student responses are shown in a grid for both OCLUE (top) and Traditional (bottom).................................................................................................................... 66 Figure 4.11. Permissions to reproduce manuscript in its entirety.............................................................. 78 Figure 4.12. Prompt A: Familiar Reaction. .................................................................................................. 80 Figure 4.13. Prompt B: Electrophilic Addition of Water to Alkyne. ............................................................ 80 Figure 4.14. Prompt C: Alkyne Deprotonation Followed by SN2. ................................................................ 81 Figure 4.15. Prompt D: Nucleophilic Attack at a Carbonyl. ........................................................................ 82 Figure 4.16. Prompt E: Unfamiliar Reaction. .............................................................................................. 83 xii Figure 4.17. Time Point One Prompt B distribution of students’ arrows and product based on the courses they took. .................................................................................................................................................... 87 Figure 4.18. Time Point One Prompt C distribution of students’ arrows and product based on the courses they took. .................................................................................................................................................... 87 Figure 4.19. Time Point One Prompt D distribution of students’ arrows and product based on the courses they took. .................................................................................................................................................... 88 Figure 4.20. Time Point Two Prompt B distribution of students’ arrows and product based on the courses they took. .................................................................................................................................................... 88 Figure 4.21. Time Point Two Prompt C distribution of students’ arrows and product based on the courses they took. .................................................................................................................................................... 88 Figure 4.22. Time Point Two Prompt D distribution of students’ arrows and product based on the courses they took. .................................................................................................................................................... 89 Figure 5.1. Summary of the data collection. ............................................................................................... 99 Figure 5.2. Percent of arrow use, and product drawn for a familiar reaction Prompt A for each cohort. The Traditional-OCLUE cohort is small (n = 16) and marked with an asterisk.......................................... 103 Figure 5.3. Percent of arrow use, and product drawn for a familiar reaction Prompt B for each cohort. The Traditional-OCLUE cohort is small (n = 16) and marked with an asterisk.......................................... 105 Figure 5.4. Percent of arrow use, and product drawn for a familiar reaction Prompt C for each cohort. The Traditional-OCLUE cohort is small (n = 16) and marked with an asterisk.......................................... 105 Figure 5.5. Percent of arrow use, and product drawn for a familiar reaction Prompt D for each cohort. The Traditional-OCLUE cohort is small (n = 16) and marked with an asterisk.......................................... 106 Figure 5.6. Percent of arrow use, and product drawn for an unfamiliar reaction for each cohort. The Traditional-OCLUE cohort is small (n = 16) and marked with an asterisk................................................. 107 Figure 5.7. Familiar Reaction OCLUE-OCLUE Change Over Time. ............................................................. 117 Figure 5.8. Familiar Reaction OCLUE-Traditional Change Over Time. ...................................................... 117 Figure 5.9. Familiar Reaction Traditional-Traditional Change Over Time................................................. 118 Figure 6.1. Standardized residual values that are positive (blue) indicated where there were more observed counts than would be expected if there was no relationship, values that are negative (red) mean there are less observed instances than would be expected if there was no relationship. ............ 132 Figure 6.2. Percent of organic chemistry student’s use of CMR and some plausible arrow use across both time points. ............................................................................................................................................... 134 xiii Figure 6.3. Percent of student’s use of CMR and some plausible arrows used at the start of OChem 2 for a familiar reaction. .................................................................................................................................... 136 Figure 6.4. Percent of student’s use of CMR and some plausible arrows used at the end of OChem 2 for a familiar reaction. ....................................................................................................................................... 137 Figure 6.5. Percent of student’s use of CMR and some plausible arrows used at the end of OChem 2 for an unfamiliar reaction. .............................................................................................................................. 137 Figure 6.6. Percent of organic chemistry student’s use of CMR and drawing all plausible arrows and a plausible across both time points. ............................................................................................................ 146 Figure 6.7. Causal Mechanistic Reasoning Prompt. .................................................................................. 148 Figure 6.8. Causal Mechanistic Reasoning Full Characterization. ............................................................. 148 Figure 7.1. Initial intramolecular reaction used as the foundation for all prompt iterations. The diagram shown the chain of inferences that are necessary to fully understand why the reaction is occurring. ... 152 Figure 7.2. Example student responses for prompt version 1a and 1b. ................................................... 160 Figure 7.3. Distribution of students drawn product for prompt version 1a and 1b. ................................ 165 Figure 7.4. Example student responses for prompt version 2a and 2b. ................................................... 167 Figure 7.5. Bryant’s example response for prompt version 2a................................................................. 169 Figure 7.6. Edgar’s example response for prompt version 2b. ................................................................. 170 Figure 7.7. Distribution of students drawn product for prompt version 2a and 2b. ................................ 171 Figure 7.8. Example student responses for prompt version 3a and 3b. ................................................... 173 Figure 7.9. Distribution of students drawn product for prompt version 3a and 3b. ................................ 174 Figure 7.10. This shows the Lewis structures that were used in the full version 4a prompt. The same wording was used as shown in Figure 10. The full Version 4a is in Appendix B. ...................................... 175 Figure 7.11. Example student responses for prompt version 4b.............................................................. 176 Figure 7.12. Distribution of students drawn product for prompt version 4a and 4b. .............................. 177 Figure 7.13. Martyn’s example response for prompt version 4a. ............................................................ 178 Figure 7.14. Distribution of students CMR for prompt version 4a and 4b. .............................................. 179 Figure 7.15. Distribution of students drawn product for prompt versions 1-4. ....................................... 181 xiv Figure 7.16. Example student responses for prompt version 4a. ............................................................. 192 xv CHAPTER I – INTRODUCTION Organic chemistry is a course necessary for many pre-professional school admission tests, as well as, biology and chemistry major requirements and for this reason the course is often considered a gateway course.1 In fact, at Michigan State University (MSU), where this research was conducted, roughly 1400 students take organic chemistry 1 in the fall and roughly 1200 students take organic chemistry 2 each spring semester. Organic chemistry requires that students call upon the knowledge that they gained in general chemistry about interactions, energy and stability, and structure property relationships. Organic chemistry builds on these principles and incorporates them into new phenomena such as spectroscopy, reaction synthesis, and acid base chemistry. There have been courses designed to support students in making sense of general and organic chemistry and are structured to help students make connections between their knowledge to build a more scientifically consistent understanding of the world. Two such curricula are Chemistry, Life, the Universe, and Everything2 (CLUE) and the subsequent Organic Chemistry, Life, the Universe, and Everything3 (OCLUE) which were designed by Cooper and Klymkowsky. These curricula are based on A Framework for K-12 Science Education (herein called Framework) which is a document that describes the vision for how students can engage with science.4 The Framework presents three-dimensional learning (3DL) as a way to support students’ understanding and is comprised of (1) core ideas, what we want students to know, (2) scientific practices, what we want students to do with that knowledge, and (3) crosscutting concepts, what aspects of a phenomena we want students to focus on.4 The curricula CLUE and OCLUE are built upon 3DL and students are frequently asked throughout the course to develop and use models, construct written explanations, and make arguments from evidence. All of this is done to give students the opportunity to engage with this way of thinking about phenomena. 1 This dissertation is situated in organic chemistry, and focuses on how students use mechanistic arrows, a tool used by disciplinary experts, to help make sense of phenomena. These mechanistic arrows serve as a way for students to model the redistribution of electron density within a molecule and are used, to make plausible and correct prediction about phenomena.5 However, numerous studies have found evidence that students struggle to understand and use mechanistic arrows the same way experts do.6–10 This work will expand not only our understanding of students’ use of mechanistic arrows but also how course type and consistency can affect students use of arrows. There are many studies published on the efficacy of CLUE1,11–18, and evidence is beginning to emerge for OCLUE19,20, this dissertation will build on this evidence and propose recommendations for continued investigations. The studies presented in this manuscript aim to understand the differences between students enrolled in OCLUE and a traditional organic chemistry course. This dissertation investigates how students make predictions with mechanistic arrows for familiar and unfamiliar reactions, as well as, how students reason and make predictions about more complex molecules. This work will help broaden our understanding of what OCLUE and Traditional students know and are able to do with their knowledge. Study Goals and Research Questions Study 1: Mechanisms, Models, and Explanations: Analyzing the Mechanistic Paths Students Take to Reach a Product for Familiar and Unfamiliar Organic Reactions The first study focused on how students draw mechanistic arrows and use them to predict products for a set of reactions. Most of the reactions were familiar to students while one was to our knowledge an unfamiliar reaction. In this study we focused on two cohorts of students: those who had OCLUE for both semesters (OCLUE-OCLUE), and those who had Traditional for both semester of organic chemistry (Traditional-Traditional). The research questions that drove this study were: 2 1. In what ways are responses from students who are enrolled in a transformed organic course similar or different to an equivalent group of students from a Traditional organic course for familiar reactions? 2. In what ways are responses from students who are enrolled in a transformed organic course similar or different to an equivalent group of students from a Traditional organic course for an unfamiliar reaction? Study 2: “What About the Students Who Switched Course Type?”: An Investigation of Inconsistent Course Sequence In Study 1 we explored students who had a consistent course type throughout the organic chemistry sequence. However, many students switch between the different course environments from transformed to traditional (OCLUE-Traditional) and from traditional to transformed (Traditional-OCLUE). This study uses the same methodology to investigate how students who switch between course type use mechanistic arrows and predict a plausible product for the same reactions that were discussed in Study 1. This study was guided by the following questions: 1. What is the impact of switching between a transformed organic chemistry course and a Traditional organic chemistry course as measured by a students’ use of mechanistic arrows to predict familiar products. 2. What is the impact of switching between a transformed organic chemistry course and a Traditional organic chemistry course as measured by a students’ use of mechanistic arrows to predict an unfamiliar product. Study 3: What is the Relationship Between Students’ Causal Mechanistic Explanations and Students use of Mechanistic Arrows In Study 3 we explored the connection between causal mechanistic reasoning and students’ use of mechanistic arrows. This work follows Study 1 and 2, however, responses from another prompt about student reasoning on nucleophilic substitution were also included. We investigated the association 3 between students’ use of causal mechanistic reasoning (CMR) on the nucleophilic substitution and their ability to draw mechanistic arrows and predict a plausible product for the familiar and unfamiliar reactions from Study 1 and 2. Because the previous studies found differences in students’ responses between different course type students general chemistry backgrounds were also explored in the analysis. The research questions for this study were: 1. What is the relationship between CMR and students mechanistic arrow use on a familiar and unfamiliar reaction? 2. How does the use of both CMR and mechanistic arrows differ between OCLUE and Traditional students? 3. How does a student’s general chemistry background impact their causal mechanistic reasoning and mechanistic arrow use on a familiar and unfamiliar reaction? Study 4: The Best-Laid Plans Oft’ Go Awry: How Changing Scaffolding May Have Unforeseen Consequences in Student Responses This goal of this final study was to design a task to elicit students’ use of mechanistic arrows and reasoning about an intramolecular ring closure. This is phenomena requires students to make connections between reactions they may be more familiar with, such as an SN2, and the more complex intramolecular reaction. This prompt required many iterations to produce what we consider an appropriate task to elicit students’ understanding of intramolecular reactions. The work presented in this study will describe the students’ responses, the changes that were made to the prompt, and the major findings from each of the prompt iterations. The research questions guiding this work are: 1. What impact does additional scaffolding have on students’ use of mechanistic arrows and predicting a product for an intramolecular ring closure? 2. What are the different types of resources students use to explain the phenomenon of an intramolecular ring closure? 4 REFERENCES (1) Matz, R. L.; Fata-Hartley, C. L.; Posey, L. A.; Laverty, J. T.; Underwood, S. M.; Carmel, J. H.; Herrington, D. G.; Stowe, R. L.; Caballero, M. D.; Ebert-May, D.; Cooper, M. M. Evaluating the Extent of a Large-Scale Transformation in Gateway Science Courses. Sci. Adv. 2018, 4 (10), eaau0554. https://doi.org/10.1126/sciadv.aau0554. (2) Cooper, M.; Klymkowsky, M. Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform. J. Chem. Educ. 2013, 90 (9), 1116–1122. https://doi.org/10.1021/ed300456y. (3) Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96 (9), 1858–1872. https://doi.org/10.1021/acs.jchemed.9b00401. (4) A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas; National Academies of Sciences, Engineering, and Medicine, Ed.; The National Academies Press: Washington, D.C, 2012. (5) Bhattacharyya, G. From Source to Sink: Mechanistic Reasoning Using the Electron-Pushing Formalism. J. Chem. Educ. 2013, 90 (10), 1282–1289. https://doi.org/10.1021/ed300765k. (6) Ferguson, R.; Bodner, G. M. Making Sense of the Arrow-Pushing Formalism among Chemistry Majors Enrolled in Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9 (2), 102–113. https://doi.org/10.1039/B806225K. (7) Bhattacharyya, G.; Bodner, G. M. “It Gets Me to the Product”: How Students Propose Organic Mechanisms. J. Chem. Educ. 2005, 82 (9), 1402. https://doi.org/10.1021/ed082p1402. (8) Flynn, A. B.; Featherstone, R. B. Language of Mechanisms: Exam Analysis Reveals Students’ Strengths, Strategies, and Errors When Using the Electron-Pushing Formalism (Curved Arrows) in New Reactions. Chem. Educ. Res. Pract. 2017, 18 (1), 64–77. https://doi.org/10.1039/C6RP00126B. (9) Graulich, N.; Bhattacharyya, G. Investigating Students’ Similarity Judgments in Organic Chemistry. Chem. Educ. Res. Pract. 2017, 18 (4), 774–784. https://doi.org/10.1039/C7RP00055C. (10) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844–849. https://doi.org/10.1021/ed2003934. (11) Williams, L. C.; Underwood, S. M.; Klymkowsky, M. W.; Cooper, M. M. Are Noncovalent Interactions an Achilles Heel in Chemistry Education? A Comparison of Instructional Approaches. J. Chem. Educ. 2015, 92 (12), 1979–1987. https://doi.org/10.1021/acs.jchemed.5b00619. (12) Noyes, K.; Cooper, M. M. Investigating Student Understanding of London Dispersion Forces: A Longitudinal Study. J. Chem. Educ. 2019, 96 (9), 1821–1832. https://doi.org/10.1021/acs.jchemed.9b00455. 5 (13) Becker, N.; Noyes, K.; Cooper, M. Characterizing Students’ Mechanistic Reasoning about London Dispersion Forces. Journal of Chemical Education 2016, 93 (10), 1713–1724. https://doi.org/10.1021/acs.jchemed.6b00298. (14) Noyes, K.; McKay, R. L.; Neumann, M.; Haudek, K. C.; Cooper, M. M. Developing Computer Resources to Automate Analysis of Students’ Explanations of London Dispersion Forces. J. Chem. Educ. 2020, 97 (11), 3923–3936. https://doi.org/10.1021/acs.jchemed.0c00445. (15) Crandell, O. M.; Kouyoumdjian, H.; Underwood, S. M.; Cooper, M. M. Reasoning about Reactions in Organic Chemistry: Starting It in General Chemistry. J. Chem. Educ. 2019, 96 (2), 213–226. https://doi.org/10.1021/acs.jchemed.8b00784. (16) Cooper, M. M.; Williams, L. C.; Underwood, S. M. Student Understanding of Intermolecular Forces: A Multimodal Study. J. Chem. Educ. 2015, 92 (8), 1288–1298. https://doi.org/10.1021/acs.jchemed.5b00169. (17) Cooper, M. M.; Underwood, S. M.; Hilley, C. Z.; Klymkowsky, M. W. Development and Assessment of a Molecular Structure and Properties Learning Progression. J. Chem. Educ. 2012, 89 (11), 1351–1357. https://doi.org/10.1021/ed300083a. (18) Underwood, S. M.; Reyes-Gastelum, D.; Cooper, M. M. When Do Students Recognize Relationships between Molecular Structure and Properties? A Longitudinal Comparison of the Impact of Traditional and Transformed Curricula. Chem. Educ. Res. Pract. 2016, 17 (2), 365–380. https://doi.org/10.1039/C5RP00217F. (19) Crandell, O. M.; Lockhart, M. A.; Cooper, M. M. Arrows on the Page Are Not a Good Gauge: Evidence for the Importance of Causal Mechanistic Explanations about Nucleophilic Substitution in Organic Chemistry. J. Chem. Educ. 2020, 97 (2), 313–327. https://doi.org/10.1021/acs.jchemed.9b00815. (20) Crandell, O. M. Investigation of Students’ Causal Mechanistic Reasoning in Undergraduate Organic Chemistry. Ph.D. Dissertation, Michigan State University, MI, 2020. 6 CHAPTER II – THEORETICAL FRAMEWORKS To set the foundation for the remaining chapters in my dissertation, I want to begin by detailing the purpose of this chapter. A theoretical framework is a system of ideas, aims, goals, theories, and assumptions about knowledge.1 Theoretical frameworks inform how research should be carried out, the types of data that are collected, and how results should be reported for an experiment. Taking stock of one’s theoretical perspective also moves the research from an objective outside individual to a participant observer who brings with them their own set of assumptions and views of the world.1 Theoretical perspectives are the lens through which a researcher views their study and helps inform the researcher on the types of questions they want to ask and the methods they will use to carry out such investigations. This chapter lays out the theoretical perspectives that have guided the research in this dissertation and the assumptions I make about knowledge in the minds of learners. Prior Knowledge and Constructivism When students enter a classroom, they bring with them a whole host of complex cognitive, social, and cultural systems that they have gained through their unique experiences in the world.2,3 This prior knowledge influences how students perceive information, and to best support our students we must have an understanding and framework of how people learn. This idea that students are not blank slates and come to the classroom with their own preconceptions is largely from constructivism and Jean Piaget’s work in developmental psychology.4,5 Piaget noted that children think about the world very differently than adults do and their view of the world is constructed by their interactions with it.4,5 Through Piaget’s work, he posits that information must be brought in by the learner based on their experiences, and well as the learner adapting new knowledge to make sense of the world. He explains that this can be done through two ways 1) assimilating knowledge; where a learner takes new information and fits it into what they already know and 2) accommodating knowledge; where a learner modifies what they already know to incorporate this new knowledge into their overall knowledge 7 structure.4 The lens of constructivism largely guides our understanding of how undergraduates learn and we use this to help our students move from a more novice understanding toward an expert understanding within our discipline of chemistry. From Novice Toward Expert Understanding The National Academies report on Discipline-Based Education Research, from here forward referred to as the DBER report, laid a foundation of how to help learners move from novice toward expert understanding.6 The report lays out how we can support students, which begins by: 1) identifying what students know, 2) how do students ideas align with scientifically-held understandings (i.e., expert knowledge), 3) how to change those ideas that are not aligned, and 4) to support students learning new concepts.6 Across the disciplines, much of this research is predicated on the assumption that instructors need to know what their students already know, because prior knowledge can either interfere with or facilitate new learning.2 Indeed, students come to the classroom with preconceptions about how the world works. If their initial understanding is not engaged, they may fail to grasp the new concepts and information that are taught, or they may learn them for purposes of a test but revert to their preconceptions outside the classroom.6 Students need to be given ample opportunities to engage with their prior knowledge along with new concepts to support their learning. We know that knowledge cannot be directly transferred from instructor to the student rather, it is co-constructed over time in a collaborative effort.2–4,7 This engagement with prior knowledge is necessary to help students build towards a more expert like understanding Conceptual Change and Prior Knowledge By, engaging with students’ prior knowledge, instructors will gain a better understanding of how students’ knowledge aligns with scientifically held understanding. Having a better understanding of what students know gives instructors the opportunity to help students learn new material or address 8 incorrect ideas students might hold. This process through which students’ initial understandings or beliefs are altered to more closely align with scientifically-held understandings is called conceptual change.8 There are many different theoretical perspectives that describe how students’ knowledge structures are organized. Two dichotomous views that can be used to illustrate the different beliefs about knowledge structures are those of Thomas Kuhn and Stephan Toulmin and their views on scientific theories. Though Kuhn and Toulmin were philosophers of science, their differing beliefs on the nature of scientific theories has influenced much of the early conceptual change research. In 1970 Thomas Kuhn published his work, The Structure of Scientific Revolutions and in this book rejected the idea that science progresses incrementally and that there is a coherence to scientific theories.9 When a new scientific theory is proposed there is a mis-match between the old scientific theory and the new scientific theory, and this can cause a crisis until the new theory can be accommodated or rejected. In Kuhn’s view of coherence, new scientific theories cannot be stated in terms of an older theory and that if new theories are accommodated, they are swapped with the old theory. Toulmin, however, rejected the idea that scientific theories are mutually exclusive and rejected their assumed coherence in his 1972 published work, Human Understanding.10 Toulmin argued that scientific theories are evolutionary not revolutionary and that scientific theories can be changed and adapted over time. In Toulmin’s approach these scientific theories could be innovated on and new selections of ideas can be added to the theory over time. Toulmin’s conjecture took a more fragmented approach to how scientific theories are iterated on and changed. Both Kuhn’s coherence view, and Toulmin’s fragmentation view were about scientific theories, however, these views are analogies used by conceptual change literature to illustrate how students’ conceptions change.8 In conceptual change literature there are two dominate perspectives on how knowledge is structured; 1) the coherence perspective, where students’ ideas are coherent unitary entities that can 9 be replaced by scientifically appropriate ideas and 2) the fragmentation perspective, students’ knowledge elements can be viewed as many resources (fragments) that can be linked and activated in a given context. Because the theory of conceptual change to which one subscribes can directly impact the actions taken as a researcher, it is important to understand both perspectives (coherence and fragmentation) to better understand our own perspectives on student knowledge structures. The Coherence Perspective and Misconceptions Leveraging Kuhn’s idea of coherence this perspectives’ key assumption is that students’ knowledge is coherent and that students’ old and new knowledge is mutually exclusive. One key aspect missing from Kuhns analogy was a mechanism by which students’ knowledge would be replaced. However, in 1982 Posner et al. proposed a process by which someone’s central concepts change from one set to another by answering two questions, 1) under what conditions do concepts get replaced by others, and 2) what features dictate the selection and accommodation of new concepts. Through their work Posner et al. proposed conditions for accommodations of new concepts. First, students need to be dissatisfied with their current concept. Second, the new conception needs to be intelligible, the student must understand the new concept. Third, this new conception must be plausible and used to explain observations by the learner. Fourth, this new conception must be fruitful, the learner believes this will be helpful in future pursuits, or to open up new areas of inquiry.11 The first step in this proposed process involves identifying misconceptions, which are understandings or explanations that differ from what is known to be scientifically correct.6 This process of supporting students involves instructors engaging with students’ prior knowledge to help identify their misconceptions. Indeed, the idea that an instructor can identify students’ misconceptions about a concept and replace them with the correct idea is enticing and much of the literature from the early 80s even to today focuses on identifying misconceptions students hold. In fact, approximately 120 chemistry papers have been published on this topic between 2000 and 2010.12 These subjects range from understanding of scale,13–15 bonding and interactions,16–19 10 structure property relationships,20 to Quantized energy21. Thus, a large portion of research is dedicated to pointing out students’ misconceptions, without accounting for how to better support students’ understanding of a subject. This focus on identifying misconceptions also puts the onus on the student needing to be “fixed” rather than focusing on how instructors can support students. Overall, there are two major limitations of this view of conceptual change. First, adopting such a view of the nature of conceptual understanding provides no analysis for the underlying mechanism for how students arrive at the misconception; all one can conclude is, simply, that they have a misconception.22 Even well-designed, validated instruments for diagnosing students’ misconceptions lack the power to explicate how the student arrived at that answer. Without the assessment of student reasoning, their answer selection provides poor evidence of the students’ underlying cognitive structure. The second major limitation is that this characterization of students’ misconceptions provide no account of productive resources that students have for advancing their understanding.22 This coherence framing of conceptions lacks a mechanism for change (step 3 in the call-to-action outlined in the DBER report6), because it assumes that the students’ conception is a fully-formed, albeit incorrect, idea. Thus, the Posner et al. mechanism of inducing cognitive dissonance to produce a wholesale change in conceptual understanding is not borne out by evidence. In fact, in 1992, Posner et al. wrote an article explaining how they believe their initial 1982 publication was misinterpreted, “We have said little that can, without further elaboration, be directly applied to instruction. We have always regarded attempts to turn our four components of conceptual change into four steps of instruction as misinterpretations of our intent…. Much as we would like to specify some direct implications for instruction, we do not believe that our work has such direct implications” (p.172).23 11 The Fragmentation and Resources Perspective In contrast with the coherence perspective is the fragmentation perspective, which views students’ knowledge as a collection of pieces or fragments of concepts and skills. From this perspective, the goal in teaching is not to replace students’ incorrect ideas but rather, create opportunities for them to use the appropriate knowledge for a specific context.22,24 The fragmentation perspective leveraged the analogy of Toulmin, that old and new theories are not mutually exclusive. For example, diSessa, from physics education, viewed students’ knowledge as having many phenomenological primitives called p-prims, which are many, loosely organized ideas that are categorically unlike theories.25,26 These p-prims are reusable, pieces of knowledge that can be used in more than one instance. These p-prims are based on personal observations of the world and thus the different contexts they may be plausibly used in is limited by the macroscopic world.24 These loosely organized ideas can be enlisted to support student learning. Many of the p-prims diSessa describes relate to macroscopic phenomena such as, pushing a box with more force means it will slide farther across the floor.26 This p-prim “more means more” implies that more effort begets more result and, diSessa argues, p-prims like this can be used to understand many phenomena students encounter.26 Another example from physics being Ohm’s law, that more effort begets more result and greater resistance begets less result. In these instances, students can call upon these p-prims to help them understand a phenomena they have encountered. However, because these p-prims are often derived from students lived experiences in the macroscopic world they may inadvertently use them to understand a phenomena when it might not be appropriate. For example, in a study investigating students’ understanding of structure-property relationships some students indicated that molecules with more bonds would have a higher boiling point.27 One could argue that in this instance the students are incorrectly leveraging the p-prim of more bonds means higher boiling point to explain the phenomenon of boiling. Table 2.1 illustrates the main differences attributed to the different perspectives on conceptual change. 12 Table 2.1. Summary of both sides of the conceptual change fault line. Conceptual Change What is the structure of How does a change in knowledge happen? Perspective students’ knowledge? Coherence Conceptions are unitary entities11 Replaced by scientifically-appropriate ideas11 Knowledge elements are fine- New knowledge elements are added; Connections Fragmentation grained, interrelated entities22 between knowledge elements are reorganized22 Hammer expanded on the fragmentation perspective and explained students’ knowledge structures as a collection of resources (fragments) that can be linked and activated in a given context.22,24 Hammer posits that when a student faces an unfamiliar problem they will search through their resources, to find ones that may be useful to them.22 Students bring with them their prior knowledge (including a number of p-prims) to the classroom. This prior knowledge can be used with the resources perspective as the raw materials that must be activated, engaged with, and selected for productive use in order to “help students ‘reweave’ the strands of their knowledge and understanding.”22 These knowledge structures are highly flexible and context dependent and thus are much less coherent in novices. Students may make a mistake in applying a resource, by supposing it is useful for solving a problem in a way that it turns out not to be. But that does not mean the resource itself is invalid.22,28 One of example that often appears in organic chemistry is steric hindrance. In one study29 students were asked to compare the nucleophilicity of negatively charged (1-methyl) ethylthiolate with a negatively charged ethanolate. Some students incorrectly used steric hindrance in their explanation about which would perform better as a nucleophile given the same reaction conditions. In another study30 students were given questions from the ACS organic chemistry exam about chair conformations and students incorrectly selected answers where the substituents were on only axial positions. Students explain that “if the methyl groups are further apart, then they are lower in energy so less sterics.”30 In these instances, the students are using the resource of steric hinderance and incorrectly applying it to explain a phenomena. The concept of steric hinderance itself is not right or wrong, however, how students apply this resource may be productive or unproductive. Instead of 13 cataloguing misconceptions or “difficulties generally attributed to stable beliefs,”6 instructors may attend to the possibility of these “wrong” ideas as counter-productive resource activations. With an understanding and appreciation for the types and richness of the cognitive resources, instructors may design (or re-design) instructional materials and learning contexts in order to activate these resources in ways that are meaningful and productive to students. Summary Our goal as a field is to support students, which begins by: 1) identifying what students know, 2) how do students ideas align with scientifically-held understandings, 3) how to change those ideas that are not aligned, and 4) to support students learning new concepts, as outlined in the 2012 DBER report.31 Although previous work has been dedicated to steps 1 and 2, future work must move past characterizing misconceptions in order to address step 3 and 4. In my work rather than viewing students’ understanding as unitary conceptions, a resources perspective allows me to explore the cognitive resources that are used by students and can be addressed directly with instruction. In adopting this resources perspective, not all preconceptions are viewed as misconceptions that must be rewritten, rather students bring complex and rich knowledge with them to the classroom that instructors can integrate with new knowledge to support student understanding. 14 REFERENCES (1) Bodner, G. M.; Orgill, M. Theoretical Frameworks for Research In Chemistry and Science Education, 1st ed.; Pearson College Division. (2) National Academies of Sciences, Engineering, and Medicine. How People Learn: Brain, Mind, Experience, and School: Expanded Edition; National Academies Press: Washington, D.C., 2000; p 9853. https://doi.org/10.17226/9853. (3) National Academies of Sciences, Engineering, and Medicine. How People Learn II: Learners, Contexts, and Cultures; National Academies Press: Washington, D.C., 2018; p 24783. https://doi.org/10.17226/24783. (4) Piaget, J. Part I: Cognitive Development in Children: Piaget Development and Learning. J. Res. Sci. Teach. 1964, 2 (3), 176–186. https://doi.org/10.1002/tea.3660020306. (5) Piaget, J.; Cook, M. The Origins of Intelligence in Children; New York, International Universities Press, 1952. (6) Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering; Singer, S. R., Nielsen, N., Schweingruber, H. A., Eds.; The National Academies Press: Washington, D.C, 2012. (7) Wood, D.; Bruner, J. S.; Ross, G. THE ROLE OF TUTORING IN PROBLEM SOLVING *. Child Psychology Psychiatry 1976, 17 (2), 89–100. https://doi.org/10.1111/j.1469- 7610.1976.tb00381.x. (8) Murphy, P. K.; Alexander, P. The Role of Knowledge, Beliefs, and Interest in the Conceptual Change Process: A Synthesis and Meta-Analysis of the Research. International Handbook of Research on Conceptual Change 2008, 583–616. (9) Kuhn, T. S. The Structure of Scientific Revolutions, [2d ed., enl.; International encyclopedia of unified science. Foundations of the unity of science, v. 2, no. 2; University of Chicago Press: Chicago, 1970. (10) Toulmin, S. Human Understanding. Philosophy and Rhetoric 1975, 8 (3), 198–200. (11) Posner, G. J.; Strike, K. A.; Hewson, P. W.; Gertzog, W. A. Accommodation of a Scientific Conception: Toward a Theory of Conceptual Change. Sci. Ed. 1982, 66 (2), 211–227. https://doi.org/10.1002/sce.3730660207. (12) Barke, H.-D.; Hazari, A.; Yitbarek, S. Misconceptions in Chemistry; Springer Berlin Heidelberg: Berlin, Heidelberg, 2009. https://doi.org/10.1007/978-3-540-70989-3. (13) Irby, S. M.; Phu, A. L.; Borda, E. J.; Haskell, T. R.; Steed, N.; Meyer, Z. Use of a Card Sort Task to Assess Students’ Ability to Coordinate Three Levels of Representation in Chemistry. Chem. Educ. Res. Pract. 2016, 17 (2), 337–352. https://doi.org/10.1039/C5RP00150A. 15 (14) Gerlach, K.; Trate, J.; Blecking, A.; Geissinger, P.; Murphy, K. Valid and Reliable Assessments To Measure Scale Literacy of Students in Introductory College Chemistry Courses. J. Chem. Educ. 2014, 91 (10), 1538–1545. https://doi.org/10.1021/ed400471a. (15) Trate, J. M.; Geissinger, P.; Blecking, A.; Murphy, K. L. Integrating Scale-Themed Instruction across the General Chemistry Curriculum. J. Chem. Educ. 2019, 96 (11), 2361–2370. https://doi.org/10.1021/acs.jchemed.9b00594. (16) Boo, H. K. Students’ Understandings of Chemical Bonds and the Energetics of Chemical Reactions. J. Res. Sci. Teach. 1998, 35 (5), 569–581. https://doi.org/10.1002/(SICI)1098- 2736(199805)35:5%3C569::AID-TEA6%3E3.0.CO;2-N. (17) Novick, S. No Energy Storage in Chemical Bonds. null 1976, 10 (3), 116–118. https://doi.org/10.1080/00219266.1976.9654072. (18) Galley, W. C. Exothermic Bond Breaking: A Persistent Misconception. J. Chem. Educ. 2004, 81 (4), 523. https://doi.org/10.1021/ed081p523. (19) Kohn, K. P.; Underwood, S. M.; Cooper, M. M. Energy Connections and Misconnections across Chemistry and Biology. LSE 2018, 17 (1), ar3. https://doi.org/10.1187/cbe.17-08-0169. (20) Williams, L. C.; Underwood, S. M.; Klymkowsky, M. W.; Cooper, M. M. Are Noncovalent Interactions an Achilles Heel in Chemistry Education? A Comparison of Instructional Approaches. J. Chem. Educ. 2015, 92 (12), 1979–1987. https://doi.org/10.1021/acs.jchemed.5b00619. (21) Roche Allred, Z. D.; Bretz, S. L. Development of the Quantization and Probability Representations Inventory as a Measure of Students’ Understandings of Particulate and Symbolic Representations of Electron Structure. J. Chem. Educ. 2019, 96 (8), 1558–1570. https://doi.org/10.1021/acs.jchemed.9b00098. (22) Hammer, D. Student Resources for Learning Introductory Physics. Am. J. Phys. 2000, 68 (S1), S52–S59. https://doi.org/10.1119/1.19520. (23) Strike, K. A.; Posner, G. J. A Revisionist Theory of Conceptual Change. In Philosophy of science, cognitive psychology, and educational theory and practice; SUNY series in science education; State University of New York Press, 1992; pp 147–176. (24) Hammer, D. Misconceptions or P-Prims: How May Alternative Perspectives of Cognitive Structure Influence Instructional Perceptions and Intentions? J. Learn. Sci. 1996, 5 (2), 97–127. (25) Smith III, J. P.; diSessa, A. A.; Roschelle, J. Misconceptions Reconceived: A Constructivist Analysis of Knowledge in Transition. Journal of the Learning Sciences 1994, 3 (2), 115–163. https://doi.org/10.1207/s15327809jls0302_1. (26) diSessa, A. A.; Sherin, B. L. What Changes in Conceptual Change? International Journal of Science Education 1998, 20 (10), 1155–1191. https://doi.org/10.1080/0950069980201002. (27) Cooper, M. M.; Corley, L. M.; Underwood, S. M. An Investigation of College Chemistry Students’ Understanding of Structure–Property Relationships. Journal of Research in Science Teaching 2013, 50 (6), 699–721. https://doi.org/10.1002/tea.21093. 16 (28) Sayre, E. C.; Wittmann, M. C. Plasticity of Intermediate Mechanics Students’ Coordinate System Choice. Phys. Rev. ST Phys. Educ. Res. 2008, 4 (2), 020105. https://doi.org/10.1103/PhysRevSTPER.4.020105. (29) Eckhard, J.; Rodemer, M.; Bernholt, S.; Graulich, N. What Do University Students Truly Learn When Watching Tutorial Videos in Organic Chemistry? An Exploratory Study Focusing on Mechanistic Reasoning. J. Chem. Educ. 2022, 99 (6), 2231–2244. https://doi.org/10.1021/acs.jchemed.2c00076. (30) Rushton, G. T.; Hardy, R. C.; Gwaltney, K. P.; Lewis, S. E. Alternative Conceptions of Organic Chemistry Topics among Fourth Year Chemistry Students. Chem. Educ. Res. Pract. 2008, 9 (2), 122–130. https://doi.org/10.1039/B806228P. (31) National Research Council. Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering; Singer, S. R, Nielson, N. R., Schweingruber, H. A., Eds.; National Academies Press: Washington, DC, 2012. 17 CHAPTER III – LITERATURE REVIEW Scaffolding and Evidence Centered Design Students bring with them knowledge about how the world works when they enter the classroom. Our goal as scientists is to help them begin to make sense of the world by engaging with their knowledge and build upon what they already understand. To do so instructors must carefully design questions to elicit what students know and what they can do with that knowledge. Fostering this type of understanding is particularly difficult in chemistry, partially because it requires students to understand ideas using different representations across different scales. To be able to best support students in reweaving their knowledge requires careful activation of resources through many contexts to give them ample opportunities to understand the material. Considering that students often use their own experiences to generate scientific explanations it stands to reason that they have difficulties with concepts for which they lack a frame of reference.1 This frame of reference can come externally from the instructor in the form of scaffolding. Scaffolding is the process by which someone more knowledgeable is able to help someone less knowledgeable accomplish a task they would otherwise be unable to accomplish.2,3 Scaffolding involves finding the appropriate amount of scaffolding – the line between what students actually know with help, and what students know on their own.2 These could come in the form of structuring the task, offering cues and hints, and even modelling the activity for the learner. Scaffolding is also informed by Vygotsky and his notion of students’ zone of actual development (ZAD) and their zone of proximal development (ZPD).4 A students’ ZDA describes what students have already mastered, while a student’s ZPD describes what they can achieve with support from an instructor, peer, or task.4 These ZPDs in conjunction with the structure of a scaffold can create a powerful tool. By engaging students with information that is initially beyond their understanding and supporting them with a scaffold to succeed at the task, Vygotsky argued this is when significant learning 18 is likely to occur.4 These scaffolds are environments created by the instructor to help students facilitate using their resources productively to learn within a discipline. In adopting a resources perspective of students’ knowledge structures along with the utility of scaffolding and ZPDs, we contend that not all knowledge students have are misconceptions. These conceptions are not yet stable and may just be the student trying to marshal the necessary resources to address the question. Depending on the context and the wording of the question students may elicit different resource to address the question at hand. One example of this in organic chemistry highlighted how changing the wording of the question altered how students constructed explanations of a simple nucleophilic substitution reaction. In this study,5 the original prompt saw only 52% of students discussing electrostatics in their response but after changing the wording of the prompt they saw 66% of students discussing electrostatics in their response. In another study,6 involving the phenomenon of protein−ligand binding, the authors evaluated how different wording and depictions of the prompt might affect the resources students use to explain protein−ligand binding. The authors found that slight changes in the task; such as changing the ligand from a glucose molecule to magnesium ion, drastically changed how the students responded to the task. These examples illustrate how revising a question to have different scaffolding can activate different resources within students. To be able to make appropriate changes to scaffolds evidence centered design (ECD) can be used to improve the scaffold in a systematic way.7,8 We have used a modified version of the original ECD approach illustrated in Figure 3.1 that involves 1) writing statements of what we want students to know and be able to do with their knowledge (in other words what resources would be necessary to engage with the question). 2) identifying what would we accept as evidence that a student has successfully demonstrated their knowledge. 3) designing a task to elicit this knowledge (or resources) from students. After the question has been given to students and their responses collect comes an iterative process. 4) Student responses will be characterized bearing in mind the resources that were deemed necessary for 19 the task and 5) we then revise the prompt to better support students if they are unable to call forth the resources necessary to be successful. Through this iterative process, ideally an optimized prompt will be created that is carefully balanced to engage the resources that students have, within their ZPD, that is not so simple that students do not have to think deeply or exercise their knowledge to answer the question. Figure 3.1. Modified Evidence Centered Design process with examples of steps 1 and 2. Adopting the resources perspective allows for instructors to create environments that give students ample opportunities to use their productive resources to engage with different phenomena. Doing so the focus is on students’ productive resources that they have and how they can be used to support and add to their knowledge rather than focusing on the misconceptions students may have. In doing so shifts some responsibility back to the instructor to ensure that the task at hand has the potential to activate the right resources for students, rather than just assuming students do not understand the material if they get the incorrect answer. By engaging students’ prior knowledge, one can help students activate the appropriate resources to relate new teaching to existing learning. To complement the commitment to building deep, contextualized knowledge, students should be given multiple opportunities to engage with the material and try cognitive resources, with feedback to reinforce new learning over long enough timescales. The feedback and guidance offered may well take the form of evidence-informed scaffolding using an ECD approach to support students in gradual 20 learning. The iterative nature of prompt development shifts the focus to the instructor to create supportive environments if students are not successful when answering a question initially. In doing so this can give novices more opportunities to use their productive resources across many contexts to support their understanding of a discipline as recommended in the DBER report.9 It is this theoretical framework that I use to inform my understanding of how knowledge is constructed and how to best support students learning within the discipline of chemistry. Making Sense of the World The DBER report laid a foundation of how to help learners move from novice toward expert understanding9 and to support student learning, instructors must engage with students’ prior knowledge to help them reweave new knowledge.10 Identifying these underlying structures allows for researchers and instructors alike to address common themes that may be present in students’ understandings.1,9 These themes are often “built from students’ experiences with the world and these experiences are used to generate understanding and explanations about how the world works around them” (p. 67).9 To support students learning new ideas and integrating them with their knowledge, instructors can engage students with “doing science” or, in our context, “doing chemistry”. This is because the goal for students is to “construct logically coherent explanations of phenomena that incorporate their current understanding of science, and are consistent with the available evidence.” (p. 52).11 Engaging learners with “making sense” of their world rather than “learning about” a phenomenon can help students refine their ideas by communicating how and why the world works.12 This idea of sense-making is a way for students to understand the world by generating, using and extending scientific knowledge within communities.12,13 To be able to do this type of sense-making, students use the same practices that scientists do when they are investigating phenomena, such as developing and using models or constructing explanations, which will be discussed in more depth later in this chapter. It stands to reason that only “learning about” a phenomenon becomes a collection of facts with no 21 inherent meaning unless students can do something with that knowledge.12,14 The current landscape of higher education often takes a “mile wide and inch deep” approach where students are provided with vast amounts of information with little support to integrate or use their knowledge.14,15 This information is often delivered via traditional lecture where students are rarely asked to engage with “doing science”.16 However, changes are beginning to emerge to address how to better engage students with making sense of the world around them rather than being passive recipients of information in the classroom and will be discussed throughout this dissertation. Three-Dimensional Learning In 2012, The National Research Council published a report that synthesized the literature on science learning and began to make explicit standards necessary for students to “do science”.11 To make this report, they called upon disciplinary experts in biology, physics, chemistry, environmental science, engineering, and learning and behavioral scientists. This group reviewed the literature, and the committee created A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas (here forward called The Framework) to guide the next generation of K-12 science education standards.11 These standards are comprised of three components for science learning: disciplinary core ideas (DCI), scientific and engineering practices (SEP), and crosscutting concepts (CCC). Disciplinary core ideas are what we want to students to know, the scientific and engineering practices are what we want students to do with that knowledge, and the crosscutting concepts are lenses or ways of focusing students on specific parts of phenomena.11 Together these make what is called three-dimensional learning (3DL) and inform ways instructors can support students in developing a more scientific understanding of the world.11,17 The goal is to help students make connections to core ideas of a discipline by using the scientific practices to justify their answers and focusing their approach to the question with the crosscutting concepts. A document called The Three-Dimensional Learning Assessment Protocol (3DLAP) was created by an interdisciplinary team and Michigan State University 22 (MSU) to help design and characterize assessment tasks in biology, chemistry, and physics.18 The various aspects of DCIs, SEPs, and CCCs are listed in Table 1. The 3DLAP crystalized a coherent approach to the design and implementation of assessment and curriculum reform in higher education.18 Table 3.1. Complete list of Disciplinary Core Ideas, Scientific Practices, and Crosscutting concepts.18 Disciplinary Core Ideas Scientific Practices Crosscutting Concepts Electrostatic and Bonding Asking questions Patterns Interactions Atomic/Molecular Cause and Effect: Mechanism Developing and using models Structure and Properties and Prediction Energy Planning investigations Scale, Proportion and Quantity Change and Stability in Analyzing and interpreting data Systems and system models Chemical Systems Using mathematics and Energy and Matter: Flows, computational thinking Cycles, and Conservation Constructing Explanations Structure and Function Engaging in Argument from Stability and Change Evidence Obtaining, evaluating, and communicating information The core ideas are fundamental principles that give students predictive power about phenomena. However, the core ideas proposed in The Framework did not align directly with the content or organization by discipline of many university science courses, and thus, the core ideas were adapted to the college level as a part of the creation of the 3DLAP.18 Key aspects of DCIs are that they must have the power to explain a wide range of phenomena and the potential for generating new ideas. By focusing learning on the DCIs, students are given more support to make connections among the concepts and build a more robust framework of knowledge.18,19 By anchoring students’ knowledge in DCIs instructors can scaffold assessments and activities to help activate students’ resources about a DCI in many different contexts. The DCIs identified for college chemistry courses are: 1) Electrostatic and Bonding Interactions, 2) Atomic/Molecular Structure and Properties, 3) Energy, and 4) Change and Stability in Chemical Systems.18 These four core ideas of chemistry can be used to understand a vast array of phenomena in the discipline, and by making many connections to these core ideas, instructors can support students in building a more coherent or expert-like understanding of a discipline. 23 These DCIs are distinct from topics, because they underpin many different phenomena in a discipline. A topic, such as resonance, might be taught in an organic chemistry class, and may resurface multiple times in a course; however, the topic of resonance lacks explanatory power. Instead, the DCI of electrostatics and bonding can be used as reasoning for the effect resonance might have on a molecule. The seemingly simple organic chemistry question, “why is carboxylic acid more acidic than methanol?” requires a long chain of inferences that connect back to the DCI of electrostatics and bonding. An expert may say carboxylic acid is more acidic than methanol, because its conjugate base is resonance stabilized; however, experts use an underlying framework of disciplinary knowledge to come to the succinct conclusion of resonance stabilization. This question requires novices to understand that the hydrogen bonded to oxygen is more polarized in carboxylic acid than methanol. Thus, the carboxylic acids’ hydrogen is more partially positive, and that the carboxylic acids’ hydrogen interacts with a base via electrostatic interactions. The conjugate base of the carboxylic acid has its electrons delocalized (resonance) between 3 molecules, while the conjugate base of methanol does not. This delocalization of electrons results in the stabilization of the conjugate base of the carboxylic acid, because the electrostatic interactions of the anion are not as strong. Needless to say, this is a long chain of connections used to explain the phenomena that requires more than the sole use of a topic, but rather a deep understanding of the DCI, electrostatics and bonding interactions. While the DCIs focus on what we want students to know, the CCCs can be thought of as various lenses students can use to deepen their own understanding about a topic, by examining a phenomenon from multiple perspectives.11,20 The CCCs, however, have seemed to be more difficult for instructors to understand and implement in their classrooms, because the applications of the CCCs have been less obvious.20,21 Osborne et al. argue that the crosscutting concepts have no scholarly basis for what the sciences (chemistry, biology, and physics) have in common, and that unless the importance of a crosscutting theme is identified in all disciplines, its salience is likely to be missed.22 That being said, the 24 applications of CCCs can be found in the literature even if they are not explicitly named as such, and though the CCCs do not have as robust a backing as the DCIs and SEPs, that does not mean they are not fruitful for instructors to use in their courses. For example, the curriculum developed by Talanquer et al., Chemical Thinking, emphasizes the application of models of structures and relationships to explain, predict, and control chemical behavior.23 These align with the CCCs of structure and function, as well as, stability and change, and can focus students on various parts of chemical interactions. To illustrate how the CCCs can be used to explore phenomena, we can use the acid base reaction between a carboxylic acid and methylamine. We could ask students to focus on structure and function by asking them to predict what parts of these molecules interact and explain why they interact. We could also ask students what direction the equilibrium favors for the reaction, which focuses on stability and change within the system. The phenomenon remains the same in this illustration, but the different CCCs give the instructor different ways to explore what students know about this reaction. Using the CCCs in this way is an example of how scaffolding can change the types of resources students bring to bear when engaging with a question. While the CCCs have been explored less than the DCIs and the SEPs, they still give instructors the necessary tools to provide students with explicit scaffolding to understand phenomena.20,21 The DCIs are what we want students to know, and the CCCs give us a way to focus students on different aspects of a phenomena, while the scientific practices are what we want students to be able to do with that knowledge.11 These practices encompass the sort of things that “chemists do” such as constructing explanations, developing and using models, or analyzing and interpreting data. By implementing these practices, students use their knowledge of core ideas to explain phenomena and make sense of the world around them.12 One affordance of the practices is that they give explicit ways to engage students with using their core ideas. For example, drawing mechanistic arrows in organic chemistry has no inherent meaning unless connected to a core idea such as electrostatics and bonding, 25 or structure-property relationships. There is ample evidence to suggest that organic students can draw mechanistic arrows without understanding how and why the reaction occurs.24–27 However, the SEP of developing and using models can better elicit student reasoning by asking students to draw and predict a mechanism. According to the 3DLAP, a question that has the potential to elicit student use of a model, “asks students to construct a representation and use it to explain or predict a phenomenon and asks students to provide the reasoning that links the representation to their explanation.”18 Often students in organic chemistry are asked to draw the mechanism for a reaction, and they may be asked to predict a product, but rarely are they asked to explain their reasoning and link that reasoning to their mechanism in traditional curricula.28 By using the practices, the instructor has the opportunity to see if their students are understanding the underlying components of a phenomena, or simply restating a skill or fact. The practices enable students to investigate and make sense of phenomena in the world by building and applying explanatory models, and by designing solutions for problems. CLUE and OCLUE Science curricula in the U.S. has historically presented too many ideas too superficially, leaving students with disconnected ideas that cannot be used to solve problems or explain phenomena they encounter in their everyday world.17 The affordance of 3DL allows instructors to see how their students are thinking about chemistry and, in turn, better respond to student ideas. The transformed undergraduate general chemistry course (Chemistry, Life, the Universe and Everything, CLUE)29 and organic chemistry course (Organic chemistry, Life, the Universe and Everything, OCLUE)30 were created using the principles of 3DL. The goal of both curricula is: “…to develop a molecular-level understanding of core chemical principles, so that students can explain and understand at the molecular level the relationship between the molecular-level structure of a substance, how and why a chemical reaction may occur, and how energy changes resulting from interacting materials can be understood.”29 26 The CLUE and OCLUE curricula engage students with the DCIs and encourage them to connect the DCIs by extending and relating these ideas across multiple contexts throughout the course. The textbook for both courses is provided to students free of charge, and it directly aligns with the lecture materials. The courses include weekly formative assessments that are administered via the online system beSocratic31, which allows students to freeform draw, as well as, respond to open response questions. Because these courses are centered on 3DL, the types of assessments given to students encourage them to think about the underlying components of phenomena. The in-class lectures incorporate student responses (iClickers) that involve making predictions or constructing explanations, as well as, interacting with peers to construct their reasoning. The homework assigned to students typically asks students to build upon their knowledge formed in lecture by asking them to connect their ideas to both familiar and unfamiliar phenomena to strengthen their understanding of the DCIs. In the weekly recitations, students work in groups to connect the ideas they have learned in lecture by completing worksheets that prompt students to draw representations, make predictions, explain phenomena, and construct arguments from evidence. These formative assessments make up roughly 40% of both CLUE and OCLUE final grades and are graded based on a “good faith effort” – that is, students are given an opportunity to make mistakes and receive constructive feedback instead of fearing a grade penalty. The summative assessments in the class comprise both multiple choice and short response questions that evaluate students learning throughout the course. The course lectures and both the formative and summative assessments consistently maintain the aspects of 3DL. The integration of 3DL into CLUE and OCLUE, has been shown to have a positive impact on the students enrolled in these courses. When investigating the implementation of CLUE compared to the pre-transformation course, one study found the DFW rates (D-grade, F-grade, and withdrawal) decreased by roughly 16%, which reflects approximately 740 more students earning a grade of 2.0 or above.19 Furthermore, these students performed above the national average on standardized general 27 chemistry ACS exams.19 Thus, more students pass the course with a greater understanding of the core ideas in chemistry, which is extremely encouraging given that general chemistry is often referred to as a gateway course and serves as a prerequisite for many different courses and areas of study. One example of the evidence supporting this transformation is from Williams et al., who asked students enrolled in both a traditional general chemistry course and the CLUE course, across multiple years and multiple institutions, to represent intermolecular forces (IMFs) in the context of a small molecule.32 The authors found that the students enrolled in the traditional course represented IMFs within molecules, while the CLUE students at two different universities represented IMFs between molecules. In fact, the author asked the students the same question again at the end of organic chemistry and found the CLUE students persisted in correctly representing IMFs between molecules.32 In another longitudinal study, Noyes et al. asked students enrolled in the CLUE curriculum and traditional general chemistry to explain and draw London Dispersion Forces (LDFs) between two molecules. The authors found that a large portion of CLUE students constructed an electrostatic response (75%) while few traditional students used electrostatics in their explanations (25%).33 By asking students to explain, the authors gained insight into what the students understood about the origins of LDFs and how CLUE can support student understanding. Research surrounding the effectiveness of CLUE and OCLUE in promoting student understanding of organic chemistry focus primarily on different types of mechanistic drawings and explanations about acid-base reactions and nucleophilic substitutions. One study by Cooper et al., investigated how organic students with various general chemistry backgrounds responded to the reaction of HCl + H2O and the reaction of NH3 + BF3.34 Students can take various forms of general chemistry and their backgrounds ranged from transferred credits for general chemistry 1 and 2, not taking general chemistry 2, a selective general-chemistry sequence which included honors students, chemistry majors, and CLUE students. The author found that CLUE students were more likely to provide responses discussing electrostatics than 28 students with any other general chemistry background for both the HCl + H2O and NH3 + BF3 reactions at the end of organic chemistry 2.34 Research on the effectiveness of OCLUE is still emerging; however, the results of such research hold largely the same findings as the CLUE studies. One such study conducted by Crandell et al., asked students from both OCLUE and a traditional organic course to draw a mechanism and explain how and why a nucleophilic substitution occurred.5 The authors found that all groups of students responded with roughly the same amount of electrostatic responses when they first learn about nucleophilic substitution in Organic 1 (55%). However, by the end of Organic 2, 62% of OCLUE students discussed electrostatics in their response, and only 38% of traditional students discussed electrostatics.5 This range of literature on CLUE and the emerging literature on OCLUE suggest that these transformations can support students’ understanding of the core ideas of chemistry, such as electrostatic interactions and structure property relationships. However, much of this literature has focused on simple tasks5,34–36, which is important to build a foundation of understanding about students’ knowledge, but we still lack an understanding about how students respond to more complex structures and reactions. To explore how students respond to more complex tasks, we must understand the prior knowledge and literature about student understanding of organic chemistry and the implications this has on teaching. Prior Literature on Mechanistic Arrows and Explanations in Organic Chemistry Mechanistic Arrows Students in organic chemistry use the Electron Pushing Formalism (EPF), which depicts the flow of electrons between molecules. This model serves as a tool for students to visualize the movement of the electrons that is happening at the subatomic level. Bhattacharyya sought to formalize a definition of EPF by conducting a nationwide survey of organic chemistry faculty regarding their understanding of the definition of EPF.37 The survey revealed that organic chemistry instructors used the term EPF to mean 29 the stepwise reorganization/redistribution of electrons during a chemical process resulting in working hypotheses that can be used to rationalize, explain, and predict the outcomes of chemical processes.37 This definition helped codify experts’ understanding of EPF; however, many students struggle when applying the EPF. One early study surrounding the EPF was by Ferguson and Bodner, who interviewed undergraduate students about organic mechanisms and how students made sense of their arrows. They found that many of these students did understand the role of electron donors and acceptors, but relied on memorization rather than application of that concept to solve questions.38 Integral to students’ understanding of the EPF is their understanding of Lewis structures. However, in a study conducted by Cooper et al., students seemingly would learn the “rules of the game” but not connect these rules with the concepts to create and understand Lewis structures. In fact, most students did not indicate that chemical information can be obtained through Lewis structures.39 Similar trends were found in graduate students by Bhattacharyya and Bodner, who interviewed 14 graduate students and asked them to verbally explain what their arrows meant while they were drawing organic mechanisms.24 They found that many students’ problem-solving process was: “It gets me to the product.” These graduate students found it very easy to force a solution, because the curved arrows had no physical meaning to them. One student explained, “I’m so intent on getting from reactant to product and if I see that something is not working I’ll try and force it to work”- Jen.24 These students were graduate level students and by no means novices to organic chemistry, yet still struggled with assigning meaning to the arrows they drew. This literature does not imply that students don’t “care” or desire to know how to be successful in these courses. This is illustrated in a case study about a student, Parker, who was a top student in general chemistry and intended to major in chemistry, but lost his footing when it came time for organic.40 Parker began organic chemistry enthusiastic to learn about the subject and continue using his knowledge from general chemistry. However, when interviewed again later in the semester, Parker 30 expressed a disconnect between what he thought the goal of the course was and what the apparent goal of the instructor was. Parker said his primary goal was to understand why chemicals behaved the way they did, while Parker perceived this was not the goal of his instructor –whether this was the instructor’s intention or not.40 Parker also contrasted his experience with his friends’, who said they were successful, because they “just memorized it” and didn’t know how to explain it to him to help him with his questions. The case study focused on Parker’s difficulties with the course, but exemplified what many students experienced in organic chemistry: that students can draw reactions but be completely unaware of “why” a reaction is occurring even if they recognize they should know why.40 There are many parts of the EPF that have been noted to cause difficulty for students.25–27,41,42 A study conducted by Grove et al. will be discussed at length in Chapter IV; however, they found that in a traditional organic curriculum, between 30% and 60% of the students did not draw mechanistic arrows for multiple reactions.25 Furthermore, they found that by watching video replays of student responses, an additional 15%-20% of students drew their arrows only after they had drawn a product. These students in particular seemed to be drawing their arrows not because they found them inherently useful in predicting a product, but rather because they were asked to do so.25 To build on this finding, a study conducted by Flynn and Featherstone asked students to either 1) draw the EPF for given starting materials and products or 2) draw the product for a given reactant and EPF.26 These different types of questions were administered to organic 1 and 2 students across different formative and summative assessments. The authors found that students were much more successful at the task that had students draw their arrows on the reactants and product (72% success rate) versus the tasks that had students predict the product from given EPF (55%). They also found that students were more successful on questions where the atom such as oxygen, nitrogen, carbon, and hydrogen were drawn explicitly in both fully or partially expanded structures.26 This indicated that students were focusing on surface level 31 features when trying to answer these questions, rather than thinking about the underlying mechanism and reactivity of the molecules. Students focusing on the surface level features was a repeated finding across many contexts in organic chemistry education research. Graulich and Bhattacharyya asked students to classify a list of reactions in any way they would like and had the students explain their categorization as they worked.41 They found that when classifying sets of reactions, students focused almost exclusively on surface level features such as functional group or solvent. Students had difficulty judging the extent to which substituents or reagents would affect the reaction outcome. Next, students were given pre-categorized groups of reactions and asked to explain why an expert had made such categories. When explaining experts’ classification of reactions, students again focused almost exclusively on surface level features such as the functional group in the product or that a reaction formed a ring.41 Similar findings were also highlighted by Flynn et al. in a card sorting task design to identify the organization patterns students used when classifying the reaction. This study also found that students sorted cards based on surface level features rather than the underlying mehcnaism.42 Recent work has expanded to include evaluation of students’ reaction coordinate diagrams and how they relate to reaction mechanisms. Caspari et al. had organic students compare two reactions and asked them to predict and explain which they thought had the lower activation energy.43 Students often used heuristics in their explanations without exploring the electrostatics behind their claim. For example, students would correctly claim tertiary carbocations are more stable than secondary carbocations, but would fail to explain why this is the case.43 Similar findings were observed by Popova and Bretz in a series of investigations about organic chemistry 2 students’ understandings of substitution and elimination reactions. They found that students did not discuss collisions in their explanations and primarily focused on the reactants, the intermediates, and the product of the reaction.44–46 Furthermore, they found students lacked a conceptual understanding of activation energy, as none of the students 32 spoke of activation energy in the context of collisions with sufficient energy for bonds to break and form.44–46 Encouragingly, DeGlopper et al. found that in a course where 3DL is emphasized, students were better able to connect drawn structures and mechanisms to a reaction energy diagram they created compared to a traditional course.47 Also, students were able to explain how the mechanism and structure impacted the relative energies of the transition states and intermediates.47 Student Reasoning in Organic Chemistry As shown, there are several studies that describe the different difficulties that students have with the EPF and structural representations, which has led researchers to investigate students’ written or spoken explanations to better understand their reasoning behind their answers. Reasoning is a critical component of organic chemistry, because the goal is to have students make plausible and correct predictions about reactions, and explanations are one way to understand if students know how and why a reaction occurs. One type of reasoning that has influenced many chemistry education researchers is causal mechanistic reasoning (CMR). Russ et al. built on previous reasoning work to define mechanistic reasoning as how the particular components of a system give rise to the behavior of a target phenomenon.48 The components include 1) identifying the entities and 2) identifying the activities of those entities within the context of a phenomenon.48 These types of explanations typically involve unpacking and explaining the behaviors of entities that exist a scalar level below the target phenomenon, as described by Krist et al.49 For example, in organic chemistry, the overall reaction is the phenomenon, and thus, the entities a scalar level below are the electrons, and the activities are the distributions and interactions of the electrons. One study conducted by Weinrich and Talanquer, interviewed students at different grade levels about feasibility of a reaction and how and why it would occur.50 In the interviews, they asked students to compare the easiness of a chemical synthesis, design the synthesis of a compound, and evaluate the feasibility of a proposed reaction. They characterized these responses as descriptive, relational, linear- 33 causal, and multi-component. Multi-component reasoning categorized students who created a causal account and explained how different variables affected the entities involved. They found that graduate students did leverage multi-component reasoning more often than early undergraduate students; however, the predominant mode of reasoning was a description of linear cause and effect relationships. Thus, many students lacked the ability to explain how and why the reactions were feasible, even at the graduate level. Bodé et al. used this same categorization for organic chemistry 2 students’ explanations about E1/SN1, E2, or SN2 mechanisms.51 They found that none of the students in this study used multi- component reasoning and that most students who provided correct claims about the reactions did not discuss the link between the relative energy levels of the intermediates or activation energy.51 Building on the findings of Weinrich and Bodé, Caspari et al. interviewed organic chemistry 2 students and their instructor about reaction mechanisms. In these interviews, Caspari et al. noted whether students used forward reasoning (ie. reasoning from the entities at the beginning of a mechanistic step to activities in the mechanistic step) or backward reasoning (ie. using information about activities in subsequent mechanistic steps to make a decision about activities of prior steps).27 They found that most students and their instructor explained the phenomenon with backwards reasoning, and, in fact, the instructor did not expect students to explain the interdependence with energetic properties.27 Thus, students were not provided with a causal basis to make appropriate decisions a scalar level below the phenomenon. It takes careful consideration to build a task with the necessary scaffolds for students to engage with this type of CMR. An example of developing this type of prompt structure was published by Cooper et al. in the context of an acid-base reaction.35 They gave students a simple acid base reaction and asked them to “explain your reasoning for what you think is happening at the molecular level for this reaction.” They found that with this original version of the prompt, roughly 40% of students only provided a description of the reaction, and 21% of students provided a causal mechanistic account of 34 the reaction.35 They modified the prompt to break the question into two parts, asking the students first to describe what is happening, and second to explain why it is happening at the molecular level. After this change, they found that 60% of students provided causal mechanistic accounts of the reaction and were able to explain how and why the reaction occurred. Furthermore, the authors found that the students who used a Lewis model rather than a Brønsted model to explain the reaction were 3 times more likely to draw the correct mechanistic arrows for the reaction. This provides evidence that engaging students with causal mechanistic reasoning can support students drawing their mechanistic arrows. This further reinforced the notion that if we want students to reason about the how and why of a reaction, then it must be an expected and supported form of reasoning throughout the course, as shown in this example from the CLUE curriculum. Summary By engaging students with three-dimensional learning, we can help students make sense of the world. Rather than expecting students to be passive recipients of knowledge, we can design assessments and course curricula that center on three-dimensional learning. This engages students with the things “scientists do”, such as construct an argument from evidence, develop and use a model, or construct an explanation. In the context of organic chemistry, there have been numerous studies identifying the different missteps students take when drawing Lewis structures, drawing with mechanistic arrows, and constructing energy diagrams. However, there is encouraging evidence emerging about courses centered on three-dimensional learning as a way to support students’ understanding and their explanations for different phenomena they encounter. 35 REFERENCES (1) National Academies of Sciences, Engineering, and Medicine. How People Learn: Brain, Mind, Experience, and School: Expanded Edition; National Academies Press: Washington, D.C., 2000; p 9853. https://doi.org/10.17226/9853. (2) Wood, D.; Bruner, J. S.; Ross, G. THE ROLE OF TUTORING IN PROBLEM SOLVING *. Child Psychology Psychiatry 1976, 17 (2), 89–100. https://doi.org/10.1111/j.1469- 7610.1976.tb00381.x. (3) van de Pol, J.; Volman, M.; Beishuizen, J. Scaffolding in Teacher–Student Interaction: A Decade of Research. Educ. Psychol. Rev. 2010, 22 (3), 271–296. https://doi.org/10.1007/s10648-010- 9127-6. (4) Vygotskiĭ, L. S.; Cole, M. Mind in Society: The Development of Higher Psychological Processes; Harvard University Press: Cambridge, 1978. (5) Crandell, O. M.; Lockhart, M. A.; Cooper, M. M. Arrows on the Page Are Not a Good Gauge: Evidence for the Importance of Causal Mechanistic Explanations about Nucleophilic Substitution in Organic Chemistry. J. Chem. Educ. 2020, 97 (2), 313–327. https://doi.org/10.1021/acs.jchemed.9b00815. (6) Noyes, K.; Carlson, C. G.; Stoltzfus, J. R.; Schwarz, C. V.; Long, T. M.; Cooper, M. M. A Deep Look into Designing a Task and Coding Scheme through the Lens of Causal Mechanistic Reasoning. J. Chem. Educ. 2022, 99 (2), 874–885. https://doi.org/10.1021/acs.jchemed.1c00959. (7) Mislevy, R. J.; Almond, R. G.; Lukas, J. F. A Brief Introduction to Evidence-Centered Design; Research Publications Office Educational Testing Service, 2003. (8) Riconscente, M. M.; Mislevy, R. J.; Corrigan, S. Handbook of Test Development, 2nd ed.; Educational Psychology Handbook Ser.; Taylor & Francis Group, 2015. (9) Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering; Singer, S. R., Nielsen, N., Schweingruber, H. A., Eds.; The National Academies Press: Washington, D.C, 2012. (10) Hammer, D. Student Resources for Learning Introductory Physics. Am. J. Phys. 2000, 68 (S1), S52–S59. https://doi.org/10.1119/1.19520. (11) A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas; National Academies of Sciences, Engineering, and Medicine, Ed.; The National Academies Press: Washington, D.C, 2012. (12) Helping Students Make Sense of the World Using Next Generation Science and Engineering Practices; National Science Teachers Association, 2017. https://doi.org/10.2505/9781938946042. (13) Odden, T. O. B.; Russ, R. S. Defining Sensemaking: Bringing Clarity to a Fragmented Theoretical Construct. Sci. Ed. 2019, 103 (1), 187–205. https://doi.org/10.1002/sce.21452. 36 (14) Cooper, M. M.; Posey, L. A.; Underwood, S. M. Core Ideas and Topics: Building Up or Drilling Down? J. Chem. Educ. 2017, 94 (5), 541–548. https://doi.org/10.1021/acs.jchemed.6b00900. (15) Cooper, M. M.; Stowe, R. L. Chemistry Education Research—From Personal Empiricism to Evidence, Theory, and Informed Practice. Chem. Rev. 2018, 118 (12), 6053–6087. https://doi.org/10.1021/acs.chemrev.8b00020. (16) Stains, M.; Harshman, J.; Barker, M. K.; Chasteen, S. V.; Cole, R.; DeChenne-Peters, S. E.; Eagan, M. K.; Esson, J. M.; Knight, J. K.; Laski, F. A.; Levis-Fitzgerald, M.; Lee, C. J.; Lo, S. M.; McDonnell, L. M.; McKay, T. A.; Michelotti, N.; Musgrove, A.; Palmer, M. S.; Plank, K. M.; Rodela, T. M.; Sanders, E. R.; Schimpf, N. G.; Schulte, P. M.; Smith, M. K.; Stetzer, M.; Van Valkenburgh, B.; Vinson, E.; Weir, L. K.; Wendel, P. J.; Wheeler, L. B.; Young, A. M. Anatomy of STEM Teaching in North American Universities. Science 2018, 359 (6383), 1468–1470. https://doi.org/10.1126/science.aap8892. (17) Krajcik, J.; Merritt, J. Understanding A Framework for K−12 Science Education. The Science Teacher 2012, 5. (18) Laverty, J. T.; Underwood, S. M.; Matz, R. L.; Posey, L. A.; Carmel, J. H.; Caballero, M. D.; Fata- Hartley, C. L.; Ebert-May, D.; Jardeleza, S. E.; Cooper, M. M. Characterizing College Science Assessments: The Three-Dimensional Learning Assessment Protocol. PLOS ONE 2016, 11 (9), e0162333. https://doi.org/10.1371/journal.pone.0162333. (19) Matz, R. L.; Fata-Hartley, C. L.; Posey, L. A.; Laverty, J. T.; Underwood, S. M.; Carmel, J. H.; Herrington, D. G.; Stowe, R. L.; Caballero, M. D.; Ebert-May, D.; Cooper, M. M. Evaluating the Extent of a Large-Scale Transformation in Gateway Science Courses. Sci. Adv. 2018, 4 (10), eaau0554. https://doi.org/10.1126/sciadv.aau0554. (20) Fick, S. J.; Nordine, J.; McElhaney, K. W. Proceedings of the Summit for Examining the Potential for Crosscutting Concepts to Support Three-Dimensional; The Rector and Visitors of the University of Virginia: Charlottesville, VA: University of Virginia, 2018; p 87. (21) Cooper, M. M. The Crosscutting Concepts: Critical Component or “Third Wheel” of Three- Dimensional Learning? J. Chem. Educ. 2020, 97 (4), 903–909. https://doi.org/10.1021/acs.jchemed.9b01134. (22) Osborne, J.; Rafanelli, S.; Kind, P. Toward a More Coherent Model for Science Education than the Crosscutting Concepts of the next Generation Science Standards: The Affordances of Styles of Reasoning. Journal of Research in Science Teaching 0 (0). https://doi.org/10.1002/tea.21460. (23) Talanquer, V.; Pollard, J. Reforming a Large Foundational Course: Successes and Challenges. J. Chem. Educ. 2017, 94 (12), 1844–1851. https://doi.org/10.1021/acs.jchemed.7b00397. (24) Bhattacharyya, G.; Bodner, G. M. “It Gets Me to the Product”: How Students Propose Organic Mechanisms. J. Chem. Educ. 2005, 82 (9), 1402. https://doi.org/10.1021/ed082p1402. (25) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844–849. https://doi.org/10.1021/ed2003934. 37 (26) Flynn, A. B.; Featherstone, R. B. Language of Mechanisms: Exam Analysis Reveals Students’ Strengths, Strategies, and Errors When Using the Electron-Pushing Formalism (Curved Arrows) in New Reactions. Chem. Educ. Res. Pract. 2017, 18 (1), 64–77. https://doi.org/10.1039/C6RP00126B. (27) Caspari, I.; Weinrich, M. L.; Sevian, H.; Graulich, N. This Mechanistic Step Is “Productive ”: Organic Chemistry Students’ Backward-Oriented Reasoning. Chem. Educ. Res. Pract. 2018, 19 (1), 42–59. https://doi.org/10.1039/C7RP00124J. (28) Stowe, R. L.; Cooper, M. M. Practicing What We Preach: Assessing “Critical Thinking” in Organic Chemistry. J. Chem. Educ. 2017, 94 (12), 1852–1859. https://doi.org/10.1021/acs.jchemed.7b00335. (29) Cooper, M.; Klymkowsky, M. Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform. Journal of Chemical Education 2013, 90 (9), 1116–1122. https://doi.org/10.1021/ed300456y. (30) Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96, 1858–1872. (31) Bryfczynski, S. P. BeSocratic: An Intelligent Tutoring System for the Recognition, Evaluation, and Analysis of Free-Form Student Input. Ph.D. Dissertation, Clemson University, SC, 2010. (32) Williams, L. C.; Underwood, S. M.; Klymkowsky, M. W.; Cooper, M. M. Are Noncovalent Interactions an Achilles Heel in Chemistry Education? A Comparison of Instructional Approaches. J. Chem. Educ. 2015, 92 (12), 1979–1987. https://doi.org/10.1021/acs.jchemed.5b00619. (33) Noyes, K.; McKay, R. L.; Neumann, M.; Haudek, K. C.; Cooper, M. M. Developing Computer Resources to Automate Analysis of Students’ Explanations of London Dispersion Forces. J. Chem. Educ. 2020, 97 (11), 3923–3936. https://doi.org/10.1021/acs.jchemed.0c00445. (34) Crandell, O. M.; Kouyoumdjian, H.; Underwood, S. M.; Cooper, M. M. Reasoning about Reactions in Organic Chemistry: Starting It in General Chemistry. J. Chem. Educ. 2019, 96 (2), 213–226. https://doi.org/10.1021/acs.jchemed.8b00784. (35) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid–Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703–1712. https://doi.org/10.1021/acs.jchemed.6b00417. (36) Noyes, K.; Cooper, M. M. Investigating Student Understanding of London Dispersion Forces: A Longitudinal Study. J. Chem. Educ. 2019, 96 (9), 1821–1832. https://doi.org/10.1021/acs.jchemed.9b00455. (37) Bhattacharyya, G. From Source to Sink: Mechanistic Reasoning Using the Electron-Pushing Formalism. J. Chem. Educ. 2013, 90 (10), 1282–1289. https://doi.org/10.1021/ed300765k. (38) Ferguson, R.; Bodner, G. M. Making Sense of the Arrow-Pushing Formalism among Chemistry Majors Enrolled in Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9 (2), 102–113. https://doi.org/10.1039/B806225K. 38 (39) Cooper, M. M.; Grove, N.; Underwood, S. M.; Klymkowsky, M. W. Lost in Lewis Structures: An Investigation of Student Difficulties in Developing Representational Competence. J. Chem. Educ. 2010, 87 (8), 869–874. https://doi.org/10.1021/ed900004y. (40) Anderson, T. L.; Bodner, G. M. What Can We Do about ‘Parker’? A Case Study of a Good Student Who Didn’t ‘Get’ Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9 (2), 93–101. https://doi.org/10.1039/B806223B. (41) Graulich, N.; Bhattacharyya, G. Investigating Students’ Similarity Judgments in Organic Chemistry. Chem. Educ. Res. Pract. 2017, 18 (4), 774–784. https://doi.org/10.1039/C7RP00055C. (42) Galloway, K. R.; Leung, M. W.; Flynn, A. B. Patterns of Reactions: A Card Sort Task to Investigate Students’ Organization of Organic Chemistry Reactions. Chem. Educ. Res. Pract. 2019, 20 (1), 30– 52. https://doi.org/10.1039/C8RP00120K. (43) Caspari, I.; Kranz, D.; Graulich, N. Resolving the Complexity of Organic Chemistry Students’ Reasoning through the Lens of a Mechanistic Framework. Chem. Educ. Res. Pract. 2018, 19 (4), 1117–1141. https://doi.org/10.1039/C8RP00131F. (44) Popova, M.; Bretz, S. L. “It’s Only the Major Product That We Care About in Organic Chemistry”: An Analysis of Students’ Annotations of Reaction Coordinate Diagrams. J. Chem. Educ. 2018, 95 (7), 1086–1093. https://doi.org/10.1021/acs.jchemed.8b00153. (45) Popova, M.; Bretz, S. L. Organic Chemistry Students’ Interpretations of the Surface Features of Reaction Coordinate Diagrams. Chem. Educ. Res. Pract. 2018, 19 (3), 919–931. https://doi.org/10.1039/C8RP00063H. (46) Popova, M.; Bretz, S. L. Organic Chemistry Students’ Challenges with Coherence Formation between Reactions and Reaction Coordinate Diagrams. Chem. Educ. Res. Pract. 2018, 19 (3), 732–745. https://doi.org/10.1039/C8RP00064F. (47) DeGlopper, K. S.; Schwarz, C. E.; Ellias, N. J.; Stowe, R. L. Impact of Assessment Emphasis on Organic Chemistry Students’ Explanations for an Alkene Addition Reaction. J. Chem. Educ. 2022, 99 (3), 1368–1382. https://doi.org/10.1021/acs.jchemed.1c01080. (48) Russ, R. S.; Scherr, R. E.; Hammer, D.; Mikeska, J. Recognizing Mechanistic Reasoning in Student Scientific Inquiry: A Framework for Discourse Analysis Developed from Philosophy of Science. Science Education 2008, 92 (3), 499–525. https://doi.org/10.1002/sce.20264. (49) Krist, C.; Schwarz, C. V.; Reiser, B. J. Identifying Essential Epistemic Heuristics for Guiding Mechanistic Reasoning in Science Learning. J. Learn. Sci. 2019, 28 (2), 160–205. https://doi.org/10.1080/10508406.2018.1510404. (50) Weinrich, M. L.; Talanquer, V. Mapping Students’ Modes of Reasoning When Thinking about Chemical Reactions Used to Make a Desired Product. Chem. Educ. Res. Pract. 2016, 17 (2), 394– 406. https://doi.org/10.1039/C5RP00208G. (51) Bodé, N. E.; Deng, J. M.; Flynn, A. B. Getting Past the Rules and to the WHY: Causal Mechanistic Arguments When Judging the Plausibility of Organic Reaction Mechanisms. J. Chem. Educ. 2019, 96 (6), 1068–1082. https://doi.org/10.1021/acs.jchemed.8b00719. 39 CHAPTER IV - MECHANISMS, MODELS, AND EXPLANATIONS: ANALYZING THE MECHANISTIC PATHS STUDENTS TAKE TO REACH A PRODUCT FOR FAMILIAR AND UNFAMILIAR ORGANIC REACTIONS Preface This research has been previously published in the Journal of Chemical Education and is reprinted with permission from. Houchlei, S. K.; Bloch, R. R.; Cooper, M. M. Mechanisms, Models, and Explanations: Analyzing the Mechanistic Paths Students Take to Reach a Product for Familiar and Unfamiliar Organic Reactions. J. Chem. Educ. 2021, 98 (9), 2751–2764. Copyright 2021 American Chemical Society. A copy of permissions obtained is included in the Appendices. Supporting Information for this manuscript is included in the Appendices. Introduction Organic chemistry is often viewed as a difficult course: a hurdle that is deemed necessary for many pre-professional schools, as well as, many STEM major requirements.1 The rationales for this requirement are often given as 1) organic chemistry can provide an understanding of reactions that are biologically important, and 2) it provides students with reasoning and critical thinking skills that will prove helpful in their future careers.2 Indeed, the ability to use structural information to predict the properties of substances, how they interact and what possible products could form when they react should be important for any science that relies on understanding molecular level phenomena. To support such understanding, organic chemists have developed the curved arrow notation that is intended to depict the flow of electrons from source to sinks, resulting in the formation and breaking of bonds to produce new products.3 The use of such curved arrows can be both explanatory (how does a reaction happen), and predictive (what might happen when two substances are mixed and react). When Bhattacharyya surveyed organic chemistry instructors to ascertain what they thought were the uses of curved arrow, the respondents indicated that “the principle uses of mechanistic reasoning … 40 are to explain and predict the outcomes of chemical processes”.3 However, numerous studies have shown that students often struggle with using this notation appropriately for both explanatory or predictive purposes.3–8 For example Bhattacharyya & Bodner interviewed both undergraduate and graduate students and found that when given a product they tended to propose arrow pushing mechanisms that might or might not be chemically plausible, but they “get me to the product”.9 Flynn & Featherstone investigated a range of different arrow pushing tasks and found that students were more successful when they were asked to draw arrows onto a reaction scheme in which the reactant and intermediate were given, than tasks where reactants and arrows were provided and the student had to interpret what the arrows meant and draw the product.6 That is students could draw appropriate arrows when shown the starting material and product for a mechanistic step, but were less successful in predicting the outcome of mechanistic arrows. While in this study, students were not asked to predict potential products by constructing appropriate mechanisms, even for the simpler tasks that were studied the authors showed that students have a great deal of difficulty interpreting the meaning of curved arrows. Such research has shown that many students struggle to use them as they were intended; as part of a model of the reacting system to predict and explain the mechanism and outcome of a reaction. A more recent review by Graulich elaborated on the nature of students’ understanding and the various factors that influence undergraduate and graduate students’ success in organic chemistry, including the use of mechanistic arrows. This review also emphasized the need for researchers and instructors to address the why and how of organic chemistry. Indeed, the author proposed that research should address the idea that emphasizing the connection between a structure and its underlying meaning by verbalizing the properties supports a deeper understanding.10 41 Theoretical Framing: Mechanistic Arrows and the Construction of Models and Causal Mechanistic Explanations In this section we discuss the relationship between the use of the electron-pushing formalism (mechanistic arrows), the construction and use of a model, and causal mechanistic explanations. While drawing a mechanism using curved arrows is a construct that is usually limited to organic chemistry it can be considered, in a broader sense, as an application of the scientific practice of constructing and using models. This is one of the eight scientific practices defined by the National Academies consensus report “A Framework for K-12 Science Education” (the Framework), in which the authors state “Science often involves the construction and use of a wide variety of models and simulations to help develop explanations about natural phenomena. Models make it possible to go beyond observables and imagine a world not yet seen.”11 While there are other approaches to the study of models and modelling that have been used specifically in chemistry, 12,13 here we use the approach to this scientific practice described in the Framework. As Schwarz, Passmore and Reiser note14 “Models are defined by how they are used… scientific models are sense-making tools that help us predict and explain the world.” Since, arrow pushing mechanisms do allow us to predict and explain the course of reactions we propose that they belong to this larger class of models. That is, they are representations of a system, using a defined set of components that allow us to provide mechanistic accounts of and predict outcomes of phenomena. The components of a mechanistic arrow model of a reaction are the structures of the reactants, the concomitant implicit information about electron distribution and properties that emerge from this distribution, and the explicit mechanistic arrows which must be used in conjunction with the implicit information gleaned from the structure. Using these components, the model constructor who constructs an arrow pushing mechanism should be able to both predict what will happen, and also explain why it will happen. That is, drawing a mechanism makes it possible to characterize both how and why a reacting system produces particular products from given reactants. (We are aware that other 42 factors will also impact why reactions occur – such as energy and entropy changes, but those factors are not the focus of this study). Unfortunately, the successful use of mechanistic arrows is predicated on the mechanism user understanding the encoded information in both the chemical structures and the arrow symbols, and as discussed earlier there is plenty of evidence to support the idea that many students have difficulty not only with the use of mechanistic arrows3–8, but also with the relationship between the structure of a molecule, the electron distribution within that molecule, and how these factors impact the ways that molecules interact.15–18 Ideally we would want students to uses arrows to construct a mechanism that is concordant with the implicit information encoded in the symbols that are being used. Furthermore, Russ and co-workers in their discussion of mechanistic reasoning propose that “…mechanisms account for observations by showing that underlying objects cause local changes in the system by acting on one another”.19 Krist and co-workers also emphasize the role of the underlying entities at a scalar level below the phenomenon.20 For our purposes, the reaction is the phenomenon, and the scalar level below includes the electrons (in bonds and lone pairs) and their distribution, which is caused by electrostatic forces among the nuclei, electrons, and atoms themselves. The use of curved arrows as part of a model to predict and explain requires that students are able to identify potential sources and sinks for electrons – that is the implicit information encoded in the structural representation, resulting from processes that take place at a scalar level below the reaction itself19,20. For most polar reactions, the arrow starts at a source of high electron density and ends at the electron “sink” where the electrons will be localized on an atom or where a new bond is formed. To use this approach effectively, students must use their cognitive resources to decide where the electron rich and deficient sites are, and sometimes decide between multiple locations of high and low electron density. Such expertise requires students to connect and extended series of inferences as shown in Figure 1. To construct an arrow-pushing mechanism students begin with the reacting structures, from 43 there they must understand and predict how electron density differences in molecules arise and then use that information to determine how molecules might interact, and then translate this understanding into the electron pushing formalism. It is our contention that the appropriate use of mechanistic arrows to predict and explain is the organic chemistry equivalent of constructing a causal mechanistic explanation. Both require that students use cognitive resources21 such as how and why electron distributions vary in a particular molecule, and how coulombic interactions (attractions and repulsions) govern the ways that molecules interact, to fashion the mechanism or explanation. Certainly, using mechanistic arrows is a far more parsimonious and efficient approach to predicting outcomes of organic reactions, but just as with chemical structures themselves, the implicit information embedded within the reactants and arrows that is required to use arrows appropriately may inhibit the use of mechanistic arrows in ways that were originally intended. Indeed, much of the extant literature on how students use curved arrows indicates that students are not using their understanding of how and why electron density differs in a structure, to guide how they draw mechanisms, and instead may “decorating” their structures with arrows22, either before or after they have written down the product of a reaction, or simply using them to “get to the product” or “connect the dots” without an underlying reasoning process that involves predicting how and why electrons move during a reaction.22,23 That is, some students may not be using the mechanism as a model at all and may be memorizing the pattern of electron flow (at best), or simply drawing random arrows (at worst). 44 Figure 4.1. The sequence of inferences and connections that students must be able to make to construct a causal mechanistic explanation about acid base reactions. Prior Work on Supporting Students use of Mechanistic Arrows The question then arises: How can we support students as they learn to draw mechanisms so that this act is based on their understanding of how and why electrons move during a reaction? There are a number of possible productive approaches to helping students learn the use of mechanistic arrows.7,24–26 For example, Flynn has developed a course in which students first learn the skills associated with arrow use, before they learn how and why reactions occur.24 The same authors have developed an online learning module “Mastering the Arrows” which showed significant learning gains from pre- to post-test, particularly for tasks in which students were asked to draw products of a reaction.27,28 In turn, Graulich has proposed that the use of contrasting cases may support students mechanistic reasoning, in particular through the use of scaffolded activities29,30. Additionally, Watts et. al have found that engaging students in scaffolded writing-to-learn prompt helps promote students mechanistic reasoning across more complex acid-base reactions.31 In a separate study these authors also 45 found that students who engage in a writing-to-learn prompt frequently made explicit and implicit mention of electron movement for a hydrolysis reaction.32 In our work and curricular development, we have taken yet another approach to the development of mechanistic expertise: we have chosen to address formal electron pushing mechanism use in the context of developing causal mechanistic explanations about reactions. The Framework emphasizes this connection between constructing models and explanations: “The goal for students is to construct logically coherent explanations of phenomena that incorporate their current understanding of science, or a model that represents it, and are consistent with the available evidence.”11 Indeed there is strong evidence (multiple replicated studies across diverse student populations in a range of disciplines) that constructing mechanistic explanations of phenomena, supports deeper understanding.33,34 As we noted earlier, we believe that drawing a mechanism and constructing a causal mechanistic explanation should call upon the same resources, and therefore we might expect students who learn to construct causal mechanistic explanations should also be able to draw mechanistic arrows. To be clear, students may draw an arrow pushing mechanism without engaging in causal mechanistic reasoning, and indeed many students appear to do exactly this. However, because we believe that the practices of constructing models and causal mechanistic explanations are inextricably intertwined, we have developed curricula where these practices are emphasized and connected. It is our hypothesis that by supporting student construction of written causal mechanistic explanations, we may also enhance student use of mechanistic arrows as models that can also predict and explain reaction outcomes. We do have some support for this hypothesis reported in earlier studies on student causal mechanistic reasoning about acid base chemistry35 and nucleophilic substitutions35,36. We found that general chemistry students who were able to construct causal mechanistic explanations for how and why simple acid base reactions occurred were also more likely to draw appropriate mechanistic arrows for the same reaction.35,36 In a study on nucleophilic substitutions we found that students explanations 46 typically correlated with the mechanistic arrows that they drew. That is if students believed a reaction was an SN2 reaction their arrows showed a simultaneous one step reaction, whereas if they described or explained an SN1 mechanism, their arrows corresponded with this mechanism. In this same study of nucleophilic substitution we found that student written explanations included more discussion of electron movement after drawing mechanistic arrows. Students were first asked to explain why the reactants interact, then to draw the mechanistic arrows for the reaction, and then asked to explain why they drew their arrows as indicated. Before drawing the mechanism 34% of responses explicitly discussed electron movement while this percentage jumped to 51% after drawing mechanistic arrows. This further supports the idea that the resources required for constructing causal mechanistic explanations and arrow pushing models may be connected under appropriate circumstances. At this point we also acknowledge that other researchers work on characterizing mechanistic reasoning has taken a somewhat different approach, for example Talanquer and co-workers have written extensively on student reasoning including mechanistic reasoning37–39, and following on from this work Flynn has analyzed student explanations about organic reactions using a framework based partially on the granularity of the student’s discussion that also encompasses the number of causal links students make from descriptive to relational to linear causal to multicomponent causal40. Prior Studies on Which This Work is Based Most organic chemistry courses do not focus on the construction of causal mechanistic explanations, but rather on the construction of mechanisms using mechanistic arrows, and it is likely that for many organic chemists the ability to draw mechanisms would be seen as more convincing evidence of expertise. It is for this reason that we decided to reinvestigate an earlier set of studies from our group,22,23 in which we characterized both how students use mechanistic arrows to predict the products of a reaction,22 and whether the use of mechanistic arrows improves students’ success in predicting the correct answer.23 In these earlier studies, students responded to a set of tasks in which 47 they were asked to draw an arrow pushing mechanism to predict the product for a number of familiar and unfamiliar reactions at four timepoints across a full year of organic chemistry. These reactions are shown in Figure 2. We used a software system that allowed us to record and replay student responses41 so that the sequence in which students drew arrows and mechanisms could be determined. The reactions that are labeled Prompt A: Electrophilic Addition of Water to Alkene, Prompt B: Electrophilic Addition of Water to Alkyne, Prompt C: Alkyne Deprotonation Followed by SN2, and Prompt D: Nucleophilic Attack at a Carbonyl, represented tasks that were similar to those presented to students in class and in their textbooks, whereas Prompt E: Unfamiliar Reaction required students to apply their knowledge of organic reactivity to a situation that, to our knowledge, they not encountered before but should have been able to work through to produce a plausible product. In these earlier studies we found that many students tended to simply draw out products of known reactions A-D rather than draw mechanisms as we had asked in the prompt.22 At each time point where data were recorded, between 30-60% of students did not use any curved arrows to help predict the products of these reactions and of those who did draw arrows, between 15 and 20% of those students drew the arrows onto the reaction scheme after they had predicted the products. We also found that for familiar reactions (prompts A-D), student use of arrows did not affect the chances of drawing the correct structure.22 There are a number of possible explanations for this finding, it may be that they simply did not see the benefit of using arrows to produce a product that they already knew was the answer, or perhaps students simply memorized the answer. However, when students were faced with reactions that they had not seen before, but “should” have been able to make predictions about, the students who attempted to draw mechanisms were more likely to predict the correct product, even so, the overall percentage of students who drew a plausible product was very small (9% of the total). 48 Figure 4.2. Summary of mechanisms administered to participants in the papers.22,23 Highlighted prompt will be discussed in this paper. Our goal in this present study is to investigate how different students, at a different university address the same tasks that were studied earlier.22,23 In this study we analyze responses from two demographically matched cohorts of students, one group who were enrolled in a traditional organic chemistry course, similar to the earlier study, using a commercial text, and a lecture format. The second group of students were enrolled in a transformed course, Organic Chemistry, Life, the Universe and Everything.42 Research Questions: 1. In what ways are responses from students who are enrolled in a transformed organic course similar or different to an equivalent group of students from a Traditional organic course for familiar reactions? 2. In what ways are responses from students who are enrolled in a transformed organic course similar or different to an equivalent group of students from a Traditional organic course for an unfamiliar reaction? 49 Methods Student Participants: Present Study This study was conducted at a large midwestern research-intensive university. Students were enrolled in a two-semester organic chemistry course for non-chemistry majors. This study was designated “exempt” and all student participants were informed of their rights as research participants in accordance with our institutions’ IRB. Student participants were enrolled in either a transformed organic chemistry course, Organic Chemistry, Life, the Universe, and Everything (referred to as OCLUE) or a traditional organic chemistry course (referred to as Traditional). At this institution students register far in advance, and students typically enroll in sections before the instructor is listed for the course time, students also enroll for a whole year at one time. To ensure that the two groups of students were similar we compared several academic and demographic measures using a Mann-Whitney U test and calculated the effect size for any differences using Cramer’s V. While Traditional students had a slightly higher OChem 2 course GPA (Mann-Whitney: Traditional = 3.32, OCLUE = 3.01, U = 14308.5, z = -2.149, p = 0.032, r = 0.11 small effect size),43 no differences were found when comparing ACT scores, GPA Prior to Organic, OC1 course grade. Appendix B Tables 4.5-4.7 provides a summary of all statistical analyses, which were performed in SPSS,44 and a report of gender and major distributions for each cohort. Both types of section meet for either three 50 minute or two 80 minute lecture classes of around 300-350 students, and a one a week 50 minute recitation taught by graduate teaching assistants. The differences between OCLUE and Traditional courses are not a matter of topical coverage – indeed many students must switch sections between semesters for scheduling reasons, so it important that the same topics are “covered”. Additionally, this course is a service course and we are aware of what external expectations are for what students should know and be able to do. As we have previously discussed, the OCLUE curriculum emphasizes biologically important mechanisms, such as acid base 50 reactions, nucleophilic additions and substitutions, but also includes material required by the more traditional approaches should students switch sections type. The development of the curriculum for the transformed course OCLUE, has been reported previously,42 but will be reviewed briefly here. OCLUE is based on what has come to be known as three- dimensional learning (3DL).11 It is organized around four core ideas of chemistry: electrostatic forces and bonding interactions, structure property relationships, stability and change in chemical systems, and energy. These core ideas are used in the context of scientific and engineering practices (SEPs), such as analysis and interpretation of data, construction of evidence based arguments, explanations and models.11 The third dimension consists of crosscutting concepts that allow instructors and students to focus on a particular aspect of a phenomenon. The aspect of OCLUE relevant to this report is the emphasis on construction and use of models, construction of explanations and the crosscutting concept of cause and effect, that combine with the core ideas of structure-property relationships, and bonding and interactions. Assessment in the OCLUE course includes bi-weekly formative assessments delivered via an online system, beSocratic,45 that allows open-ended drawn and written responses, group recitation worksheets facilitated by graduate teaching assistants, in-class clicker questions, and summative exams that contain both multiple choice and open ended responses. Approximately 45% of the course grade comes from formative assessments that are designed to support and extend student learning, rather than test it. For example, in the bi-weekly formative assessments delivered on beSocratic students complete both retrospective tasks that are designed to consolidate ideas, and prospective tasks that introduce new material. This is done to allow students to use their knowledge and skills to predict outcomes for systems that will be discussed in the next class.42 Additionally, all of the formative assessments are graded based on student participation rather correctness, with the goal of providing students with a “safe space” to make mistakes without penalty. For all the formative assessments 51 (homework and recitation) students are provided with contextual feedback, either from a teaching assistant, or in class. The summative assessments are three mid-term and one final examination which typically evenly split between multiple choice and open response questions.42 Both the summative and formative assessments in OCLUE include 3D tasks, that is they require the student to construct, predict, explain, argue, analyze in the context of core ideas and crosscutting concepts. For example, students might be asked to draw a reaction energy diagram (a model) and use it to predict, explain, and draw mechanisms to show how different products are formed under different reaction conditions. That is students use the core ideas of energy and structure property relationships, the scientific practices of modelling and explanations, and the crosscutting concept of cause and effect. The Traditional course uses a commercial textbook46 that is organized mostly by functional group, and the topics in the course are typically taught in the same order as the textbook. By agreement, both Traditional and OCLUE sections cover the same material in the first semester, so that students who switch sections will have been exposed to the same general content, including the familiar reactions A-D. The course is taught in a lecture format, and there is ample opportunity to ask and answer questions, the instructor provides the course material and solves a wide range of problems for students throughout the lecture. Homework using the associated publishers’ online materials is suggested but not required. Recitation sections are also taught by graduate teaching assistants, and typically involve short quizzes worth 20% of their overall grade in the course, a short lecture, and/or a question-and-answer session. Eighty percent of students’ grades comes from performance on three summative exams worth 40% of the overall grade and a final exam also worth 40% of their overall grade. These exams are all open response (no multiple choice) and consist of short answer predict-the-product, -reactant or -reagent, draw a mechanism, analyze an unknown from spectra, and synthesize a given molecule. In other words, the assessments are typical organic chemistry questions that are found in many institutions, reflective of similar content assessed on the ACS organic exam.47 Just as Stowe et al. 52 found in their analysis of exams from elite institutions across the country,48 these questions are not 3D. In particular, they do not elicit evidence that students have engaged with the scientific practices. All exams are hand-graded, and feedback is provided on the examination by the instructor or graduate teaching assistant. In summary, while both are large enrollment courses with around 300-350 students per section that “cover” the same material, OCLUE and Traditional organic chemistry have different approaches to both curriculum design and assessment, and these differences appear to have led to a different class culture (manuscript in review). For example, students in a traditional section are more likely to say that they are assessed on what they can memorize, whereas OCLUE students are more likely to indicate that their use of knowledge is assessed. Although students may switch section type between the first and second semester, in this study we focus solely on students who were enrolled either in both semesters of OCLUE, or both semesters of Traditional organic chemistry. We will report on how students who “switch” sections fare in a subsequent publication. Prompt Timing and Administration The prompts A-E that were administered in the previous study (Figure 2) were also administered to students in the current study. Time Point One (two weeks into OChem 2) was chosen because it should provide information about what students had learned in OChem 1. Students were asked to respond to prompts A-D because these are typical reactions students would be expected to know and be able to do in OChem 1 regardless of the students' course type. Prompts A-D are all familiar reactions, and similar reactions to these were assessed by both course types in OChem 1. These will be referred to as familiar reactions from now on. At Time Point Two (one week before the end of the second semester) students were asked to respond again to prompts A-D, and to prompt E which involved the reaction that, to our knowledge, 53 students had not seen before in either the OCLUE or Traditional sections of OChem. Prompt E is referred to as an unfamiliar reaction in the earlier papers.22,23 Figure 3 shows a summary of the administrations of the prompts, and the numbers of students who answered each one. Students at Time Point Two were selected from the group of students who completed Time Point One, so that only students who had answered both prompts appear in this dataset. There was some attrition from Time Point One to Time point Two for both courses, about 10% for OCLUE and 5% for Traditional. Figure 4.3. Summary of the data collection. In the present study we used the online homework and research platform beSocratic to capture the mechanisms that students drew.45 Using this system students can draw reactions and mechanisms, just as they would on a hand-written paper assignment. The advantage of using this system is that the responses are recorded and can be replayed to show stroke-by-stroke what students drew and in what order they drew it. Figure 4 is an example of the screen students see when working in beSocratic. In the previous study a stand-alone system, OrganicPad,41 which has similar but more limited capabilities was used. This software is no longer available for use. 54 Potential for Bias We, as researchers, acknowledge that there is an inherent conflict when a researcher sets out to investigate the impacts of a curriculum or intervention of their own design. Such a conflict exists here, and in our implementation of the study and the data analysis we have attempted to remove or negate sources of bias. In both sections the activity was given online as an extra credit activity. For OCLUE this extra credit would contribute 0.1% to the overall grade, in the traditional sections the credit would add 1%. Both sections were familiar with the homework system since it is also used for general chemistry. Once the data were collected, they were de-identified and coded anonymously, so that coders were not aware of the source of the data. Figure 4.4. Example of a student’s completed mechanism in beSocratic. Data Analysis Our original intent was to use the original coding scheme22, but because of the differences in the way the data was recorded, for some of the prompts we were able to expand on the original and develop a richer coding scheme. This expanded revised coding scheme encompasses the original coding scheme, so it is still possible to compare to the students from the previous study and we can also provide here a richer picture of the mechanistic approaches used by students in the present study. In both the original work and current study, we are interested in not only the products students drew but 55 what steps they took to get there. A summary of the revised coding schemes for all five prompts are provided in Appendix C Figure 4.11-4.15. Because we are interested in appropriate use of mechanistic arrows, for the revised coding scheme we generated codes for every mechanistically plausible step that students took to get to a plausible product. That is, an arrow that starts at an electron rich site and ends at an electron sink, or generates an appropriate resonance structure is considered reasonable. For example, for prompt B, we expanded the plausible products to the enol, and the geminal diol (the hydrate of the carbonyl) and the vicinal diol. In the current scheme a plausible product is not only the major product but also the minor products that make mechanistic sense for this reaction. This approach is more consistent with the intent of the original coding scheme, it simply extends it to a somewhat broader range of products. The number of codes for each prompt A-E varied from 7-22 to fully capture and characterize students’ mechanistic pathways to the product (whether major or minor) which are provided in the in Appendix C Figure 4.11-4.15. There were also a small number of students who did not engage with the prompt, for example by drawing a line through the prompt or writing “I don’t know”. Over all prompts and time points there were 11 OCLUE students and 16 Traditional students who did not engage with any of prompts and thus were removed from the data set. Because there were so many codes for each reaction, we chose to determine inter-rater reliability (IRR) for each code, rather than for the mechanism as a whole. Two coders, authors (SKH) and a trained post-baccalaureate coder (RB), blind-coded the anonymized student responses so that the coders were not aware of the students’ background (course type) while coding. The two coders took sets of 15 responses and coded them then discussed differences to reach agreement and occasionally additional codes were added for other plausible ways students drew their arrows. Once agreement was reached on the codes, both coders evaluated 60 random responses from each prompt to obtain kappa values and the author (SKH) coded the remaining responses. Cohen’s Kappa values for all prompts 56 ranged between 0.78 and 1.0 were obtained for all prompts, indicating substantial agreement which are provided in in Appendix C Table 4.8-4.12.43,49 Since the mechanisms for each reaction often generated large numbers of codes, for the purposes of comparison the responses were assigned to one of five code groupings based on the description outlined in Figure 5. A plausible arrow refers to any arrow that starts at a source of electron density and ends at an electron sink. Figure 4.5. Overarching code groupings for all reactions. The color scheme shows the colors of the code groupings in the graphs in the results section. 57 Results Research Question 1: In what ways are responses from students who are enrolled in a transformed organic course similar or different to an equivalent group of students from a Traditional organic course for familiar reactions? Finding 1: The frequency of plausible products drawn depends on the reaction for familiar reactions The responses for prompts A-D were classified into two bins: students who proposed a plausible product as discussed earlier, and those who did not. Table 1 shows the percentage of students who drew a plausible product for prompts A-D at Time Point One and Time Point Two. We used a series of Chi-square tests of independence to compare the cohorts based on the percentage of students who drew a plausible product.50 While there are some statistically significant differences between the cohorts the effect size for all prompts A-D is small, indicating there is not a strong correlation between the course type and drawing a plausible product after both semesters of organic chemistry.43 In general, after one semester Traditional students are significantly better at drawing a plausible product for reaction A, but this difference is removed after two semesters. At Time Point Two, as one might expect all cohorts produced a plausible product more frequently than they did at Time Point One. It appears that after two semesters there is little difference in the ability of students from either cohort to predict the outcome of familiar reactions. Table 4.1. Percent of Students Who Drew a Plausible Product For all chi-square analysis α = 0.01. (α) students in OCLUE outperformed Traditional, (β)students in Traditional outperformed OCLUE. Time Point One Prompt OCLUE (count/total n) Traditional (count/total n) χ2 (df =1) p-value Cramer’s V A 58% (95/165) 69% (129/187) 4.930 β 0.026 β 0.12 β B 41% (68/167) 41% (76/185) 0.005 0.945 - C 14% (22/162) 4% (7/184) 10.722 α 0.001 α 0.18 α D 27% (44/164) 6% (11/183) 29.466 α <0.001 α 0.29 α Time Point Two Prompt OCLUE (count/total n) Traditional (count/total n) χ2 (df =1) p-value Cramer’s V A 76% (126/165) 75% (129/187) 0.044 0.833 - B 62% (104/167) 48% (88/185) 7.658α 0.006 α 0.15 α C 31% (50/162) 9% (17/184) 25.801α <0.001 α 0.27 α D 68% (112/164) 55% (101/183) 6.263 0.012 - 58 Finding 2: OCLUE students tend to use plausible arrows more frequently than Traditional Students We will now turn our attention to how students predicted the products: that is whether students used mechanistic arrows to predict the outcome of the reaction. As discussed earlier, at each Time Point, students were classified into five separate groups based on the product they produced and the way they used mechanistic arrows: 1) students who drew plausible arrows and a plausible product, 2) students who drew a mixture of mechanistically reasonable steps and incorrect arrows and a plausible product, 3) students who drew no arrows or incorrect arrows and a plausible product, 4) students who drew mixture of mechanistically reasonable steps and incorrect arrows and an incorrect product, and 5) those who drew no arrows or incorrect arrows and an incorrect product. It should be noted that in the earlier studies both groups 1 and 2 were “counted” as using mechanistic arrows in a fruitful manner. An example of a student response for each code is shown in Figure 5. Figure 6 shows the distribution of the five code groupings for prompt A at Time Point 1. The figures showing the percent of students who drew the various coding groups for Prompt B-D at both time points are provided in Appendix D. As shown in Table 2 at Time Point One for prompts A and B it appears that students enrolled in the OCLUE course were significantly more likely (with a large effect size) to draw all mechanistic steps correctly than those who were in a Traditional course. 59 Figure 4.6. Percent of students who drew all mechanistic steps correctly, students who drew some mechanistic steps correctly, students who drew no mechanistic steps correctly, and students who got the incorrect product based on the courses they took. Table 4.2. Time Point One: Percent of Students Who Drew All Plausible Arrows and Predicted a Plausible Product. For all chi-square analysis α = 0.01. Prompt OCLUE (count/total n) Traditional (count/total n) χ2 (df =2) p-value Cramer’s V A 40% (64/165) 11% (20/187) 71.449α <0.001 0.56 B 22% (37/167) 8% (15/185) 24.024α <0.001 0.41 C 6% (10/162) 1% (2/184) - - - D 17% (26/164) 3% (6/189) - - - At Time Point Two, the frequency with which both groups drew plausible mechanistic arrow increased from Time Point One. The frequency with which students draw all mechanistic steps correctly, now differs between the Traditional and OCLUE students across all prompts A-D as shown in Table 3. For example, Figure 7 shows the students who drew all mechanistic steps correctly for a simple alkene hydrolysis reaction. 60 Figure 4.7. Prompt A: Percent of students who drew all mechanistic steps correctly, students who drew some mechanistic steps correctly, students who drew no mechanistic steps correctly, and students who got the incorrect product based in the courses they took. Table 4.3. Time Point Two: Percent of Students Who Drew All Plausible Arrows and Predicted a Plausible Product. For all chi-square analysis α = 0.01. Prompt OCLUE (count/total n) Traditional (count/total n) χ2 (df =2) p-value Cramer’s V A 64% (106/165) 21% (38/187) 87.479 α <0.001 0.57 B 38% (64/167) 10% (18/185) 50.479 α <0.001 0.51 C 18% (29/162) 1% (2/184) 27.991 α <0.001 0.65 D 53% (88/164) 21% (38/183) 48.835 α <0.001 0.48 Finding 3: A higher percentage of OCLUE students improve their responses over the course of two semesters than Traditional Students Since this is a longitudinal study, student responses at both Time Point One and Time point Two were plotted on Sankey diagrams. Figure 8 shows how the responses provided by each student changed for Prompt A the Familiar Alkene. The width of the pathways between the two time points represents the proportion of students who took that path. For example, at Time Point One 40% of OCLUE students 61 who drew all plausible arrows and predicted a plausible product, while at time point 2 there were 64% who did this. However only 34% of OCLUE students drew all plausible arrows and predicted a plausible product at both. Similarly for Traditional students while the overall percent increases, only 3% of students completed the prompt correctly for both time points. We grouped the students by the change in coding groups between the two time points, increasing, decreasing, or stayed the same (tied), these will be referred to as ranks. Overall, there is a significant difference between the changes in the ranks for the OCLUE students and Traditional students with a medium effect size as shown in Table 4 (χ2 (2) = 24.141, p <0.001, Cramer’s V = 0.26). The specific differences between the ranks from Time Point One to Time Point Two show that very few students in OCLUE are decreasing in their rank over time (8%) while, Traditional students a quarter of student fell to a lower coding group by Time Point Two. 62 Figure 4.8. Prompt A: Percent of students who drew all mechanistic steps correctly, students who drew some mechanistic steps correctly, students who drew no mechanistic steps correctly, and students who got the incorrect product based in the courses they took. 63 Table 4.4. Prompt A Change in Ranks for Students from Time Point One to Time Point Two. For all chi- square analysis α = 0.01. OCLUE Traditional Ranks χ2 (df =2) p-value Cramer’s V (count/total n) (count/total n) Positive 40% (67/165) 47% (87/187) Tie 52% (85/165) 30% (56/187) 24.141 <0.001 0.26 Negative 8% (13/165) 23% (44/187) Research Question 2: In what ways are responses from students who are enrolled in a transformed organic course similar or different to an equivalent group of students from a Traditional organic course for an unfamiliar reaction? Thus far we have explored reactions (prompt A-D) that we know are familiar to students. However, to our knowledge, prompt E is an unfamiliar reaction that students have not seen before. Nevertheless, plausible products for this reaction can be predicted if students use mechanistic arrows as a prediction tool. For example, one might expect that students would use the lone pair on the alcohol oxygen to initiate intramolecular attack at a carbonyl carbon, producing a tetrahedral intermediate, followed by loss of the best leaving group (chloride) to product a lactone. Prompt E was administered only at Time Point Two near the end of the two-semester sequence to minimize the chance that students would have seen it before in an earlier administration. Analysis of the student responses indicated that there was significant difference between the percent of students who proposed a plausible structure for the product between the students in OCLUE (45%) and Traditional (8%) as shown in Figure 9. This difference is significant (χ2 (1) = 60.009, p <0.001, Cramer’s V = 0.42, medium-large effect size). The number of correct responses in our previous study was 9%, which is similar to the Traditional student cohort.23 Just as with the more familiar prompts A-D, OCLUE students were more likely to draw all mechanistic steps correctly than Traditional for the unfamiliar prompt. Analysis of the mechanistic approach taken by students is shown in Figure 9. The numbers of Traditional students who were able to complete this task appropriately are too small to compare statistically, but again we see a larger percent 64 of students from OCLUE sections both drawing appropriate mechanisms and producing a plausible product. Figure 4.9. Percent of students who attempted drawing a mechanism but still drew and incorrect product and student who only drew and incorrect product with no arrows. This finding further supports the idea that students who habitually draw mechanisms as part of course expectations to predict outcomes for reactions are more likely to produce plausible products. Further analysis of the students who produced an implausible product also showed differences between the two cohorts. For example, there were far fewer students in OCLUE who did not attempt some kind of plausible mechanism (23% OCLUE vs 71% Traditional) (Figure 9). As an illustration of this phenomenon we show Figure 10, which is a screenshot from beSocratic in which student responses are shown in a grid for both OCLUE and Traditional. These responses are representative samples of each cohort and show qualitatively the different approaches between the two student cohorts. We can see from inspection that OCLUE students are more likely to draw mechanistic arrows and intermediates than traditional students. 65 Figure 4.10. Screenshot from beSocratic in which student responses are shown in a grid for both OCLUE (top) and Traditional (bottom). 66 Research Question 3: How do the mechanism attempts from the earlier studies compare to the current study? The major goal of this study is to characterize ways in which students in a transformed and a traditional course respond to the prompts from the original study, however there is an opportunity here to briefly compare how students from the original and present studies respond. The students in the original study and the students in this study are comparable (demographic information is provided in Appendix E Table 4.14), but we have chosen only to compare the original students and the traditional cohort because they participated in a similar course environment. In the previous study, between 30 - 60% of students simply wrote down a product and did not attempt to draw a mechanism.22 Students were counted as not drawing a mechanism if they did not draw any arrows or intermediate structures when answering the prompt. Here we find a very similar pattern of responses, depending on the prompt, between 27-68% did not attempt to draw a mechanism, as shown Appendix E Table 4.15. Additionally, just as in the original study, where 15-20% of students drew their arrows after they had already drawn their product for all prompts22, there were students in the Traditional course who also drew a product before drawing a mechanism for between 9- 20% of their responses. It is striking that two different groups of students, in two different universities, ten years apart provide responses that appear quite similar.22 Discussion Our goal in this study was to characterize the ways in which students in two different organic chemistry courses, Traditional and OCLUE, complete a set of mechanism tasks that were first studied over ten years ago in a Traditional course setting. Our findings indicate that after one semester there is little difference in the ability of students from either course to draw a plausible product for familiar reactions. By the end of two semesters (Time Point Two) both cohorts had improved in their prediction of products, and for prompts A, B, and D at least 50% of students were able to produce a plausible product. Both cohorts were less successful in predicting the outcome of prompt C, which involves 67 deprotonation of an alkyne, which appeared not well remembered, and without sufficient knowledge students would not be able to predict this outcome. Again, any significant differences in product prediction were marked by small effect sizes. However, the approach that students use to reach the known product does seem to differ. At both time points, OCLUE students were significantly more likely to draw plausible arrows to produce a chemically feasible product, than were Traditional students. By the end of the second semester this difference was significant with a large effect size for all familiar tasks (A-D). Perhaps we should not be surprised that many students opt not to draw a mechanism for a known reaction. The idea that organic chemistry requires a huge amount of memorization is well recognized both as part of student lore, and has been documented in a number of research studies51,52. As Graulich noted “It is evident from the current studies that students still rely heavily on rote- memorization and that traditional give-the-product exercises are frequently solved without a deeper understanding.”10 It is almost certainly not the intent of organic instructors that this should be the case, but it is clear that this idea is quite pervasive, and as such may impact how students address the work in the course. In a recent study (in review) when students were asked in an open response survey about what they were assessed on in an organic course, a majority of Traditional students responded that they were assessed on their ability to memorize a large amount of material. Perhaps what is more interesting is the larger percentage of OCLUE students who do attempt to draw mechanisms, even for familiar reactions. It is our hypothesis that the willingness to draw mechanisms stems from the design and enactment of the course itself, and we believe that there are two particular characteristics that contribute to this. The first is that students routinely (at least two or three times a week) complete three-dimensional homework and recitation tasks, where (when relevant) reaction mechanisms and causal mechanistic explanations are a central focus. These formative assessments account for approximately 45% of the course grade and are graded based on student 68 participation rather correctness, with the intent of fostering a “safe space” for students to make mistakes without penalty. Such tasks are designed to support the connection of resources that serve to support both the explanation and the model. For example, students might be asked to construct an explanation for both how and why a reaction proceeds as it does (using resources such as coulombic interactions and electron density distributions), and then asked to construct a mechanism35,36. That is, for students the explanation and mechanism are typically part of the same activity, which may help consolidate the connection of cognitive resources. One of the major findings of studies on learning is that the development of expertise involves supporting the connection of knowledge into a coherent framework53,54. We know that experts’ knowledge frameworks are more likely to involve connections, so that the knowledge is contextualized and useful. Indeed, the efforts to design learning experiences around “big ideas” are intended to help student connect their knowledge in order to develop more expert-like frameworks11,42,55–61. These connections are established by helping students use their knowledge – for example by constructing models and explanations. Indeed, in the study discussed earlier (in review) a majority of OCLUE students reported that they were assessed on how they use their knowledge, rather than what they know. A consequence of this design is that the connection between the resources required to construct an explanation and those to construct a mechanism in a meaningful way (i.e., from source to sink) is made quite explicit to students, and we believe, allows a greater number of students to attempt to draw a mechanism for a reaction with which they were unfamiliar. The second characteristic is that during most of the course activities students are not penalized for making mistakes, on homework or in recitation. Indeed, for many activities we explicitly ask them to try to figure out what is happening. Students receive full points for a “good faith effort” and are therefore more likely to try to address the task at hand. Indeed, as Bodner once said, when solving problems it is important to “try something” then “try something else”62. Often the first step is 69 addressing any unknown task is that first step, and we know that some students are unwilling to begin a task if they do not know where it is going63,64. Regardless of why OCLUE students are more likely to use arrow pushing mechanisms to predict outcomes of reactions, when we look at the responses for the unfamiliar reaction E, we see that OCLUE students significantly outperform their traditional peers in drawing plausible arrows and predicting a plausible product, 27% and 2% respectively. Just as we found earlier, students who use mechanistic arrows are more likely to predict plausible products, than those who do not. The difference between this study and the earlier ones is that, in the earlier studies (and similar) to Traditional students, only 9% of students were able to predict a reasonable product, compared to 45% of OCLUE students in this study, 42% of whom used at least one plausible mechanistic arrow. In summary, these two matched cohorts are of similar ability, when it comes to predicting a product for a familiar reaction, but there is a significant difference in the outcome for an unfamiliar reaction. We ascribe this difference to two main factors: 1) the explicit treatment of mechanisms and explanations as complementary practices, that use the same cognitive resources, and 2) the learning environment of the OCLUE course were students are given the freedom to make errors without penalty, which may make students more willing to attempt a response in the face of a new task. Implications for Teaching For those who value mechanistic reasoning in organic chemistry but are unwilling or unable to completely transform a course, there are several actionable approaches that may improve outcomes. 1) Emphasize the explanatory and predictive power of mechanistic reasoning early and often in the context of both using mechanistic arrows, and constructing written explanations about how and why reactions occur in a particular way. If we want students to use mechanistic arrows and causal mechanistic reasoning, we have to ensure that students’ understand what their purpose is. As discussed earlier we have previously shown that students who are able to construct causal mechanistic written 70 explanations (that is, for both how and why reactions occur) are also more likely to draw appropriate mechanistic arrow models.35,36 The act of constructing an explanation can help link and consolidate ideas leading to a more expert-like understanding. 2) Provide frequent opportunities for students to practice such model construction and use, with formative assessment tasks that count towards the ultimate grade, but that are rewarded for effort not correctness. Obviously, these formative tasks must come with appropriate feedback, which can be provided in various ways. In OCLUE, because of the large numbers involved, feedback cannot be provided to individual students – but rather homework responses are used to drive the next class discussion. These activities provide students with opportunities to explore their ideas, and then they can go back and reconstruct answers that need more work. Students also have the opportunity to engage with these idea during their weekly recitation and receive feedback on their work from graduate teaching assistants. It has also been shown that feedback is more likely to be addressed on non-graded assessments.65 3) To ensure that students get the message such tasks should be incorporated in summative examinations. That is students should be asked to explain, as well as, draw products for reactions. (We note that there are other important aspects of organic chemistry not discussed here such as constructing an argument from spectroscopic evidence, developing syntheses of desired compounds, and predicting kinetic and thermodynamic outcomes for reactions, using multi-component reasoning, but these are not the focus of this report) In summary, we believe that the goal of education should be to provide students with the tools to use their knowledge in a productive manner, to think flexibly and to take that leap into the unknown. To do this requires more than changing how we teach, it also requires us to think about what is important and change what we teach and how we assess what students know and can do. If students 71 expect that success can be achieved by memorization, then we should not be surprised when this is what they do. Limitations As noted in the results section, the OCLUE curriculum was developed by a team that includes our research group, and as such there is a potential for bias not only in the analysis of the results but in their interpretation. We have attempted to minimize this bias as discussed earlier, but readers should be aware that it exists. This study was a repeat of a previous study with different student groups. However, the same prompts that were administered in the original study were also administered in this new study. The original study included multiple familiar prompts but only one unfamiliar prompt which limits the generalizability to more complex unfamiliar questions. Additionally, the original software (OrganicPad)41 we used to deliver the tasks was obsolete. In the original study the students used tablet computers that were provided for this purpose in a laboratory setting that was not connected directly to the course where the material was learned. In the new study, students completed the tasks online on the beSocratic system for homework. That being said, there is a remarkable similarity in the patterns of responses for Traditional students. Additionally, in the present study OCLUE students do use the beSocratic system for homework on a weekly basis, which may mean that they were more familiar with the system. However, most (over 95%) of these students in both OCLUE and Traditional had taken other courses in which the beSocratic system is used for homework, and they had also previously completed several other extra credit tasks on this system. 72 REFERENCES (1) Seymour, E.; Hunter, A.-B. Talking about Leaving Revisited: Persistence, Relocation, and Loss in Undergraduate STEM Education; Springer International Publishing: New York, 2019. (2) Stowe, R. L.; Cooper, M. M. Practicing What We Preach: Assessing “Critical Thinking” in Organic Chemistry. Journal of Chemical Education 2017, 94 (12), 1852–1859. https://doi.org/10.1021/acs.jchemed.7b00335. (3) Bhattacharyya, G. From Source to Sink: Mechanistic Reasoning Using the Electron-Pushing Formalism. J. Chem. Educ. 2013, 90 (10), 1282–1289. https://doi.org/10.1021/ed300765k. (4) Gerlach, K.; Trate, J.; Blecking, A.; Geissinger, P.; Murphy, K. Valid and Reliable Assessments To Measure Scale Literacy of Students in Introductory College Chemistry Courses. J. Chem. Educ. 2014, 91 (10), 1538–1545. https://doi.org/10.1021/ed400471a. (5) Bhattacharyya, G. Trials and Tribulations: Student Approaches and Difficulties with Proposing Mechanisms Using the Electron-Pushing Formalism. Chem. Educ. Res. Pract. 2014, 15 (4), 594– 609. https://doi.org/10.1039/C3RP00127J. (6) Flynn, A. B.; Featherstone, R. B. Language of Mechanisms: Exam Analysis Reveals Students’ Strengths, Strategies, and Errors When Using the Electron-Pushing Formalism (Curved Arrows) in New Reactions. Chem. Educ. Res. Pract. 2017, 18 (1), 64–77. https://doi.org/10.1039/C6RP00126B. (7) Webber, D. M.; Flynn, A. B. How Are Students Solving Familiar and Unfamiliar Organic Chemistry Mechanism Questions in a New Curriculum? J. Chem. Educ. 2018, 95 (9), 1451–1467. https://doi.org/10.1021/acs.jchemed.8b00158. (8) Anzovino, M. E.; Lowery Bretz, S. Organic Chemistry Students’ Ideas about Nucleophiles and Electrophiles: The Role of Charges and Mechanisms. Chem. Educ. Res. Pract. 2015, 16 (4), 797– 810. https://doi.org/10.1039/C5RP00113G. (9) Bhattacharyya, G.; Bodner, G. M. “It Gets Me to the Product”: How Students Propose Organic Mechanisms. J. Chem. Educ. 2005, 82 (9), 1402. https://doi.org/10.1021/ed082p1402. (10) Graulich, N. The Tip of the Iceberg in Organic Chemistry Classes: How Do Students Deal with the Invisible? Chem. Educ. Res. Pract. 2015, 16 (1), 9–21. https://doi.org/10.1039/C4RP00165F. (11) A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas; National Academies Press: Washington, D.C., 2012. (12) Visualization in Science Education; Gilbert, J., Ed.; Models and modeling in science education; Springer: Dordrecht, 2005. (13) Finkenstaedt-Quinn, S. A.; Watts, F. M.; Petterson, M. N.; Archer, S. R.; Snyder-White, E. P.; Shultz, G. V. Exploring Student Thinking about Addition Reactions. J. Chem. Educ. 2020, 97 (7), 1852–1862. https://doi.org/10.1021/acs.jchemed.0c00141. 73 (14) Helping Students Make Sense of the World Using Next Generation Science and Engineering Practices; National Science Teachers Association, 2017. https://doi.org/10.2505/9781938946042. (15) Cooper, M. M.; Underwood, S. M.; Hilley, C. Z.; Klymkowsky, M. W. Development and Assessment of a Molecular Structure and Properties Learning Progression. J. Chem. Educ. 2012, 89 (11), 1351–1357. https://doi.org/10.1021/ed300083a. (16) Cooper, M. M.; Corley, L. M.; Underwood, S. M. An Investigation of College Chemistry Students’ Understanding of Structure–Property Relationships. J. Res. Sci. Teach. 2013, 50 (6), 699–721. https://doi.org/10.1002/tea.21093. (17) Becker, N. M.; Noyes, K.; Cooper, M. M. Characterizing Students’ Mechanistic Reasoning about London Dispersion Forces. J. Chem. Educ. 2016, 93 (10), 1713–1724. https://doi.org/10.1021/acs.jchemed.6b00298. (18) Noyes, K.; Cooper, M. M. Investigating Student Understanding of London Dispersion Forces: A Longitudinal Study. J. Chem. Educ. 2019, 96 (9), 1821–1832. https://doi.org/10.1021/acs.jchemed.9b00455. (19) Russ, R. S.; Scherr, R. E.; Hammer, D.; Mikeska, J. Recognizing Mechanistic Reasoning in Student Scientific Inquiry: A Framework for Discourse Analysis Developed from Philosophy of Science. Science Education 2008, 92 (3), 499–525. https://doi.org/10.1002/sce.20264. (20) Krist, C.; Schwarz, C. V.; Reiser, B. J. Identifying Essential Epistemic Heuristics for Guiding Mechanistic Reasoning in Science Learning. Journal of the Learning Sciences 2019, 28 (2), 160– 205. https://doi.org/10.1080/10508406.2018.1510404. (21) Hammer, D. Student Resources for Learning Introductory Physics. American Journal of Physics 2000, 68 (S1), S52–S59. https://doi.org/10.1119/1.19520. (22) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844–849. https://doi.org/10.1021/ed2003934. (23) Grove, N. P.; Cooper, M. M.; Cox, E. L. Does Mechanistic Thinking Improve Student Success in Organic Chemistry? J. Chem. Educ. 2012, 89 (7), 850–853. https://doi.org/10.1021/ed200394d. (24) Flynn, A. B.; Ogilvie, W. W. Mechanisms before Reactions: A Mechanistic Approach to the Organic Chemistry Curriculum Based on Patterns of Electron Flow. J. Chem. Educ. 2015, 92 (5), 803–810. https://doi.org/10.1021/ed500284d. (25) Dood, A. J.; Dood, J. C.; Cruz-Ramírez de Arellano, D.; Fields, K. B.; Raker, J. R. Using the Research Literature to Develop an Adaptive Intervention to Improve Student Explanations of an S N 1 Reaction Mechanism. J. Chem. Educ. 2020, 97 (10), 3551–3562. https://doi.org/10.1021/acs.jchemed.0c00569. (26) Mistry, N.; Nicholson, S. Investigating Students Understanding of Reaction Mechanisms from Performing Chemistry Experiments. 2019, 14 (1), 6. 74 (27) Carle, M. S.; Visser, R.; Flynn, A. B. Evaluating Students’ Learning Gains, Strategies, and Errors Using OrgChem101’s Module: Organic Mechanisms—Mastering the Arrows. Chem. Educ. Res. Pract. 2020, 21 (2), 582–596. https://doi.org/10.1039/C9RP00274J. (28) Visser, R.; Flynn, A. B. What Are Students’ Learning and Experiences in an Online Learning Tool Designed for Cognitive and Metacognitive Skill Development? CELT 2018, 11. https://doi.org/10.22329/celt.v11i0.5039. (29) Caspari, I.; Kranz, D.; Graulich, N. Resolving the Complexity of Organic Chemistry Students’ Reasoning through the Lens of a Mechanistic Framework. Chem. Educ. Res. Pract. 2018, 19 (4), 1117–1141. https://doi.org/10.1039/C8RP00131F. (30) Graulich, N.; Caspari, I. Designing a Scaffold for Mechanistic Reasoning in Organic Chemistry. Chemistry Teacher International 2021, 3 (1), 19–30. https://doi.org/10.1515/cti-2020-0001. (31) Watts, F. M.; Schmidt-McCormack, J. A.; Wilhelm, C. A.; Karlin, A.; Sattar, A.; Thompson, B. C.; Gere, A. R.; Shultz, G. V. What Students Write about When Students Write about Mechanisms: Analysis of Features Present in Students’ Written Descriptions of an Organic Reaction Mechanism. Chem. Educ. Res. Pract. 2020, 21 (4), 1148–1172. https://doi.org/10.1039/C9RP00185A. (32) Schmidt-McCormack, J. A.; Judge, J. A.; Spahr, K.; Yang, E.; Pugh, R.; Karlin, A.; Sattar, A.; Thompson, B. C.; Gere, A. R.; Shultz, G. V. Analysis of the Role of a Writing-to-Learn Assignment in Student Understanding of Organic Acid–Base Concepts. Chem. Educ. Res. Pract. 2019, 20 (2), 383–398. https://doi.org/10.1039/C8RP00260F. (33) Pashler, H.; Bain, P. M.; Bottge, B. A.; Graesser, A.; Koedinger, K.; McDaniel, M.; Metcalfe, J. Organizing Instruction and Study to Improve Student Learning, 2007. https://doi.org/10.1037/e607972011-001. (34) Cooper, M. M. Why Ask Why? J. Chem. Educ. 2015, 92 (8), 1273–1279. https://doi.org/10.1021/acs.jchemed.5b00203. (35) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid–Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703–1712. https://doi.org/10.1021/acs.jchemed.6b00417. (36) Crandell, O. M.; Lockhart, M. A.; Cooper, M. M. Arrows on the Page Are Not a Good Gauge: Evidence for the Importance of Causal Mechanistic Explanations about Nucleophilic Substitution in Organic Chemistry. J. Chem. Educ. 2020, 97 (2), 313–327. https://doi.org/10.1021/acs.jchemed.9b00815. (37) Talanquer, V. Importance of Understanding Fundamental Chemical Mechanisms. J. Chem. Educ. 2018, 95 (11), 1905–1911. https://doi.org/10.1021/acs.jchemed.8b00508. (38) Weinrich, M. L.; Talanquer, V. Mapping Students’ Modes of Reasoning When Thinking about Chemical Reactions Used to Make a Desired Product. Chem. Educ. Res. Pract. 2016, 17 (2), 394– 406. https://doi.org/10.1039/C5RP00208G. 75 (39) Talanquer, V. Progressions in Reasoning about Structure–Property Relationships. Chem. Educ. Res. Pract. 2018, 19 (4), 998–1009. https://doi.org/10.1039/C7RP00187H. (40) Bodé, N. E.; Deng, J. M.; Flynn, A. B. Getting Past the Rules and to the WHY: Causal Mechanistic Arguments When Judging the Plausibility of Organic Reaction Mechanisms. J. Chem. Educ. 2019, 96 (6), 1068–1082. https://doi.org/10.1021/acs.jchemed.8b00719. (41) Cooper, M. M.; Grove, N. P.; Pargas, R.; Bryfczynski, S. P.; Gatlin, T. OrganicPad: An Interactive Freehand Drawing Application for Drawing Lewis Structures and the Development of Skills in Organic Chemistry. Chem. Educ. Res. Pract. 2009, 10, 296–301. https://doi.org/10.1039/B920835F. (42) Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96 (9), 1858–1872. https://doi.org/10.1021/acs.jchemed.9b00401. (43) Cohen, J. A Coefficient of Agreement for Nominal Scales. Educ. Psychol. Meas. 1960, 20 (1), 37– 46. https://doi.org/10.1177/001316446002000104. (44) IBM Corp. SPSS Statistics for Windows; IBM Crop.: Armonk, NY, USA, 2017. (45) Bryfczynski, S. P. BeSocratic: An Intelligent Tutoring System for the Recognition, Evaluation, and Analysis of Free-Form Student Input. Ph.D. Dissertation, Clemson University, SC, 2010. (46) Wade, L. G.; Simek, J. W. Organic Chemistry, 9th ed.; Pearson: New York City, New York, 2017. (47) Raker, J.; Holme, T.; Murphy, K. The ACS Exams Institute Undergraduate Chemistry Anchoring Concepts Content Map II: Organic Chemistry. J. Chem. Educ. 2013, 90 (11), 1443–1445. https://doi.org/10.1021/ed400175w. (48) Stowe, R. L.; Cooper, M. M. Practicing What We Preach: Assessing “Critical Thinking” in Organic Chemistry. J. Chem. Educ. 2017, 8. (49) Landis, J. R.; Koch, G. G. The Measurement of Observer Agreement for Categorical Data. Biometrics 1977, 33 (1), 159. https://doi.org/10.2307/2529310. (50) Green, S.; Salkind, N. Using SPSS for Windows and Macintosh: Analyzing and Understanding Data; Pearson Education Inc: Boston, MA, USA, 2010. (51) Grove, N. P.; Bretz, S. L. Perry’s Scheme of Intellectual and Epistemological Development as a Framework for Describing Student Difficulties in Learning Organic Chemistry. Chem. Educ. Res. Pract. 2010, 11 (3), 207–211. https://doi.org/10.1039/C005469K. (52) Grove, N. P.; Lowery Bretz, S. A Continuum of Learning: From Rote Memorization to Meaningful Learning in Organic Chemistry. Chem. Educ. Res. Pract. 2012, 13 (3), 201–208. https://doi.org/10.1039/C1RP90069B. (53) How People Learn: Brain, Mind, Experience, and School: Expanded Edition; National Academies Press: Washington, D.C., 2000; p 9853. https://doi.org/10.17226/9853. 76 (54) Committee on How People Learn II: The Science and Practice of Learning; Board on Behavioral, Cognitive, and Sensory Sciences; Board on Science Education; Division of Behavioral and Social Sciences and Education; National Academies of Sciences, Engineering, and Medicine. How People Learn II: Learners, Contexts, and Cultures; National Academies Press: Washington, D.C., 2018; p 24783. https://doi.org/10.17226/24783. (55) Cooper, M.; Klymkowsky, M. Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform. Journal of Chemical Education 2013, 90 (9), 1116–1122. https://doi.org/10.1021/ed300456y. (56) Sevian, H.; Talanquer, V. Rethinking Chemistry: A Learning Progression on Chemical Thinking. Chem. Educ. Res. Pract. 2014, 15 (1), 10–23. https://doi.org/10.1039/C3RP00111C. (57) College Board. Advances in AP: AP Chemistry. (58) College Board. College Board Standards for College Success: Science. (59) Holme, T.; Luxford, C.; Murphy, K. Updating the General Chemistry Anchoring Concepts Content Map. J. Chem. Educ. 2015, 92 (6), 1115–1116. https://doi.org/10.1021/ed500712k. (60) Holme, T.; Murphy, K. The ACS Exams Institute Undergraduate Chemistry Anchoring Concepts Content Map I: General Chemistry. J. Chem. Educ. 2012, 89 (6), 721–723. https://doi.org/10.1021/ed300050q. (61) Laverty, J. T.; Underwood, S. M.; Matz, R. L.; Posey, L. A.; Carmel, J. H.; Caballero, M. D.; Fata- Hartley, C. L.; Ebert-May, D.; Jardeleza, S. E.; Cooper, M. M. Characterizing College Science Assessments: The Three-Dimensional Learning Assessment Protocol. PLOS ONE 2016, 11 (9), e0162333. https://doi.org/10.1371/journal.pone.0162333. (62) Bodner, G. M.; Herron, J. D. Problem-Solving in Chemistry. In Chemical Education: Towards Research-based Practice; Gilbert, J. K., Jong, O., Justi, R., Treagust, D. F., Driel, J. H., Eds.; Science & Technology Education Library; Kluwer Academic Publishers: Dordrecht, 2003; Vol. 17, pp 235– 266. https://doi.org/10.1007/0-306-47977-X_11. (63) Sandi-Urena, S.; Cooper, M. M.; Gatlin, T. A.; Bhattacharyya, G. Students’ Experience in a General Chemistry Cooperative Problem Based Laboratory. Chem. Educ. Res. Pract. 2011, 12 (4), 434–442. https://doi.org/10.1039/C1RP90047A. (64) Schoenfeld, A. H. Explorations of Students’ Mathematical Beliefs and Behavior. J. Res. Math. Educ. 1989, 20 (4), 338–355. https://doi.org/doi:10.2307/749440. (65) Winstone, N. E.; Nash, R. A.; Rowntree, J.; Parker, M. ‘It’d Be Useful, but I Wouldn’t Use It’: Barriers to University Students’ Feedback Seeking and Recipience. Stud. High Educ. 2017, 42 (11), 2026–2041. https://doi.org/10.1080/03075079.2015.1130032. 77 APPENDIX A: PERMISSIONS Figure 4.11. Permissions to reproduce manuscript in its entirety. 78 APPENDIX B: STUDIES PARTICIPANT DEMOGRAPHICS AND INTERRATER RELIABILITY Table 4.5. Mann – Whitney Comparison of Traditional and OCLUE students.43,44,50 Measure Cohort N Mean Mann-Whitney U z p-value Effect Size r Traditional 189 26.53 ACT 15225.00 -1.134 0.257 OCLUE 173 27.01 GPA Prior Traditional 189 3.58 15055.00 -1.336 0.182 to F17 OCLUE 173 3.60 OC1 Course Traditional 189 3.58 14.582.00 -2.054 0.040 Grade OCLUE 173 3.65 OC2 Course Traditional 189 3.32 14308.50 -2.149 0.032 0.113 small Grade OCLUE 173 3.01 Table 4.6. Description of gender of Traditional and OCLUE students.43,44,50 Deg of Cohort N Male (N) Female (N) Pearson Chi-Square p-value Freedom Traditional 189 28% (52) 72% (137) 0.172 1 0.679 OCLUE 173 29% (51) 71% (122) Table 4.7. Description of intended major of Traditional and OCLUE students. Preprofessional and Animal and Plant Physical Science Cohort N Other Health Sciences Sciences and Engineering Traditional 189 74% 17% 2% 6% OCLUE 173 83% 11% 3% 3% 79 APPENDIX C: INDIVIDUAL CODING MECHANISMS AND INTERRATER RELIABILITY Figure 4.12. Prompt A: Familiar Reaction. Figure 4.13. Prompt B: Electrophilic Addition of Water to Alkyne. 80 Figure 4.14. Prompt C: Alkyne Deprotonation Followed by SN2. 81 Figure 4.15. Prompt D: Nucleophilic Attack at a Carbonyl. 82 Figure 4.16. Prompt E: Unfamiliar Reaction. Table 4.8. Interrater Reliability Prompt A. Code (N=60) Kappa Protonate Alkene 0.96 Primary Carbocation 0.95 Secondary Carbocation 0.92 Attack with Water #1 0.87 Attack with Water #2 0.86 Deprotonate #1 0.90 Deprotonate #2 0.92 Product Primary Alcohol 0.93 Product Secondary Alcohol 0.98 No Mechanism Only Product 1.00 Drew Product First Then 0.83 Mechanism 83 Table 4.9. Interrater Reliability Prompt B. Code (N=60) Kappa Protonate Alkyne 0.83 Secondary Carbocation #1 0.86 Attack with Water #1 0.90 Collapse Oxygen Lone Pair 0.81 Protonate Alkene #1 0.83 Protonate Alkene #2 0.88 Primary Carbocation 0.90 Secondary Carbocation #2 0.92 Attack with Water #2 0.79 Attack with Water #3 0.81 Deprotonate #1 0.86 Deprotonate #2 0.90 Deprotonate #3 0.87 Deprotonate #4 0.87 Stop at Enol Product 0.96 Ketone 0.91 Diol (Geminal) 0.88 Diol (Vicinal) 0.91 No Mechanism Only Product 0.86 Drew Product First Then 0.97 Mechanism Table 4.10. Interrater Reliability Prompt C. Code (N=60) Kappa Deprotonate Alkyne 0.98 Attack with Alkyne Br 0.89 Leaves (SN2 Like) Br Leaves (SN1 like) 0.92 Attack with Alkyne at 1.00 Carbocation Product 2-Hexyne 0.96 No Mechanism Only 0.98 Product Drew Product First Then 1.00 Mechanism 84 Table 4.11. Interrater Reliability Prompt D. Code (N=60) Kappa Nucleophilic Attack First 0.98 Protonate First 0.87 Protonate Carbonyl #1 0.80 Resonance structure 0.89 Protonate second 0.83 Nucleophilic Attack second 0.92 Protonate resonance 0.82 structure Attack resonance 0.96 carbocation Protonate nitrogen #1 0.78 H2O Attack #1 0.96 Protonate Nitrogen #2 0.99 Remake Carbonyl #1 0.83 Break nitrogen double 0.90 bond Protonate Carbonyl #2 0.87 Protonate Nitrogen #3 0.85 H2O Attack #2 0.92 Break Carbonyl 0.79 Remake Carbonyl #2 0.89 NH3 Leaves 0.88 Nitrile product 0.92 Carboxylic Acid Product 0.85 No Mechanism Only 0.92 Product Drew Product First Then 0.98 Mechanism 85 Table 4.12. Interrater Reliability Prompt E. Code (N=60) Kappa Protonate Carbonyl 0.86 OH Attack Carbonyl 0.90 H2O Attack 0.78 OH Collapses Lone Pair #1 0.78 Cl leaves 0.97 (SN2 like) #1 Cl leaves 0.94 (SN1 like) OH collapses lone pair #2 0.99 Cl leaves 0.93 (SN2 like) #2 OH collapses lone pair #2 0.93 FC after Cl leaves 0.97 Deprotonate #1 0.98 Deprotonate #2 0.93 Deprotonate #3 0.84 Deprotonate #4 0.94 Deprotonate #5 0.90 Heterocycle with Carbonyl 0.80 Product Carboxylic Acid and OH 0.93 Product No Mechanism Only Product 1.00 Drew Product First Then 0.93 Mechanism Protonate Carbonyl 0.86 OH Attack Carbonyl 0.90 86 APPENDIX D: ADDITIONAL RESULTS Table 4.13. Time Point One: Percent of Students Who Drew All Arrows Correctly and Predicted a Plausible Product. For all chi-square analysis α = 0.01. (α) students in OCLUE out preformed Traditional. Prompt OCLUE Traditional χ2 (df =2) p-value Cramer’s V A 40% 11% 71.449α <0.001α 0.56α B 22% 8% 24.024α <0.001α 0.41α C 6% 1% - - - D 17% 3% - - - Figure 4.17. Time Point One Prompt B distribution of students’ arrows and product based on the courses they took. Figure 4.18. Time Point One Prompt C distribution of students’ arrows and product based on the courses they took. 87 Figure 4.19. Time Point One Prompt D distribution of students’ arrows and product based on the courses they took. Figure 4.20. Time Point Two Prompt B distribution of students’ arrows and product based on the courses they took. Figure 4.21. Time Point Two Prompt C distribution of students’ arrows and product based on the courses they took. 88 Figure 4.22. Time Point Two Prompt D distribution of students’ arrows and product based on the courses they took. Table 4.14. Percent of Traditional Students in the Current Study Who Drew a Plausible Product Before Drawing Mechanistic Arrows. Time Point One Time Point Two Prompt Percent Prompt Percent A 14% A 20% B 12% B 14% C 10% C 10% D 9% D 12% Was not administered E E 9% at Time Point One Table 4.15. Percent of students who drew the Major product for prompts A-E at Time Point Two. Prompt OCLUE Traditional A 74% 60% B 20% 9% C 31% 9% D 65% 55% E 41% 7% 89 APPENDIX E: ADDITIONAL DEMOGRAPHIC COMPARISON WITH EARLIER STUDY In the present study we will also compare Traditional students’ responses to those reported in the Original study Decorating With Arrows22. It was not our intent to replicate the previous study with a comparable cohort of students, but rather to extend the findings of that study to new student populations. Although no specific SAT data were reported for the previously published study, which was carried out at a Southeastern research-intensive institution, by using publicly available records the average SAT score is 1310, (28 ACT), while the present institution the average ACT score 26 (SAT 1230- 1250) and the students enrolled in these organic courses also have an average ACT of 26 (1230-1250 SAT). That is the earlier study was performed at a somewhat more selective institution. S5 Table shows that the make-up of the classes based on major and gender was similar. Table 4.16. Description of student from the Previous Studies and Current Study. Descriptor Previous Study Current Study Traditional (n=198) Preprofessional Major 70% 79% Gender (percent female) 60% 71% Table 4.17. How mechanism attempts from the earlier studies compare to the current study. Time Point One: Percent of Students Who Did Not Draw a Mechanism Prompt Previous Paper Current Study Traditional Prompt A: Familiar Reaction 43% 50% Prompt B: Electrophilic Addition of Water to Alkyne 38% 53% Prompt C: Alkyne Deprotonation Followed by SN2 34% 67% Prompt D: Nucleophilic Attack at a Carbonyl - 64% Time Point Two: Percent of Students Who Did Not Draw a Mechanism Prompt Previous Paper Current Study Traditional Prompt A: Familiar Reaction 30% 27% Prompt B: Electrophilic Addition of Water to Alkyne 41% 34% Prompt C: Alkyne Deprotonation Followed by SN2 41% 46% Prompt D: Nucleophilic Attack at a Carbonyl 36% 38% Prompt E: Unfamiliar Reaction 60% 43% 90 CHAPTER V – “WHAT ABOUT THE STUDENTS WHO SWITCHED COURSE TYPE?”: AN INVESTIGATION OF INCONSISTENT COURSE Introduction Driven by calls to action1, funding agencies’ priorities2, evidence about what and how students are learning,1,3 and calls for more equitable approaches to learning,4,5 the impetus to transform higher education STEM teaching and learning has become increasingly urgent. Proposed improvements have ranged from relatively minor add-ons such as technology-based tutorials designed to improve particular skills,6 to changes in pedagogy that engage students in their learning,7–9 to whole course and curricular transformations that may incorporate all of these mechanisms for improvement10–13. In some cases, there have been demonstrated improvements in overall grades and in students’ ability to construct explanations and models.10,12,14 However, despite all the resources and research evidence to support various transformations, studies show that the uptake of these approaches has been slow: Traditional lecture is still the predominant mode of instruction for most higher education STEM courses15, and students rarely called upon to use their knowledge to model, predict or explain phenomena either during instruction or on course assessments.16 Certainly, there are many reasons for the glacial pace of change including: the typical faculty reward structure is not designed to address instructional excellence, a satisfaction with the status quo (it was good enough for me), and an inconsistent approach to how meaningful change can be incentivized, recognized, and sustained. Indeed, much of the research on change in higher education points to the need for an integrated approach in which researchers, practitioners, administrators, and faculty developers all contribute to the proposed transformation,17–19 and that a culture that rewards teaching is promoted. Lone faculty, however promising their instructional transformations, are unlikely to be able to accomplish systemic change. 91 In this light, it is important to recognize that while there are many ongoing individual initiatives, systemic reforms, and research on their impact, are rare. More often students move through a patchwork of course experiences as they move through a degree program’s course requirements. Most of the research on the impact of transformations is limited to outcomes for students in a particular transformed course, while longitudinal studies (longer than one semester) are rare, despite the fact that this was one of the priorities noted in the now ten year old National Academies consensus report on Discipline Based Education Research.1 This lack of longitudinal research can perhaps be attributed to various causes, including the difficulty in conducting long-term studies – since students may disperse among many pathways and can be difficult to track, and the fact that funding for such studies usually does not extend over extended time spans.1 As a result, there is also almost no research on how students navigate among course type and course culture, either from the perspective of how students feel about such disconnects, or from the standpoint of what students gain (or lose) in terms of learning as they transition from course to course. In this paper we report how students fare as they move between a transformed organic chemistry course and a more Traditional course and vice versa over the course of two semesters. At our institution these two types of organic courses that are taught simultaneously. Some sections are taught in a Traditional manner, using a published textbook and instructional sequence20, while the transformed organic course uses the Organic Chemistry, Life, the Universe, and Everything (OCLUE) curriculum.12 Students enroll in these courses often before instructors have been assigned, and sign up for both courses in the two-semester sequence at the same time. This means that 1) they are not aware of which curriculum they will be using, and 2) they may unintentionally switch between curriculum types after the first semester. To minimize disruption, instructors have all agreed to cover the same material in the first semester so that students are prepared for the second semester. This arrangement results in a natural experiment; students are free to choose their own schedules, but they may have different 92 experiences depending on the sequence of courses they take. This enables us to examine how switching after the first semester impacts how students respond to two previously studied sets of tasks. We discuss the implications for students who are exposed to different expectations, instructional methods, and course cultures. Theoretical Framing: course culture For any course there is a set of norms that the instructors and students participate in as they collectively navigate the expectations and activities in the course. Such norms, which are often implicit rather than explicit, are sometimes referred to as the course culture, which may be thought of as “systems of meaning and practices that embody group norms and is reproduced.”21 Ways in which an instructor might communicate these “systems of meanings and practice” are the structure, content and expectations of the learning environments, the types of assessments, and ways in which students are expected to engage with the content of the course. For example, in the Traditional sections the class resources are a commercially published textbook20 and recommended end of chapter homework problems, instruction is via lecture, and the course grade is comprised only of student performance on several high stakes tests and a final exam. While recommended homework problems are assigned, it is not part of the course grade. Recitation sections are offered in which graduate student assistants go over the homework or give short tests. In effect, the Traditional course does not explicitly support student engagement with the material, and the high stakes tests may serve to focus students’ attention on those assessments that encourage rote learning. On the other hand in the OCLUE sections, which use an open education textbook12 and homework system beSocratic22, a significant proportion of the course grade (40%) is allocated to activities where students earn credit for “good faith effort” rather than correctness. Instruction is somewhat more interactive, clicker questions and short group activities intersperse the lectures, and in recitation sessions students work in groups to complete a set of more complex tasks. However, perhaps 93 the major difference between the two approaches is the expectations for what students should be able to do with their knowledge. Assessments in the Traditional sections typically include completing reactions, drawing mechanisms, devising syntheses, and identifying substances from spectroscopic data – that is, the Traditional kinds of organic chemistry assessments found in many courses.23 While OCLUE students are also expected to complete such tasks, they are also asked to construct explanations and reason about phenomena – that is, they are often asked to explain how and why something happens and to provide explicit reasoning.24 Interestingly, it appears that students are aware of the differences in expectations between the two courses. When asked, “what are you assessed on” “or how are you expected to think” the majority of OCLUE students indicated that they were assessed on ideas that were coded as “apply and reason”, such as “reasons why things happen a certain way “. In contrast, students from the Traditional section were more likely to indicate that they believed they were assessed on memorization, for example “The majority of the exams are memorization of the reactions.”25 It is almost certain that this outcome was not the intent of the instructors of the Traditional sections, but written responses of the students imply that they perceived the major goal of the course was to memorize a great deal of information. When the course operations and expectations are taken together with the student perceptions, it appears that the Traditional and OCLUE sections developed a different class culture, which we might expect to have impacts on the outcomes for each type of course. Theoretical Framing: Resource activation In addition to experiencing a different course culture, students in Traditional and OCLUE sections are expected to provide evidence of their knowledge in different ways. As noted earlier, students perceive that memorization is the major mode of assessment in the Traditional course, while OCLUE students understand that they need to use knowledge and reason with it. For example, while students in both courses are expected to be able to predict the outcome of an SN2 reaction, in OCLUE, 94 students are also asked to explain in words, why the reaction proceeds as it does. This requires students to explicitly state and connect ideas about what makes for a good nucleophile or leaving group and why the substrate is susceptible to nucleophilic attack. Certainly, the Traditional students are also taught these ideas, but the typical assessments do not require them to be explicitly stated. In our previous work we have used the resources perspective on learning to understand this process. We know that students are not blank slates, and that they “come to the classroom with preconceptions about how the world works.”4,26 This prior knowledge, as well as, the knowledge that students learn during instruction can be thought of as conceptual resources.27 These resources act as the building blocks that must be activated, engaged with, and selected for productive use in order to “[help] students ‘unravel’ and ‘reweave’ the strands of their knowledge and understanding” within their discipline.28 Using this perspective, the difference between expectations in OCLUE and Traditional are best seen in the expectations for what students should know and be able to do for each course – which is reflected in the types of assessment items used in each course. In OCLUE, students are continuously and explicitly expected to connect their conceptual resources to explain and predict phenomena both on homework tasks and exams. Such activities are specifically designed to activate appropriate resources,27 and provide opportunities for students to connect them together, ideally resulting in a more expert-like knowledge structure in which their conceptual resources are connected, contextualized and useful.26 While the Traditional students may also be using these resources, the assessments do not make this process visible. For example, a question in which students are asked to predict a product could be answered by memorization, whereas if they are asked to explain both how and why a particular product is formed, they must explicitly connect ideas about electrostatic interactions, stability, and reactivity. Consequently students who participate in courses where these explicit connections are not required seem more likely to perceive each reaction or each idea as separate – not necessarily connected.25 95 Building on our Prior Work We have previously reported on specific outcomes for students who take either the full two semester OCLUE sequence, or the full two semester Traditional sequence29 and we will be using data and coding schemes for the present study. Therefore, we briefly discuss the previous study (Chapter IV): how students draw mechanistic arrows for familiar and unfamiliar reactions30 Previous Work: Mechanistic Arrow Use In this study, we investigated how students draw mechanistic arrows to predict the products of both familiar and unfamiliar reactions.30,31 We revised a coding scheme from a previous report,31 to compare how matched cohorts of students from OCLUE and Traditional courses addressed each task. We defined a plausible mechanism as one where mechanistic arrows start at a source of high electron density and end at a location of low electron density, while a plausible product may be either the expected major product or minor products that make mechanistic sense for this reaction. Students were classified into four separate groups based on the following criteria: 1) A plausible product via a plausible mechanism and 2) A plausible product via a mixture of mechanistically reasonable steps and incorrect arrows, 3) A plausible product with incorrect arrows or no arrows, and 4) A chemically implausible product with either a mixture of mechanistically reasonable steps and incorrect arrows, or no arrows. An example of a student response for each code is shown in the Chapter IV Figure 4.5. This coding scheme was used to characterize responses to five different prompts, shown in Chapter IV Figure 4.2.30,31 In the previous chapter, we provide examples of student responses for the two reactions that are further explored in this paper; Prompt A: a familiar reaction (electrophilic addition of water to alkene) and what was, to our knowledge, Prompt E: an unfamiliar reaction (a ring closure at a carbonyl) for which students should have been able to predict a product by using mechanistic arrow pushing. The responses for Prompt B-D will also be discussed as they are also recognizable reactions for students. Responses from students who had OCLUE for both semesters of OChem (OCLUE-OCLUE) and 96 those who had a Traditional course for both semesters of OChem (Traditional-Traditional) were characterized. Our analysis showed that by the end of OChem 2, the majority of students (75%) from both cohorts were able to predict the product for the familiar reaction. However, there was a large and significant difference between the numbers of students who used appropriate mechanistic arrows to reach the plausible product. 64% of students in the OCLUE-OCLUE cohort drew plausible arrows and a plausible product while 21% of the Traditional-Traditional students drew plausible arrows and a plausible product.30 When students were asked to draw a mechanism to predict the product for an unfamiliar reaction, the difference between the two cohorts became even more apparent. 45% of the OCLUE- OCLUE cohort drew a plausible product with 42% using at least some appropriate mechanistic arrows, whereas only 8% of Traditional-Traditional students drew a plausible product with 4% drawing some appropriate mechanistic arrows.30 Research Question guiding this work In the prior, we showed that OCLUE-OCLUE students were more likely to draw mechanisms to predict products compared to Traditional-Traditional students at the end of OChem 2. The question here is, what happens to students who switch sections? Here we report on students who switched course types, to compare outcomes to the previous findings for drawing mechanistic arrows. The study is guided by the overarching research questions: 1. What is the impact of switching between a transformed organic chemistry course and a Traditional organic chemistry course as measured by a students’ use of mechanistic arrows to predict familiar products? 2. What is the impact of switching between a transformed organic chemistry course and a Traditional organic chemistry course as measured by a students’ use of mechanistic arrows to predict an unfamiliar product? 97 Methods Student Participants In this report, we will present findings from 433 students divided into 4 cohorts based on their OChem course experience pathways. Students who were enrolled in OCLUE for two consecutive semesters will be referred to as OCLUE-OCLUE (n = 102) while, students who were enrolled in a Traditional OChem for both semesters will be referred to as Traditional-Traditional (n = 125). These two- part names are meant to indicate the first and second semester course experience for that group. As noted earlier, because of scheduling restraints and the fact that students enroll months before instructor assignments are listed, some students did not take the same organic course type for their first and second semester (i.e. they may take OCLUE for OChem 1 and then take Traditional OChem 2 or vice versa). This led to an OCLUE-Traditional cohort (n = 195) and a Traditional-OCLUE cohort (n = 16). The OCLUE-Traditional cohort is larger than the others because there were more students who took this course pathway. In the 2017-2018 academic year, there were two sections of OCLUE offered for OChem 1 and only one section of Traditional OChem 1. Then in the spring, there was only one section of OCLUE offered compared to two sections of Traditional for OChem 2. This means there were many more opportunities for students to be enrolled in the OCLUE-Traditional course pathway resulting in the larger cohort (n = 190) and less opportunities for students to be enrolled in the Traditional-OCLUE pathway resulting in a smaller cohort (n = 16). As a result, we will not report statistical analyses of data from the Traditional-OCLUE cohort but we will include data for these few students to show the trends for descriptive purposes only. Figure 5.1 shows a summary of the administrations of the prompts. 98 Figure 5.1. Summary of the data collection. All 433 students responded to all the prompts at each of the time points discussed in Data Collection section. We note that the student sample is slightly smaller than the previously published cohorts (Chapter IV), because we are limiting the analysis to students who took all of the prompts, as well as, another published works prompt for future comparisons24. We further refined the sample by including only those students for whom we could record complete general chemistry 1 and 2 course grades, organic chemistry 1 and 2 course grades, an ACT score, and their GPA prior to starting organic chemistry. Mann-Whitney U tests were used to compare each of these cohorts on course grades, GPA prior to organic chemistry, and their ACT score. These analyses were run in SPSS and results are provided in the Appendix B. No differences were found when comparing general chemistry grades, GPA prior to starting organic chemistry, and organic chemistry 1 grades, however, differences were observed in organic chemistry 2 grades and ACT scores. Students enrolled in the Traditional course for OChem 2 had higher OChem 2 grades when compared to OCLUE students (small effect size).32 OCLUE-Traditional students had higher ACT scores compared to Traditional-Traditional students (small effect size).32 There is no apparent grade penalty for students who switch sections. 99 Data Collection This study was administered using the beSocratic system,22 an online formative assessment tool that allows students to write and draw responses. All data is recorded and can be replayed and coded at a later date. For OCLUE students, these activities were part of their homework assignments. OCLUE students complete approximately 20-22 required homework assignments throughout the semester. OCLUE homework assignments are not graded for accuracy and count for 15% of their course grade with this particular data collection, which was extra credit, counting for <1% of their total course grade. For Traditional students, the assignments were also offered as extra credit (approximately 2% of their course grade). In the following subsection, we will describe each of the tasks in full detail. Mechanistic Arrow Use Data Analysis For this study, students were asked to “Predict the product of the following reaction by drawing a mechanism.” Students were provided with structures of the starting materials and reactants and a space to draw mechanistic arrows, intermediates, and products. Screen shots of the prompts are provided in Chapter IV Figure 4.2 30 By timing the data collections at the start and end of OChem 2, we hoped to capture both the impact of their learning in OChem 1, which we believe would be representative of students’ OChem 1 course experience and then the impact of their OChem 2 course at the end of the semester. Responses from all time points were coded anonymously by author (S.K.H) and a trained postbaccalaureate coder such that the coders were not aware of the students’ background when reviewing a response. Both coders evaluated 60 random responses from each prompt to obtain Cohen’s kappa values that ranged from 0.78-1.0 which indicates substantial agreement Which are found in Chapter IV Appendix C.30,32 Reducing Potential Bias We recognize the bias that are inherent when researchers investigate the curricular impacts of an intervention that they themselves designed. In an effort to reduce the impact of these biases, 100 responses were collected and analyzed in such a way that they were deidentified from their course type and coded anonymously. This was done so coders were not aware of students’ course type or identity while coding responses. Results Research Question 1: What is the impact of switching between a transformed organic chemistry course and a Traditional organic chemistry course as measured by a students’ use of mechanistic arrows to predict familiar products? Prompt A: Familiar reaction electrophilic addition of water to alkene As discussed earlier, our original work found that similar percentages of OCLUE-OCLUE and Traditional-Traditional students can produce a plausible product for a familiar reaction.30 However, we observed that students enrolled in the OCLUE-OCLUE sequence were significantly more likely to draw all mechanistic steps correctly (with a Sidak adjusted alpha level33 of .017 per test (.001/3) and a medium to large effect size of Cramer’s V = 0. 335,) than those in the Traditional-Traditional course sequence. To avoid the likelihood of making any Type 1 errors in determining significance with multiple Chi-square tests, we used the more conservative Sidak adjusted alpha value of 0.017 instead of the usual 0.05.33 Cramer’s V is used to interpret the magnitude of a significant finding, it is interpreted as: small effect size between 0.1 and 0.3, medium effect size between 0.3 and 0.5, and large effect size greater than 0.5. We also have excluded the Traditional-OCLUE cohort from any statistical analyses because their number is too small (n=16). The data reported in this manuscript are a subset of the large population of students reported previously, that is, only those who took all the prompts at each time point are included. When we include the responses from students who switched sections after one semester, an interesting pattern appeared, as shown in Figure 5.2 and Table 5.1. As previously discussed, the number of students who moved from Traditional to OCLUE is not large enough to provide statistical information, but they are included here for completeness. 101 At the start of the second semester, students’ responses appear to reflect their previous experience in OChem 1: students who have switched from OCLUE to Traditional have similar performances to their OCLUE peers, while the performances of students who switched from Traditional to OCLUE look like their Traditional peers. However, by the end of the second semester we see that the situation has changed. While all cohorts are still able to draw the product for acid catalyzed hydration, the percentage of students in the OCLUE-Traditional cohort who draw mechanisms has decreased significantly as compared to the OCLUE- OCLUE cohort. That is, a smaller percent of students who transferred from OCLUE to Traditional use curved arrows to predict the product, than those who remained in OCLUE. Conversely, there are a greater percentage of students who switched from Traditional to OCLUE who draw mechanistic arrows than those who stay in the Traditional curriculum for both semesters. By the end of OChem 2, the patterns of mechanism use for students who switch tend to resemble the main cohort into which they switched. As shown in Table 1: While the performances of the OCLUE-OCLUE, and OCLUE-Traditional cohorts are similar at the beginning of OChem 2, by the end there is now a significant difference with a large effect size32 between the two cohorts. Similarly, at the beginning of OChem 2 the performances of the OCLUE-Traditional and Traditional-Traditional cohorts are significantly different with a large effect size, and while they are still statistically different at the end of OChem 2, the effect size is now small.32 102 Figure 5.2. Percent of arrow use, and product drawn for a familiar reaction Prompt A for each cohort. The Traditional-OCLUE cohort is small (n = 16) and marked with an asterisk. 103 Table 5.1. Differences in students use of arrows for a Familiar Reaction between course type.a Time Point Cohorts χ2 (df = 1) p-value Cramer’s V Start OChem 2 OCLUE-OCLUE 25.426 < 0.001a 0.335 (medium) Traditional-Traditional Start OChem 2 OCLUE-OCLUE 0.038 0.845 0.011 OCLUE-Traditional Start OChem 2 Traditional-Traditional 28.774 < 0.001a 0.302 (medium) OCLUE-Traditional End OChem 2 OCLUE-OCLUE 51.769 < 0.001a 0.475 (medium to large) Traditional-Traditional End OChem 2 OCLUE-OCLUE 26.331 < 0.001a 0.300 (medium) OCLUE-Traditional End OChem 2 Traditional-Traditional 8.968 0.003 0.168 (small to medium) OCLUE-Traditional aFor all Chi-square analysis, a Sidak adjusted alpha of α = 0.017 was used.33 Interpretation of Cramer’s V is as follows: small effect size between 0.1 and 0.3, medium effect size between 0.3 and 0.5, and large effect size greater than 0.5.32 Prompt B- D: Other familiar reactions Prompts B, C, and D, though familiar reactions (explicitly covered in class), are not encountered as often as the electrophilic addition. These reactions also offered insight into how students are using their arrows across multiple different reactions. At the beginning of OChem 1 most students experience difficulties with these three reactions, and very few students were drawing a plausible product, let alone drawing plausible arrows throughout their mechanisms. However, by the end of OChem 2 students who had OCLUE in the second semester were more likely to draw a plausible mechanism and product for these three reactions. Tables showing the chi-square values for these reactions can be found in Appendix C, in all cases, students who had OCLUE for OChem 2 were able to draw plausible arrows and a plausible product more frequently than students who had Traditional for OChem 2. This is also consistent with the pattern observed for prompt A the hydration of an alkene, students as the end of OChem 2 resemble the course they switched in to. 104 Figure 5.3. Percent of arrow use, and product drawn for a familiar reaction Prompt B for each cohort. The Traditional-OCLUE cohort is small (n = 16) and marked with an asterisk. Figure 5.4. Percent of arrow use, and product drawn for a familiar reaction Prompt C for each cohort. The Traditional-OCLUE cohort is small (n = 16) and marked with an asterisk. 105 Figure 5.5. Percent of arrow use, and product drawn for a familiar reaction Prompt D for each cohort. The Traditional-OCLUE cohort is small (n = 16) and marked with an asterisk. Research Question 2: What is the impact of switching between a transformed organic chemistry course and a Traditional organic chemistry course as measured by a students’ use of mechanistic arrows to predict an unfamiliar product? For the second part of this study students were asked to use mechanistic arrows to predict a plausible product for a reaction that, to our knowledge, students had not seen before. That being said, students should be able to use the principles of mechanistic arrow pushing and an understanding of how carbonyl functional groups behave to predict plausible products for this reaction. In the original study using this prompt,31 students struggled to complete this task, with only about 8% of the students successfully drawing a plausible product.31 For this study students were asked to complete this task at the end of OChem 2 and the responses were coded in a similar fashion to the familiar reactions. While students find this task much more difficult, it is now clear that the use of arrow pushing does help students produce a plausible product. We see that 45% of OCLUE-OCLUE and 44% of Traditional-OCLUE students drew a plausible 106 product, the majority of whom drew at least some plausible mechanistic arrows. However only 16% of OCLUE-Traditional and 10% of Traditional-Traditional proposed a plausible product as shown in Figure 6 and Table 2. Figure 5.6. Percent of arrow use, and product drawn for an unfamiliar reaction for each cohort. The Traditional-OCLUE cohort is small (n = 16) and marked with an asterisk. Table 5.2. Differences in students use of arrows for a unfamiliar Reaction between course type.a Time Point Cohorts χ2 (df = 1) p-value Cramer’s V End OChem 2 OCLUE-OCLUE 28.374a < 0.001a 0.354 (medium to large) Traditional-Traditional End OChem 2 OCLUE-OCLUE 25.077a < 0.001a 0.293 (medium) OCLUE-Traditional End OChem 2 Traditional-Traditional 2.040a 0.080 0.153 (small to medium) OCLUE-Traditional aFor all Chi-square analysis, a Sidak adjusted alpha of α = 0.017 was used.33 Interpretation of Cramer’s V is as follows: small effect size between 0.1 and 0.3, medium effect size between 0.3 and 0.5, and large effect size greater than 0.5.32 Again, the same pattern emerges, in which the switchers’ responses are closer to the responses typical of the course in which they are enrolled. 107 Discussion The goal of this study to better understand what happens as students move between different courses, ostensibly covering the same material, but using different teaching methods, with different classroom cultures. In this study, we find students who switch sections after one semester of a two semester course, tend to perform similarly to the course into which they switch. Most problematically, some students appear to “regress” in their ability to use mechanistic arrows or to provide a mechanistic explanation, if they switch from a transformed to a Traditional course section. One might imagine that once students have learned a particular skill or, what Krist et. al. call, an epistemic heuristic (a general thinking strategy) students would subsequently use that skill to guide their reasoning about phenomena.34 We might expect that once learned, the ability to correctly draw mechanistic arrows or to construct causal mechanistic explanations, would persist through the next course – particularly if it builds on the prior material. Surprisingly we find that switching from OCLUE to Traditional after the first semester, many of the original OCLUE students have relinquished the use of mechanisms, so that they appear more like Traditional-Traditional students. This change becomes even more apparent in the results from the unfamiliar task, where the OCLUE-Traditional cohort appears to find the task just as difficult as the Traditional-Traditional cohort. On the other hand, while the numbers of students are smaller, it appears that some students can acquire these ways of thinking upon transferring into a course with a different emphasis. It is worth pointing that there appears to be no overall grade penalty for switching, despite the fact that each course tends to emphasize and value different aspects of organic chemistry found in Appendix B. These findings suggest that students are able to adapt quickly to course culture and assessment strategies: the explicit and more subtle implicit messages about what is important in the course are quickly learned and acted upon. However, this positive finding is offset by our data showing that some students who transfer from the transformed courses do not retain, or at least no longer use, what they 108 learned. There are a number of possible reasons for this. The expectations and course culture in the two different iterations of organic chemistry are manifestly different, and different types of thinking and problem solving are valued and rewarded in each course. In OCLUE, students routinely (at least twice a week) complete formative tasks for homework on beSocratic, that require them to engage in mechanistic reasoning at the same time as drawing electron pushing mechanisms. Whereas in the Traditional course, students are certainly asked to draw mechanisms, but it is not coupled with such reasoning tasks. This is also apparent in Traditional students reported perceptions of the course; very few of whom believe that they are expected to apply and reason in their organic course while the majority of students believe they are expected to memorize the material.25 Implications Transformed courses have been shown to improve important student outcomes ranging from skills such as drawing Lewis structures35 and mechanistic arrows,30,31,36 to more complex outcomes such as constructing explanations, models and arguments and engaging in mechanistic reasoning.24,29,37 The finding from this study that, for whatever reasons, some students tend not to retain, or at least use, their abilities if they transfer to a more Traditional course, means that the “one shot” or “lone instructor” approach to transformation can never achieve sustainable change. If we agree that transformations such as the one described in this paper, can provide improvements in student learning of ideas and practices we value, then we have to search for better ways to sustain and extend transformations. While we have provided evidence that even one course allows students to build useful knowledge, we also have evidence that these increases are fragile and easily lost. Because of the paucity of longitudinal studies, we do not know what “dose” of transformation would lead to lasting improvements in student learning. However, in a previous study, we did find that students who had taken a two-semester transformed general chemistry course (CLUE),10 provided more sophisticated 109 explanations for simple acid base reactions at several points throughout a Traditional two semester organic course,29 than students who had taken more Traditional general chemistry courses. That is, a two semester general chemistry course that emphasized mechanistic reasoning appeared to have some lasting effect. That being said, we believe that it is in students’ interest that they see a coherent sequenced approach to their science courses in which scientific practices such as constructing models, explanations and arguments are used to help students connect and use their resources and construct more expert-like knowledge structures. Future Directions It is our contention that there is a strong and positive link between constructing mechanistic explanations and drawing mechanisms. In our work where we analyze student mechanistic drawing we see that students who are also expected to engage in constructing mechanistic explanations make more use of mechanistic arrows, and are better able to use them to predict the products of unfamiliar reactions30. Indeed, students at the end of OCLUE, tend to use more mechanistic arrows and are far more successful in predicting the outcome of an unfamiliar reaction. We propose that constructing mechanistic explanations can be a mechanism that helps students make connections between appropriate cognitive resources. An emphasis on reasoning throughout a course, coupled with opportunities to practice on ungraded formative (low stakes) assessments, makes these resources accessible, strengthens the connections between them, and leads to a culture where a willingness to “try out” ideas without penalty. We also recognize that students are rational actors, and in a class where students perceive that memorization is an appropriate strategy for learning material,25 then the ideas learned earlier (about how and why reactions occur) may seem less valuable and are not used. Whatever the reasons, our findings here have significant ramifications for the development of sustainable transformed sequences of courses. 110 Limitations The limitations of this study are that it was conducted in a single university, and although there are quite large numbers of students in each cohort it is not replicated (because of limitations due to scheduling and the COVID pandemic). The two cohorts took all the assignments as extra credit activities on the beSocratic platform, which was used routinely for OCLUE but not for Traditional sections. However, most Traditional students took general chemistry using the same platform and therefore were familiar with it. Students in Traditional sections received greater credit for completing the task with “good faith effort” than did those in OCLUE. 111 REFERENCES (1) Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering; Singer, S. R., Nielsen, N., Schweingruber, H. A., Eds.; The National Academies Press: Washington, D.C, 2012. (2) PCAST, President’s Council of Advisors on Science and Technology. Engage To Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and Mathematics.; Presidential Report; President’s Council of Advisors on Science and Technology, 2010. (3) Helping Students Make Sense of the World Using Next Generation Science and Engineering Practices; National Science Teachers Association, 2017. https://doi.org/10.2505/9781938946042. (4) National Academies of Sciences, Engineering, and Medicine. How People Learn II: Learners, Contexts, and Cultures; National Academies Press: Washington, D.C., 2018; p 24783. https://doi.org/10.17226/24783. (5) Seymour, E.; Hunter, A.-B. Talking about Leaving Revisited: Persistence, Relocation, and Loss in Undergraduate STEM Education; Springer International Publishing: New York, 2019. (6) Carle, M. S.; Visser, R.; Flynn, A. B. Evaluating Students’ Learning Gains, Strategies, and Errors Using OrgChem101’s Module: Organic Mechanisms—Mastering the Arrows. Chem. Educ. Res. Pract. 2020, 21 (2), 582–596. https://doi.org/10.1039/C9RP00274J. (7) Walker, L.; Warfa, A.-R. M. Process Oriented Guided Inquiry Learning (POGIL®) Marginally Effects Student Achievement Measures but Substantially Increases the Odds of Passing a Course. PLoS ONE 2017, 12 (10), e0186203. https://doi.org/10.1371/journal.pone.0186203. (8) van de Pol, J.; Volman, M.; Beishuizen, J. Scaffolding in Teacher–Student Interaction: A Decade of Research. Educ. Psychol. Rev. 2010, 22 (3), 271–296. https://doi.org/10.1007/s10648-010- 9127-6. (9) Freeman, S.; Eddy, S. L.; McDonough, M.; Smith, M. K.; Okoroafor, N.; Jordt, H.; Wenderoth, M. P. Active Learning Increases Student Performance in Science, Engineering, and Mathematics. Proc. Natl. Acad. Sci. U.S.A. 2014, 111 (23), 8410–8415. https://doi.org/10.1073/pnas.1319030111. (10) Cooper, M.; Klymkowsky, M. Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform. J. Chem. Educ. 2013, 90 (9), 1116–1122. https://doi.org/10.1021/ed300456y. (11) Sevian, H.; Talanquer, V. Rethinking Chemistry: A Learning Progression on Chemical Thinking. Chem. Educ. Res. Pract. 2014, 15 (1), 10–23. https://doi.org/10.1039/C3RP00111C. (12) Cooper, M. M.; Klymkowsky, M. K. Organic Chemistry, Life, the Universe & Everything. 193. 112 (13) Flynn, A. B.; Ogilvie, W. W. Mechanisms before Reactions: A Mechanistic Approach to the Organic Chemistry Curriculum Based on Patterns of Electron Flow. J. Chem. Educ. 2015, 92 (5), 803–810. https://doi.org/10.1021/ed500284d. (14) Bain, K.; Bender, L.; Bergeron, P.; Caballero, M. D.; Carmel, J. H.; Duffy, E. M.; Ebert-May, D.; Fata-Hartley, C. L.; Herrington, D. G.; Laverty, J. T.; Matz, R. L.; Nelson, P. C.; Posey, L. A.; Stoltzfus, J. R.; Stowe, R. L.; Sweeder, R. D.; Tessmer, S. H.; Underwood, S. M.; Urban-Lurain, M.; Cooper, M. M. Characterizing College Science Instruction: The Three-Dimensional Learning Observation Protocol. PLoS ONE 2020, 15 (6), e0234640. https://doi.org/10.1371/journal.pone.0234640. (15) Stains, M.; Harshman, J.; Barker, M. K.; Chasteen, S. V.; Cole, R.; DeChenne-Peters, S. E.; Eagan, M. K.; Esson, J. M.; Knight, J. K.; Laski, F. A.; Levis-Fitzgerald, M.; Lee, C. J.; Lo, S. M.; McDonnell, L. M.; McKay, T. A.; Michelotti, N.; Musgrove, A.; Palmer, M. S.; Plank, K. M.; Rodela, T. M.; Sanders, E. R.; Schimpf, N. G.; Schulte, P. M.; Smith, M. K.; Stetzer, M.; Van Valkenburgh, B.; Vinson, E.; Weir, L. K.; Wendel, P. J.; Wheeler, L. B.; Young, A. M. Anatomy of STEM Teaching in North American Universities. Science 2018, 359 (6383), 1468–1470. https://doi.org/10.1126/science.aap8892. (16) Stowe, R. L.; Cooper, M. M. Practicing What We Preach: Assessing “Critical Thinking” in Organic Chemistry. J. Chem. Educ. 2017, 94 (12), 1852–1859. https://doi.org/10.1021/acs.jchemed.7b00335. (17) Henderson, C.; Beach, A.; Finkelstein, N. Facilitating Change in Undergraduate STEM Instructional Practices: An Analytic Review of the Literature. J. Res. Sci. Teach. 2011, 48 (8), 952– 984. https://doi.org/10.1002/tea.20439. (18) Sturtevant, H.; Wheeler, L. The STEM Faculty Instructional Barriers and Identity Survey (FIBIS): Development and Exploratory Results. Int. J. STEM Educ. 2019, 6 (1), 35. https://doi.org/10.1186/s40594-019-0185-0. (19) Reinholz, D. L.; Apkarian, N. Four Frames for Systemic Change in STEM Departments. Int. J. STEM Educ. 2018, 5 (1), 3. https://doi.org/10.1186/s40594-018-0103-x. (20) Wade, L. G.; Simek, J. W. Organic Chemistry, 9th ed.; Pearson: New York City, New York, 2017. (21) Nasir, N. S.; Hand, V. M. Exploring Sociocultural Perspectives on Race, Culture, and Learning. Rev. Educ. Res. 2006, 76 (4), 449–475. https://doi.org/10.3102/00346543076004449. (22) Bryfczynski, S. P. BeSocratic: An Intelligent Tutoring System for the Recognition, Evaluation, and Analysis of Free-Form Student Input. Ph.D. Dissertation, Clemson University, SC, 2010. (23) Bowen, R. S. Student Perceptions of “Critical Thinking”: Insights into Clarifying an Amorphous Construct. Chem. Educ. Res. Pract. 2022. https://doi.org/10.1039/D2RP00097K. (24) Crandell, O. M.; Lockhart, M. A.; Cooper, M. M. Arrows on the Page Are Not a Good Gauge: Evidence for the Importance of Causal Mechanistic Explanations about Nucleophilic Substitution in Organic Chemistry. J. Chem. Educ. 2020, 97 (2), 313–327. https://doi.org/10.1021/acs.jchemed.9b00815. 113 (25) Bowen, R. S.; Flaherty, A. A.; Cooper, M. M. Investigating Student Perceptions of Transformational Intent and Classroom Culture in Organic Chemistry Courses. Chem. Educ. Res. Pract. 2022, 10.1039.D2RP00010E. https://doi.org/10.1039/D2RP00010E. (26) National Academies of Sciences, Engineering, and Medicine. How People Learn: Brain, Mind, Experience, and School: Expanded Edition; National Academies Press: Washington, D.C., 2000; p 9853. https://doi.org/10.17226/9853. (27) Hammer, D.; Elby, A. Tapping Epistemological Resources for Learning Physics. J. Learn. Sci. 2003, 12 (1), 53–90. https://doi.org/10.1207/S15327809JLS1201_3. (28) Hammer, D. Student Resources for Learning Introductory Physics. Am. J. Phys. 2000, 68 (S1), S52–S59. https://doi.org/10.1119/1.19520. (29) Crandell, O. M.; Kouyoumdjian, H.; Underwood, S. M.; Cooper, M. M. Reasoning about Reactions in Organic Chemistry: Starting It in General Chemistry. J. Chem. Educ. 2019, 96 (2), 213–226. https://doi.org/10.1021/acs.jchemed.8b00784. (30) Houchlei, S. K.; Bloch, R. R.; Cooper, M. M. Mechanisms, Models, and Explanations: Analyzing the Mechanistic Paths Students Take to Reach a Product for Familiar and Unfamiliar Organic Reactions. J. Chem. Educ. 2021, 98 (9), 2751–2764. https://doi.org/10.1021/acs.jchemed.1c00099. (31) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844–849. https://doi.org/10.1021/ed2003934. (32) Cohen, J. A Coefficient of Agreement for Nominal Scales. Educ. Psychol. Meas. 1960, 20 (1), 37– 46. https://doi.org/10.1177/001316446002000104. (33) Shaffer, J. Multiple Hypothesis Testing. Annu. Rev. Psychol. 1995, No. 46, 561–584. (34) Krist, C.; Schwarz, C. V.; Reiser, B. J. Identifying Essential Epistemic Heuristics for Guiding Mechanistic Reasoning in Science Learning. J. Learn. Sci. 2019, 28 (2), 160–205. https://doi.org/10.1080/10508406.2018.1510404. (35) Cooper, M. M.; Underwood, S. M.; Hilley, C. Z.; Klymkowsky, M. W. Development and Assessment of a Molecular Structure and Properties Learning Progression. J. Chem. Educ. 2012, 89 (11), 1351–1357. https://doi.org/10.1021/ed300083a. (36) Grove, N. P.; Cooper, M. M.; Cox, E. L. Does Mechanistic Thinking Improve Student Success in Organic Chemistry? J. Chem. Educ. 2012, 89 (7), 850–853. https://doi.org/10.1021/ed200394d. (37) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid–Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703–1712. https://doi.org/10.1021/acs.jchemed.6b00417. (38) IBM Corp. SPSS Statistics for Windows; IBM Crop.: Armonk, NY, USA, 2017. 114 (39) Green, S.; Salkind, N. Using SPSS for Windows and Macintosh: Analyzing and Understanding Data; Pearson Education Inc: Boston, MA, USA, 2010. 115 APPENDIX A: STUDIES PARTICIPANT DEMOGRAPHICS Table 5.3. Mann – Whitney Comparison of student participants’ demographics.32,38,39 Measure Cohort N Mean Median Mann-Whitney U z p-value Effect Size OCLUE-OCLUE 102 27.31 27 ACT 5861.00 -1.049 0.294 Traditional-Traditional 125 26.78 27 Gen Chem 1 OCLUE-OCLUE 102 3.57 4.0 5686.00 -1.504 0.133 Course Grade Traditional-Traditional 125 3.48 3.5 Gen Chem 2 OCLUE-OCLUE 102 3.51 3.5 5812.50 -1.201 0.230 Course Grade Traditional-Traditional 125 3.37 3.5 GPA Prior OCLUE-OCLUE 102 3.65 3.78 6042.00 -0.678 0.498 to OChem 1 Traditional-Traditional 125 3.64 3.69 OChem 1 OCLUE-OCLUE 102 3.77 4.0 5742.50 -1.583 0.113 Course Grade Traditional-Traditional 125 3.68 4.0 OChem 2 OCLUE-OCLUE 102 3.25 3.5 5375.50 -2.161 0.031 0.143 small Course Grade Traditional-Traditional 125 3.45 4.0 Measure Cohort N Mean Median Mann-Whitney U z p-value Effect Size OCLUE-OCLUE 102 27.31 27 ACT 9326.00 -0.531 0.595 OCLUE-Traditional 190 27.51 28 Gen Chem 1 OCLUE-OCLUE 102 3.57 4.0 8953.50 -1.155 0.248 Course Grade OCLUE-Traditional 190 3.48 3.5 Gen Chem 2 OCLUE-OCLUE 102 3.51 3.5 9546.00 -0.222 0.825 Course Grade OCLUE-Traditional 190 3.41 3.5 GPA Prior OCLUE-OCLUE 102 3.65 3.78 9452.00 -0.347 0.729 to OChem 1 OCLUE-Traditional 190 3.64 3.75 OChem 1 OCLUE-OCLUE 102 3.77 4.0 9132.00 -1.015 0.310 Course Grade OCLUE-Traditional 190 3.71 4.0 OChem 2 OCLUE-OCLUE 102 3.25 3.5 8366.00 -2.059 0.039 0.120 small Course Grade OCLUE-Traditional 190 3.34 4.0 Measure Cohort N Mean Median Mann-Whitney U z p-value Effect Size OCLUE-Traditional 190 27.51 28 ACT 10308.00 -1.991 0.047 0.112 small Traditional-Traditional 125 26.78 27 Gen Chem 1 OCLUE-Traditional 190 3.48 3.5 11494.50 -0.512 0.609 Course Grade Traditional-Traditional 125 3.48 3.5 Gen Chem 2 OCLUE-Traditional 190 3.41 3.5 11115.50 -1.012 0.312 Course Grade Traditional-Traditional 125 3.37 3.5 GPA Prior OCLUE-Traditional 190 3.64 3.75 11583.00 -0.37 0.711 to OChem 1 Traditional-Traditional 125 3.64 3.69 OChem 1 OCLUE-Traditional 190 3.71 4.0 11386.50 -0.746 0.456 Course Grade Traditional-Traditional 125 3.68 4.0 OChem 2 OCLUE-Traditional 190 3.34 4.0 11659.50 -0.301 0.764 Course Grade Traditional-Traditional 125 3.45 4.0 Table 5.4. Description of gender and intended major of student participants. Values in the table are the absolute counts for each group of students. Preprofessional and Human Biological Cohort N Male Female Chemistry Other Health Sciences Biology Sciences OCLUE-OCLUE 102 31 71 9 34 43 0 16 OCLUE-Traditional 190 65 125 41 52 74 1 22 Traditional-OCLUE 16 7 9 1 4 6 1 4 Traditional-Traditional 125 35 90 16 38 46 3 22 116 APPENDIX B: CHANGE OVER TIME Figure 5.7. Familiar Reaction OCLUE-OCLUE Change Over Time. Figure 5.8. Familiar Reaction OCLUE-Traditional Change Over Time. 117 Figure 5.9. Familiar Reaction Traditional-Traditional Change Over Time. Table 5.5. Familiar Reaction Change in Ranks for Students from Time Point One to Time Point Two. OCLUE-OCLUE OCLUE-Traditional Traditional -Traditional Ranks (count/total n) (count/total n) (count/total n) Positive 39% (40/102) 33% (62/190) 37% (46/125) Tie 52% (53/102) 38% (72/190) 34% (43/125) Negative 9% (9/102) 29% (56/190) 29% (36/125) Positive Ranks are students who increase in their rank from the start of OChem 1 to OChem 2, Tie Ranks are students who do not change their rank from the start of OChem 1 to OChem 2, and Negative Ranks are students who decrease their rank from the start of OChem 1 to OChem 2. 118 APPENDIX C: COMPARISON OF OTHER FAMILIAR PROMPTS OVER TIME Table 5.6. Differences in students use of arrows for Prompt B: Electrophilic Addition of Water to Alkyne between course type.a Time Point Cohorts χ2 (df = 1) p-value Cramer’s V OCLUE-OCLUE Start OChem 2 7.548 0.006 0.182 (small to medium) Traditional-Traditional OCLUE-OCLUE Start OChem 2 6.294 0.012 0.147 (small) OCLUE-Traditional Traditional-Traditional Start OChem 2 .361 0.548 0.034 OCLUE-Traditional OCLUE-OCLUE a End OChem 2 25.011 < 0.001 0.332 (medium) Traditional-Traditional OCLUE-OCLUE a End OChem 2 21.546 < 0.001 0.272 (medium) OCLUE-Traditional Traditional-Traditional End OChem 2 1.186 0.276 0.061 OCLUE-Traditional a For all Chi-square analysis, a Sidak adjusted alpha of α = 0.017 was used. Interpretation of Cramer’s V is as follows: small effect size between 0.1 and 0.3, medium effect size between 0.3 and 0.5, and large effect size greater than 0.5. Table 5.7. Differences in students use of arrows for Prompt C: Alkyne Deprotonation Followed by SN2 between course type.a 2 Time Point Cohorts χ (df = 1) p-value Cramer’s V OCLUE-OCLUE Start OChem 2 4.086 0.043 0.134 (small to medium) Traditional-Traditional OCLUE-OCLUE Start OChem 2 .579 0.447 0.045 OCLUE-Traditional Traditional-Traditional Start OChem 2 2.201 0.138 0.084 OCLUE-Traditional OCLUE-OCLUE There are too few students who drew plausible product End OChem 2 Traditional-Traditional and plausible arrows to perform a statistical comparison OCLUE-OCLUE End OChem 2 7.431 0.006 0.160 (small to medium) OCLUE-Traditional Traditional-Traditional There are too few students who drew plausible product End OChem 2 OCLUE-Traditional and plausible arrows to perform a statistical comparison aFor all Chi-square analysis, a Sidak adjusted alpha of α = 0.017 was used. Interpretation of Cramer’s V is as follows: small effect size between 0.1 and 0.3, medium effect size between 0.3 and 0.5, and large effect size greater than 0.5. 119 Table 5.8. Differences in students use of arrows for Prompt D: Nucleophilic Attack at a Carbonyl between course type.a Time Point Cohorts χ2 (df = 1) p-value Cramer’s V OCLUE-OCLUE Start OChem 2 12.133 < 0.001a 0.231 (small to medium) Traditional-Traditional OCLUE-OCLUE Start OChem 2 6.949 0.008 0.154 (small to medium) OCLUE-Traditional Traditional-Traditional Start OChem 2 1.959 0.162 0.079 OCLUE-Traditional OCLUE-OCLUE End OChem 2 25.424 0.000 0.335 (medium to large) Traditional-Traditional OCLUE-OCLUE End OChem 2 27.485 0.000 0.307 (medium) OCLUE-Traditional Traditional-Traditional End OChem 2 .148 0.701 0.022 OCLUE-Traditional aFor all Chi-square analysis, a Sidak adjusted alpha of α = 0.017 was used. Interpretation of Cramer’s V is as follows: small effect size between 0.1 and 0.3, medium effect size between 0.3 and 0.5, and large effect size greater than 0.5. 120 CHAPTER VI –WHAT IS THE RELATIONSHIP BETWEEN STUDENTS’ CAUSAL MECHANISTIC EXPLANATIONS AND STUDENTS USE OF MECHANISTIC ARROWS? Preface This study builds upon the work of Olivia M. Crandell 1,in addition to that reported in Chapter IV and V. All of these studies are based on the same groups of students, which gives us the opportunity to investigate whether there is any relationship between students constructing causal mechanistic explanations and their ability to draw plausible mechanistic arrows. Introduction There have been numerous studies that have described the difficulties students have with various topics across organic chemistry. Many have been discussed in Chapter III, ranging from students difficulties with mechanistic arrows2–9, spectroscopy10–13, and constructing energy diagrams.14–18 To better understand what students know and are able to do in organic chemistry we believe you must elicit their reasoning about a given phenomenon. A Framework for K-12 Science Education stated that “the goal of science is to construct explanations for the causes of phenomena.”19 Constructing explanations about organic chemistry reactions is one way students can provide evidence of their understanding and can show instructors how they connect their knowledge in different contexts. By giving students opportunities to explain how and why, we can gain insight into the different resources they are calling upon to build their explanations.1,20 This emphasis on the how and why of a phenomenon is the basis for causal mechanistic reasoning, which supports students reasoning about the causes and consequences of the phenomenon. Russ et al. define causal mechanistic reasoning (CMR) as “how the particular components of a system give rise to the behavior of a target phenomenon.21 The components of CMR are characterized by 1) identifying the entities and 2) identifying the activities of those entities.21 121 In previous work by the Cooper group causal mechanistic reasoning has been used as a lens to characterize student explanations for London dispersion forces22–24, acid-base reactions6,25, nucleophilic substitution1, and protein binding.26 Within these context students were able to use CMR when constructing explanations by describing the how and the why of the phenomenon. In these studies students from various course backgrounds responded to the prompt and we found that with each phenomena students enrolled in transformed courses more frequently used CMR in their drawing and responses when compared to a Traditional cohort. The transformed courses are Chemistry, Life, the Universe, and Everything (CLUE)27 and Organic Chemistry, Life, the Universe, and Everything (OCLUE)28 and are courses that focus on three-dimensional learning as described in The Framework, and frequently have students construct arguments, provide explanations, and develop and use models. In the context of organic chemistry CMR requires students to think about how and why a reaction occurred which gives us evidence of what students know and can do with their knowledge of reactions. Because of this we believe that CMR can support students drawing mechanistic arrows for reactions because it requires thinking about the cause and effect of the entities involved. Other researchers have investigated students’ mechanistic reasoning across different phenomena within chemistry. Talanquer et al. has interviewed students, from new general chemistry students all the way through graduate level chemistry students, about how and why a reaction occurs.29 These authors characterized students reasoning as linear or multi-component, where multicomponent explanations include how different variables affected the entities involved in a system, and found that most students across all levels tended to provide a linear causal explanation about the reaction, a cause- and-effect relationship, rather than about the underlying properties. In another study by Bodé et al., the same characterization was used for Organic 2 students responses about E1/SN1, E2, or SN2 mechanisms.30 Just as Talanquer reported, these authors also found that student could provide a correct claim about the reaction presented but could not explain the underlying properties for the 122 reaction.30 These studies showed that students struggle with constructing multi-component reasoning which requires students to make multiple connections about the effect multiple variables have on the phenomena. The following study will investigate how students use of CMR can impact their use of arrows in both transformed courses (CLUE and OCLUE) and in a Traditional course. Prior Work This work will combine the coding and evaluation of two studies that were conducted with the same sets of students about their mechanistic arrow use and causal mechanistic reasoning. We will briefly discuss these studies here: Study 1: how students draw mechanistic arrows for familiar and unfamiliar reactions5 and Study 2: how students construct a causal mechanistic explanation of how and why a simple nucleophilic substitution reaction occurs.1 Mechanistic Arrows In Chapter IV and Chapter V, I investigated how different courses affected students’ ability to draw mechanistic arrows and predict a product for both a familiar and unfamiliar reaction. Arrows were coded such that we captured if students were starting at an area of high electron density and ending at a plausible area of low electron density. In this way we could capture the plausible arrows students were drawing for their mechanisms and account for their step-by-step creation of their mechanism and product. In Chapter IV, I showed that even though both cohorts draw a plausible product for a familiar reaction at roughly the same rate (≈ 75%) by the end of OChem 2, OCLUE-OCLUE students draw mechanistic arrows to predict their product more frequently (64%) than Traditional-Traditional students (21%). Additionally, OCLUE-OCLUE students were able to use their arrows to predict a product for an unfamiliar reaction more frequently (27%) than the Traditional-Traditional students (2%). In Chapter V we showed how switching courses, from OCLUE to Traditional and vice versa can affect students’ responses. For example, OCLUE-Traditional students changed in their response from the beginning to 123 the end of OChem 2 for the familiar reaction. At the start of OChem 2 their responses were similar to OCLUE-OCLUE students. However, by the end of OChem 2 the OCLUE-Traditional students’ responses no longer resembled their OCLUE-OCLUE peers (35% plausible arrows and product compared to 67% respectively) in fact they responded more like the Traditional-Traditional students (20%). The opposite was found for the Traditional-OCLUE students, and though the number of responses was limited, these students responded more like Traditional-Traditional at the beginning of OChem 1. By the end of OChem 2 the Traditional-OCLUE students (67%) responded more like the OCLUE-OCLUE students (69%). These same patterns were found for the unfamiliar reaction. Causal Mechanistic Reasoning The work of this section was developed, coded, evaluated, and published by Crandell et al. However, because the same students were in both Crandell et al.’s study1 and in the work presented in Chapter IV and V, I was able to investigate the association between causal mechanistic reasoning and mechanistic arrow use. While “arrow pushing” is the explicit manifestation of how a reaction mechanism occurs, the ability to draw a mechanism relies on a great deal of implicit knowledge about how and why arrows are drawn in such a particular way. In our work we have explicitly linked the ability to draw mechanistic arrows with the scientific practice of constructing causal mechanistic explanations, in particular about how and why reactions occur.6,25 Crandell et al. have built on their prior work1 to define causal mechanistic explanations (CM) as those that include reasoning about both how the electrons move during the reaction and the electrostatic interactions that can cause this movement. An example of a student response that was characterized as a causal mechanistic explanation for a simple SN2 reaction on an alkyl halide was “The carbon has a partial positive on it due to the Br and so the negatively charged O attacks positive carbon with its lone pair breaking the bond of C-Br and those electrons go to the Br.”1 Such explanations require students to explicitly connect and use conceptual resources, such as 124 electrostatic attraction and bond polarity to construct the explanation. Crandell et al. reported longitudinal trends in student reasoning for a simple SN2 reaction for two matched (by prior achievement and demographic information) cohorts: OCLUE-OCLUE and Traditional-Traditional.1 In their first semester both cohorts engaged in CM reasoning similarly with 56-58% of both cohorts providing a CM response. However, by the end of OChem 2, the percent of OCLUE-OCLUE students who provided a CM response increased to 65% whereas the percent of Traditional-Traditional students’ responses decreased to 40%.1 This trend was replicated the following year.1 Additionally 89% of OCLUE-OCLUE students correctly drew an SN2 mechanism and product at the end of OChem 2 while 66% of Traditional- Traditional students drew a correct SN2 mechanism and product. Furthermore, just as with Chapter V, when comparing OCLUE-Traditional and Traditional-OCLUE students the same pattern emerged. At the end of OChem 2 OCLUE-Traditional responded more like Traditional-Traditional, 51% and 40% respectively.31 While Traditional-OCLUE students responded more like OCLUE-OCLUE, 63% and 65% respectively.31 Research Question In this manuscript I report on the correlation between causal mechanistic reasoning and mechanistic arrow use for a familiar and unfamiliar reaction. The guiding questions for this work are: 1. What is the relationship between CMR and students mechanistic arrow use on a familiar and unfamiliar reaction? 2. How does the use of both CMR and mechanistic arrows differ between OCLUE and Trad students? 3. How does a student’s general chemistry background impact their causal mechanistic reasoning and mechanistic arrow use on a familiar and unfamiliar reaction? 125 Methods Design of assessment task The assessment tasks for Study 1 are the same as were discussed in Chapter IV and Chapter V; though here, I will focus solely on the familiar reaction (prompt A) and unfamiliar reaction (prompt E) that are shown in Chapter IV Figure 4.2. The prompt asked students to “Draw a mechanism and predict the product of the following reaction” and then students were shown one of the reactions to complete. The prompt for Study 2 is the same as discussed in the published work by Crandell et al. “Arrows on the Page Are Not a Good Gauge…”.1 For a simple SN2 reaction the prompt asks students to: 1) Classify the reaction and explain their reasoning behind their choice, 2) Describe the sequence of event occurring at the molecular level, 3) Please explain why these reactants interact, and 4) Draw mechanistic arrows to indicate how the reaction occurs, and 5) Explain why you drew your arrows as indicated. Student Participants and Data Collection Both the mechanistic arrow tasks (Study 1) and the causal mechanistic reasoning task (Study 2) were administered using the beSocratic system,32 which allows students to draw freeform responses and write explanations. The students included in this analysis are the same students as those in Chapter V, and was the same prompt timing and administration, the beginning and end of OChem 2. Students were again classified by their OChem course experience path just as in Chapter IV and Chapter V; however, we further disaggregated the cohorts based on students’ general chemistry (GenChem) experience shown in Table 1. In the following subsection we will describe the coding analysis in detail. 126 Table 6.1. Summary of Student responses coded. Traditional is abbreviated Trad for simplicity. Research Question 1 Cohort Total number (OChem1-OChem2) of Students OCLUE-OCLUE 102 Trad-Trad 125 Research Question 2 Cohorts Total number (GenChem1-GenChem2-OChem1-OChem2) of Students CLUE-CLUE-OCLUE-OCLUE 64 TRAD-TRAD-OCLUE-OCLUE 35 CLUE-CLUE-TRAD-TRAD 59 TRAD-TRAD-TRAD-TRAD 61 Study 1: Mechanistic Arrow Use Data Analysis I initially coded the responses in the same way as described in the previous chapters, and the full coding schemes are shown in Chapter IV Appendix C. In a slightly different approach than the previous chapters, we grouped students’ drawings based on if they drew at least one mechanistically reasonable step (called: some plausible arrows). These student could have either a plausible or incorrect product, but did uses at least one mechanistically reasonable step. This allowed us to investigate the ways students are using their arrows even if they are not completely successful at drawing the entire mechanism. I also coded student’s responses based on students mechanistically plausible steps and plausible products they drew for the reactions. However, due to the insufficient number of students in the Traditional cohort, this analysis is included in Appendix A. Study 2: Causal Mechanistic Reasoning Data Analysis Crandell et al. characterized a spectrum of responses that range from descriptive to causal mechanistic. For the simplicity in this paper, I condensed this coding scheme into two categories: causal mechanistic (which include both discussion of electron movement and electrostatic interactions) and non-causal mechanistic (everything else). The expanded coding characterizations are provided in the Appendix B and are published elsewhere.1 127 In Study 2, students were asked to explain an SN2 reaction between CH3Br and OH- by responding to a set of scaffolded prompts. In this publication, I solely report on students’ reasoning since their mechanistic arrow drawings for this prompt have been reported elsewhere and in the introduction.1 A screenshot of this prompt is provided in Appendix B and has been published elsewhere.1 As part of her previous work, Crandell exported the written explanations out of the beSocratic platform32 into a spreadsheet and all pieces of the explanation were coded together and characterized according to the previously published scheme.1 All explanations were previously coded by Crandell et al. for their previous publication1 with the assistance of two undergraduate coders. In instances where disagreements arose, coding was discussed until a complete agreement was reached. As discussed above, for simplicity, I will report responses in terms of causal mechanistic or non-causal mechanistic here, but the full characterization results that were characterized by Crandell at al. are reported in Appendix B. Overlapping analysis of Study 1 and Study 2 The above coding schemes for Study 1 and Study 2 were combined in to four different coding groups that are shown in Table 2. Because we are interested in the relationship between students CMR and student use of mechanistic arrows we separated student responses into those who used CMR and those who did not (Non-CM) for the simple SN2 reaction and those who used some plausible arrows and those that drew no plausible arrows in their mechanisms of a familiar reaction and unfamiliar reaction. The goal is to investigate whether there is a relationship between students’ ability to use CMR and draw mechanistic arrows for both a familiar and unfamiliar reaction. The examples shown in Table 2 are a student’s response for both Study 1 and Study 2 to illustrate the types of responses we are referring to. For example, the responses for Causal Mechanistic and Some Plausible Arrows group (darker orange) are from a student who used CMR for Study 2 and drew at least one plausible arrow for the familiar 128 reaction. The examples in Table 2 are shown for the familiar reaction, and the same groups were created for the unfamiliar reaction. Results Research Question 1: What is the relationship between CMR and students mechanistic arrow use on a familiar and unfamiliar reaction? Finding 1: There is a moderate association between CMR and students use of some plausible arrows Responses from Study 1 were classified into two bins, students who used some (at least one) plausible arrows in their reaction regardless of the product they put and students who did not draw any plausible arrows regardless of the product they put; this was done for both the familiar and unfamiliar reaction. Responses for Study 2 were classified as causal mechanistic and non-causal mechanistic. We will investigate how OCLUE-OCLUE students who use CMR and draw some plausible arrows compare to OCLUE-OCLUE students who did not use CMR but still draw some plausible arrows. The same will be compared for Traditional-Traditional students. This analysis revealed that at the end of OChem 2 both cohorts of students’ explanations did not significantly impact their ability to draw some plausible arrows for a familiar reaction, shown in Table 4 (OCLUE-OCLUE: χ2=2.963; p=0.085; Cramer’s V=0.170) (Traditional-Traditional: χ2=1.733; p=0.188; Cramer’s V=0.118). This may indicate that students memorize this reaction so it seems likely that they will draw at least one arrow correctly and thus the impact of using CMR is low. 129 Table 6.2. Code groups for student’s use of CMR & Some Plausible Arrows. The colors correspond to the graphs in the results section. Student Examples Code Grouping Description of Code Grouping Study 1 Study 2 Study 1: Student drew at least one mechanistically reasonable step (product can be correct or incorrect) Causal Mechanistic and Study 2: Student responses that include Some Plausible Arrows reasoning about both how the electrons move during the reaction and the electrostatic the o has - charged lone pairs which are attracted to the sightly interactions that can cause this movement. positive c the c is positive bc the br is electronegative so it pulls the electrons in the bond towards it Study 1: Student drew no mechanistically reasonable steps (product can be correct or incorrect) Causal Mechanistic and No Plausible Arrows Study 2: Student responses that include reasoning about both how the electrons move The bond between the Br and C is broken by the solvent. Along during the reaction and the electrostatic with it the electrons that were shared go to bromine. The lone interactions that can cause this movement. pair on the oxygen is used form a bond between carbon and OH. Study 1: Student drew at least one mechanistically reasonable step (product can be correct or incorrect) Non- Causal Mechanistic and Some Study 2: Student responses that do not include Plausible Arrows reasoning about both how the electrons move I drew my arrows as indicated because the oxygen donates during the reaction and the electrostatic electrons to the central carbon and the bromine accepts the interactions that can cause this movement. extra electrons. Study 1: Student drew no mechanistically reasonable steps (product can be correct or incorrect) Non- Causal Mechanistic and No Study 2: Student responses that do not include Plausible Arrows reasoning about both how the electrons move during the reaction and the electrostatic The Hydrogen moves to the oxygen and then the oxygen moves interactions that can cause this movement. to the next hydrogen. The Br breaks off and becomes negative 130 However, at the end of OChem 2 there is a moderate association between students using CMR and students’ ability to draw some plausible arrows (OCLUE-OCLUE: χ2=5.259; p=0.022; Cramer’s V=0.227) (Traditional-Traditional: χ2=7.933; p=0.005; Cramer’s V=0.252). There is a medium effect size for both cohorts and thus a moderate association between student’s use of CMR and drawing some plausible arrows. This indicated that, students who use CMR for their explanations of a simple SN2 are more likely to draw some plausible arrows for an unknown reaction they have not seen before. We conducted a post-hoc analysis and calculated the standardized residuals to better understand what was driving the relationship between both OCLUE-OCLUE and Traditional-Traditional students reasoning, and the arrows they draw for the unfamiliar reaction shown in Figure 2 below. Standardized residual values that are positive (blue) indicated when there were more observed counts than would be expected if there was no relationship between CMR and drawing some arrows for an unfamiliar reaction. The standardized residual values that are negative (red) mean there are less observed instances than would be expected it there was no relationship. If the standardized residual is larger than 2.58 it is considered significant and that the relationship in that given cell is a primary driver of the significant relationship. Figure 2 revealed that a positive association between using CMR and drawing some plausible arrows was the primary driver of significance for both groups of students. It therefore follows that there is a negative association between Non-CMR and drawing some arrows, indicating that when students are Non-CM they are less likely to draw some plausible arrows for the unfamiliar reaction. This is not a reaction students can readily memorize (to our knowledge) and thus we are better able to see the effect CMR has on students engaging with drawing some plausible arrows. Regardless of course type students who use CMR for a simple SN2 are more likely to draw some plausible arrows for an unfamiliar reaction. 131 Table 6.3. Percent of Students who used CMR and Some Plausible Arrows for both OCLUE-OCLUE and Traditional-Traditional courses. Course Time Point Cohorts Counts χ2 (df = 1) p-value Cramer’s V Causal Mechanistic (Causal Mechanistic & Some Arrows/ (62/66) = 94% End OChem 2 Causal Mechanistic total) 2.963 0.085a 0.170 Study 1: Familiar Non- Causal Mechanistic (Non- Causal Mechanistic & Some (30/36) = 83% Arrows/ Non-Causal Mechanistic total) OCLUE-OCLUE Causal Mechanistic (Causal Mechanistic & Some Arrows/ (54/66) = 82% 0.227 End OChem 2 Causal Mechanistic total) 5.259 0.022a (small- Study 1: Unfamiliar Non- Causal Mechanistic (Non- Causal Mechanistic & Some (22/36) = 61% medium) Arrows/ Non-Causal Mechanistic total) Causal Mechanistic (Causal Mechanistic & Some Arrows/ (30/50) = 60% End OChem 2 Causal Mechanistic total) 1.733 0.188a 0.118 Study 1: Familiar Non- Causal Mechanistic (Non- Causal Mechanistic & Some (36/75) = 48% Arrows/ Non-Causal Mechanistic total) Trad-Trad Causal Mechanistic (Causal Mechanistic & Some Arrows/ (20/50) = 40% 0.252 End OChem 2 Causal Mechanistic total) 7.933 0.005a (medium- Study 1: Unfamiliar Non- Causal Mechanistic (Non- Causal Mechanistic & Some (13/75) = 17% large) Arrows/ Non-Causal Mechanistic total) aFor all Chi-square analysis, α = 0.01 was used, significant finding are bolded. Traditional was abbreviated Trad to simplify the table. Interpretation of Cramer’s V is as follows: small effect size between 0.1 and 0.3, medium effect size between 0.3 and 0.5, and large effect size greater than 0.5. Figure 6.1. Standardized residual values that are positive (blue) indicated where there were more observed counts than would be expected if there was no relationship, values that are negative (red) mean there are less observed instances than would be expected if there was no relationship. 132 Research Question 2: How does the use of both CMR and mechanistic arrows differ between OCLUE and Traditional students? Finding 2: OCLUE-OCLUE students tend to use CMR and some plausible arrows more frequently than Traditional-Traditional students We will now focus on the association students’ course type has on their use of CMR and mechanistic arrows. We found that OCLUE-OCLUE students were more likely to use CMR and draw some plausible arrows compared to Traditional-Traditional students (darker orange group) across both timepoints and both reactions with medium to large effect sizes (0.415-0.431) shown in Figure 1 and Table 3. Thus, there is a strong association between student’s course type and their ability to use CMR and draw some plausible arrows. This aligns with what we have seen previously that OCLUE-OCLUE students are more successful at writing causal mechanistic explanations and drawing mechanistic arrows to predict a product.1,5 Table 6.4. Comparison of OCLUE-OCLUE students to Traditional-Traditional students use of CMR and drawing some plausible arrows. Counts Time Point Cohorts (Causal Mechanistic & Some χ2 (df = 1) p-value Cramer’s V Arrows/ Causal Mechanistic total) Start OChem 2 OCLUE-OCLUE n = 102 (54/60) = 90% 0.424 20.177 < 0.001a Study 1: Familiar Trad-Trad n = 125 (27/52) = 52% (medium-large) End OChem 2 OCLUE-OCLUE n = 102 (62/66) = 94% 0.415 19.970 < 0.001a Study 1: Familiar Trad-Trad n = 125 (30/50) = 60% (medium-large) End OChem 2 OCLUE-OCLUE n = 102 (54/66) = 82% 0.431 21.539 < 0.001a Study 1: Unfamiliar Trad-Trad n = 125 (20/50) = 40% (medium-large) aFor all Chi-square analysis, α = 0.01 was used. Traditional was abbreviated Trad to simplify the table. Interpretation of Cramer’s V is as follows: small effect size between 0.1 and 0.3, medium effect size between 0.3 and 0.5, and large effect size greater than 0.5. 133 Figure 6.2. Percent of organic chemistry student’s use of CMR and some plausible arrow use across both time points. Research Question 3: How does a student’s general chemistry background impact their causal mechanistic reasoning and mechanistic arrow use on a familiar and unfamiliar reaction? Finding 3: Students’ general chemistry background does not impact their use of CMR and use of some plausible arrows We next explored the effect student’s general chemistry background had on their use of CMR and arrows in organic chemistry. While there are a number of different course combinations students can enroll through their general chemistry and organic chemistry sequence, we have chosen to highlight a few groups. Though the number of students in each of these groups is limited it serves as a basis for further investigation on students’ course trajectory at the institution. Students’ courses will always be listed in chronological order, GenChem1- GenChem2- OChem 1- OChem 2. For the purposes of this manuscript, I selected 4 cohorts of students to compare students who only had transformed courses (CLUE-CLUE-OCLUE-OCLUE), students who had transformed GenChem and traditional OChem (CLUE- CLUE-TRAD-TRAD), students who had a traditional GenChem and transformed OChem (TRAD-TRAD- 134 OCLUE-OCLUE), and finally students who had only traditional GenChem and OChem (TRAD-TRAD-TRAD- TRAD). In this way I could begin to analyze select amounts of transformed and traditional curriculum when comparing groups of students. At the end of OChem 2 CLUE-CLUE-OCLUE-OCLUE and TRAD-TRAD-OCLUE-OCLUE students who used CMR were far more likely to draw some plausible arrows for the familiar reaction compared to the other groups of students shown in Table 5 and Figures 3-5. This would indicate that there may not be a strong relationship between students’ general chemistry background and their use of CMR and mechanistic arrows. We found that 95% of CLUE-CLUE-OCLUE-OCLUE and TRAD-TRAD-OCLUE-OCLUE students who used CMR also drew some plausible arrows while 65% of CLUE-CLUE-TRAD-TRAD students and 52% of TRAD-TRAD-TRAD-TRAD did the same. For the unfamiliar reaction the same pattern was observed, CLUE-CLUE-OCLUE-OCLUE and TRAD-TRAD-OCLUE-OCLUE students who used CMR were more likely to draw some plausible arrows, compared to CLUE-CLUE-TRAD-TRAD and TRAD-TRAD-TRAD-TRAD students as shown in Figure 6. Because the unfamiliar reaction is much more difficult and the number of successful students in both CLUE-CLUE-TRAD-TRAD and TRAD-TRAD-TRAD-TRAD is small statistical comparison of these groups is not feasible. However, in line with our previous finding, students who had OCLUE for their OChem course are more successful at drawing some plausible arrows for an unfamiliar reaction if they used CMR then students who had Traditional for OChem. 135 Table 6.5. Percent of Students who used CMR and Some Plausible Arrows for various course sequences. Cohorts Counts Time Point (CMR & Some (GenChem1-GenChem2-OChem1-OChem2) Arrows/CMR total) CLUE-CLUE-OCLUE-OCLUE n = 64 (34/37) = 92% Start OChem 2 TRAD-TRAD-OCLUE-OCLUE n =35 (19/21) = 90% Study 1: Familiar CLUE-CLUE-TRAD-TRAD n = 59 (12/26) = 46% TRAD-TRAD-TRAD-TRAD n = 61 (14/24) = 58% CLUE-CLUE-OCLUE-OCLUE n = 64 (40/42) = 95% End OChem 2 TRAD-TRAD-OCLUE-OCLUE n =35 (21/22) = 95% Study 1: Familiar CLUE-CLUE-TRAD-TRAD n = 59 (15/23) = 65% TRAD-TRAD-TRAD-TRAD n = 61 (13/25) = 52% CLUE-CLUE-OCLUE-OCLUE n = 64 (33/42) = 79% End OChem 2 TRAD-TRAD-OCLUE-OCLUE n =35 (19/22) = 86% Study 1: CLUE-CLUE-TRAD-TRAD n = 59 (9/23) = 39% Unfamiliar TRAD-TRAD-TRAD-TRAD n = 61 (10/25) = 40% Traditional was abbreviated Trad to simplify the table. Course groups are listed in order of GenChem1-GenChem2-OChem1-OChem2 Figure 6.3. Percent of student’s use of CMR and some plausible arrows used at the start of OChem 2 for a familiar reaction. 136 Figure 6.4. Percent of student’s use of CMR and some plausible arrows used at the end of OChem 2 for a familiar reaction. Figure 6.5. Percent of student’s use of CMR and some plausible arrows used at the end of OChem 2 for an unfamiliar reaction. 137 Discussion The goals of this study were to investigate the relationship between CMR and students use of mechanistic arrows on familiar and unfamiliar reactions, and to determine whether course sequence impacted this relationship. We chose to simplify the comparisons by comparing student use of some plausible arrows (rather than all plausible arrows) because it expands the number of students for our study. These results showed that students who use CMR for a simple SN2 mechanism were more likely to draw some plausible arrows for an unfamiliar reaction regardless of their course trajectory. Mechanistic arrows are a tool used to show the implicit flow of electrons from high electron density to low electron density and as the literature has suggested students may simply use the arrows without considering the role of electrons.2–9 Causal mechanistic reasoning in an organic reaction requires that students engage with how and why the electrostatic interactions occurs. By asking students to explain their reasoning in words it gives us insight into how they are thinking about a reaction. These finding suggest that students who have taken OCLUE for their OChem sequence are more likely to use CMR for a simple SN2 reaction and are more likely to draw plausible mechanisms for both a familiar and unfamiliar reaction. However, students use of CMR is not correlated with students’ ability to draw some plausible arrows for a familiar reaction but students use of CMR does correlate with students’ ability to draw some plausible arrows for an unfamiliar reaction. This may be because the familiar reaction is common and easy to memorize for students, thus the likelihood they draw at least one plausible arrow for the reaction is high. However, because the unfamiliar reaction is not a mechanism student have seen before, to our knowledge, it seems likely that those who use CMR for a simple SN2 reaction they are more likely to draw some plausible arrows for an unfamiliar reaction. We posit that students who have shown us evidence that they can reason about the how and why a reaction occurs (CMR for the simple SN2) it is likely that they can understand, at least in part, the electrostatic interactions involved in an unfamiliar reaction. 138 Furthermore, when extending the findings to students’ entire course sequence from GenChem 1 to OChem 2 we found that students general chemistry experience had little to no impact on their use of CMR and mechanistic arrows. The CLUE-CLUE-TRAD-TRAD students appear to use CMR and some plausible arrows less frequently than CLUE-CLUE-OCLUE-OCLUE whom they had GenChem with. Though student learn about the Brønsted model and the Lewis acid-base model in CLUE these are much smaller molecules typically with all lone pairs and atoms drawn explicitly. Research has shown that students general chemistry experience can impact their use of CMR in OChem when these simpler molecular are involved.25 However, it may be that by the end of OChem 2 for more complex molecules the CLUE-CLUE- TRAD-TRAD students no longer use these same ways of thinking they acquired in CLUE GenChem. Conversely, students who had TRAD-TRAD-OCLUE-OCLUE are able to integrate these ways of thinking about how and why reactions occur, into their reasoning because as we have shown, they reason similarly to CLUE-CLUE-OCLUE-OCLUE students. This finding echo’s that of Chapter V, if we are to support students constructing explanations and drawing mechanistic arrows, it needs to be consistently assessed on and supported throughout the curriculum such as in the case of CLUE and OCLUE. Implications and Future directions The information provided in this chapter supports the notion that there is a strong association between students constructing causal mechanistic explanations and their ability to draw mechanistic arrows. Because students must know how and why a reaction occurs to be able to explain using causal mechanistic reasoning, this may be one way to activate students’ resources related to the how and why of mechanistic arrows. Courses such as CLUE and OCLUE that emphasize such reasoning give students the opportunity to use this type of reasoning throughout the course. By helping to make connections between the appropriate resources frequently throughout the course, students are more likely to call upon that reasoning when faced with an unfamiliar reaction. Because the goal in organic chemistry is to 139 have students make plausible and correct predictions about reactions and transformed courses like CLUE and OCLUE are one example of how to help students make meaning of organic chemistry. Limitations One limitation of this study is that there is a limited number of students’ in the groups comparing GenChem trajectories, which makes comparisons somewhat difficult. Future iterations of this prompt may be administered more broadly to capture the ideas of students in these different groups. The additionally the cohort of students in this study took these assessments as an extra credit activity on the beSocratic platform over the course of a week. While most students have used the platform for their homework assignments before, there may be some students who are unfamiliar with it. Additionally, because students were given a week to complete the assessment and though we asked them to complete the tasks individually and to not consult their notes or their TA there is not a guarantee that students did not use outside help when answering these questions. 140 REFERENCES (1) Crandell, O. M.; Lockhart, M. A.; Cooper, M. M. Arrows on the Page Are Not a Good Gauge: Evidence for the Importance of Causal Mechanistic Explanations about Nucleophilic Substitution in Organic Chemistry. J. Chem. Educ. 2020, 97 (2), 313–327. https://doi.org/10.1021/acs.jchemed.9b00815. (2) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844–849. https://doi.org/10.1021/ed2003934. (3) Flynn, A. B.; Featherstone, R. B. Language of Mechanisms: Exam Analysis Reveals Students’ Strengths, Strategies, and Errors When Using the Electron-Pushing Formalism (Curved Arrows) in New Reactions. Chem. Educ. Res. Pract. 2017, 18 (1), 64–77. https://doi.org/10.1039/C6RP00126B. (4) Caspari, I.; Weinrich, M. L.; Sevian, H.; Graulich, N. This Mechanistic Step Is “Productive ”: Organic Chemistry Students’ Backward-Oriented Reasoning. Chem. Educ. Res. Pract. 2018, 19 (1), 42–59. https://doi.org/10.1039/C7RP00124J. (5) Houchlei, S. K.; Bloch, R. R.; Cooper, M. M. Mechanisms, Models, and Explanations: Analyzing the Mechanistic Paths Students Take to Reach a Product for Familiar and Unfamiliar Organic Reactions. J. Chem. Educ. 2021, 98 (9), 2751–2764. https://doi.org/10.1021/acs.jchemed.1c00099. (6) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid–Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703–1712. https://doi.org/10.1021/acs.jchemed.6b00417. (7) Ferguson, R.; Bodner, G. M. Making Sense of the Arrow-Pushing Formalism among Chemistry Majors Enrolled in Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9 (2), 102–113. https://doi.org/10.1039/B806225K. (8) Graulich, N.; Bhattacharyya, G. Investigating Students’ Similarity Judgments in Organic Chemistry. Chem. Educ. Res. Pract. 2017, 18 (4), 774–784. https://doi.org/10.1039/C7RP00055C. (9) Galloway, K. R.; Leung, M. W.; Flynn, A. B. Patterns of Reactions: A Card Sort Task to Investigate Students’ Organization of Organic Chemistry Reactions. Chem. Educ. Res. Pract. 2019, 20 (1), 30– 52. https://doi.org/10.1039/C8RP00120K. (10) Cartrette, D. P.; Bodner, G. M. Non-Mathematical Problem Solving in Organic Chemistry. Journal of Research in Science Teaching 2010, 47 (6), 643–660. https://doi.org/10.1002/tea.20306. (11) Connor, M. C.; Glass, B. H.; Shultz, G. V. Development of the NMR Lexical Representational Competence (NMR-LRC) Instrument As a Formative Assessment of Lexical Ability in 1 H NMR Spectroscopy. J. Chem. Educ. 2021, 98 (9), 2786–2798. https://doi.org/10.1021/acs.jchemed.1c00332. 141 (12) Connor, M. C.; Glass, B. H.; Finkenstaedt-Quinn, S. A.; Shultz, G. V. Developing Expertise in 1 H NMR Spectral Interpretation. J. Org. Chem. 2021, 86 (2), 1385–1395. https://doi.org/10.1021/acs.joc.0c01398. (13) Connor, M. C.; Finkenstaedt-Quinn, S. A.; Shultz, G. V. Constraints on Organic Chemistry Students’ Reasoning during IR and 1 H NMR Spectral Interpretation. Chem. Educ. Res. Pract. 2019, 20 (3), 522–541. https://doi.org/10.1039/C9RP00033J. (14) Caspari, I.; Kranz, D.; Graulich, N. Resolving the Complexity of Organic Chemistry Students’ Reasoning through the Lens of a Mechanistic Framework. Chem. Educ. Res. Pract. 2018, 19 (4), 1117–1141. https://doi.org/10.1039/C8RP00131F. (15) Popova, M.; Bretz, S. L. “It’s Only the Major Product That We Care About in Organic Chemistry”: An Analysis of Students’ Annotations of Reaction Coordinate Diagrams. J. Chem. Educ. 2018, 95 (7), 1086–1093. https://doi.org/10.1021/acs.jchemed.8b00153. (16) Popova, M.; Bretz, S. L. Organic Chemistry Students’ Interpretations of the Surface Features of Reaction Coordinate Diagrams. Chem. Educ. Res. Pract. 2018, 19 (3), 919–931. https://doi.org/10.1039/C8RP00063H. (17) Popova, M.; Bretz, S. L. Organic Chemistry Students’ Challenges with Coherence Formation between Reactions and Reaction Coordinate Diagrams. Chem. Educ. Res. Pract. 2018, 19 (3), 732–745. https://doi.org/10.1039/C8RP00064F. (18) DeGlopper, K. S.; Schwarz, C. E.; Ellias, N. J.; Stowe, R. L. Impact of Assessment Emphasis on Organic Chemistry Students’ Explanations for an Alkene Addition Reaction. J. Chem. Educ. 2022, 99 (3), 1368–1382. https://doi.org/10.1021/acs.jchemed.1c01080. (19) A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas; National Academies of Sciences, Engineering, and Medicine, Ed.; The National Academies Press: Washington, D.C, 2012. (20) Hammer, D. Student Resources for Learning Introductory Physics. Am. J. Phys. 2000, 68 (S1), S52–S59. https://doi.org/10.1119/1.19520. (21) Russ, R. S.; Scherr, R. E.; Hammer, D.; Mikeska, J. Recognizing Mechanistic Reasoning in Student Scientific Inquiry: A Framework for Discourse Analysis Developed from Philosophy of Science. Science Education 2008, 92 (3), 499–525. https://doi.org/10.1002/sce.20264. (22) Becker, N.; Noyes, K.; Cooper, M. Characterizing Students’ Mechanistic Reasoning about London Dispersion Forces. Journal of Chemical Education 2016, 93 (10), 1713–1724. https://doi.org/10.1021/acs.jchemed.6b00298. (23) Noyes, K.; Cooper, M. M. Investigating Student Understanding of London Dispersion Forces: A Longitudinal Study. J. Chem. Educ. 2019, 96 (9), 1821–1832. https://doi.org/10.1021/acs.jchemed.9b00455. (24) Noyes, K.; McKay, R. L.; Neumann, M.; Haudek, K. C.; Cooper, M. M. Developing Computer Resources to Automate Analysis of Students’ Explanations of London Dispersion Forces. J. Chem. Educ. 2020, 97 (11), 3923–3936. https://doi.org/10.1021/acs.jchemed.0c00445. 142 (25) Crandell, O. M.; Kouyoumdjian, H.; Underwood, S. M.; Cooper, M. M. Reasoning about Reactions in Organic Chemistry: Starting It in General Chemistry. J. Chem. Educ. 2019, 96 (2), 213–226. https://doi.org/10.1021/acs.jchemed.8b00784. (26) Noyes, K.; Carlson, C. G.; Stoltzfus, J. R.; Schwarz, C. V.; Long, T. M.; Cooper, M. M. A Deep Look into Designing a Task and Coding Scheme through the Lens of Causal Mechanistic Reasoning. J. Chem. Educ. 2022, 99 (2), 874–885. https://doi.org/10.1021/acs.jchemed.1c00959. (27) Cooper, M.; Klymkowsky, M. Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform. J. Chem. Educ. 2013, 90 (9), 1116–1122. https://doi.org/10.1021/ed300456y. (28) Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96 (9), 1858–1872. https://doi.org/10.1021/acs.jchemed.9b00401. (29) Weinrich, M. L.; Talanquer, V. Mapping Students’ Modes of Reasoning When Thinking about Chemical Reactions Used to Make a Desired Product. Chem. Educ. Res. Pract. 2016, 17 (2), 394– 406. https://doi.org/10.1039/C5RP00208G. (30) Bodé, N. E.; Deng, J. M.; Flynn, A. B. Getting Past the Rules and to the WHY: Causal Mechanistic Arguments When Judging the Plausibility of Organic Reaction Mechanisms. J. Chem. Educ. 2019, 96 (6), 1068–1082. https://doi.org/10.1021/acs.jchemed.8b00719. (31) Crandell, O. M. Investigation of Students’ Causal Mechanistic Reasoning in Undergraduate Organic Chemistry. Ph.D. Dissertation, Michigan State University, MI, 2020. (32) Bryfczynski, S. P. BeSocratic: An Intelligent Tutoring System for the Recognition, Evaluation, and Analysis of Free-Form Student Input. Ph.D. Dissertation, Clemson University, SC, 2010. 143 APPENDIX A: ALL PLAUSIBLE ARROWS CODING SCHEME AND ANALYSIS Table 6.6. Code groups for student’s use of CMR and Plausible Arrows/Products. Student Examples Code Grouping Description of Code Grouping Study 1 Study 2 Study 1: Student drew all mechanistically plausible steps and a plausible product Causal Mechanistic and Plausible Study 2: Student responses that include reasoning Product & Plausible Partial positive charge on the carbon is attracted to about both how the electrons move during the Arrows the negative charge on the oxygen. The electrons reaction and the electrostatic interactions that can between the C-Br bond are retained to the cause this movement. electronegative Bromide. Study 1: Student drew a mixture of mechanistically plausible steps, incorrect arrows, and a mixture of plausible and incorrect product Causal Mechanistic and Assortment of Study 2: Student responses that include reasoning This reacts because of the electromagnetic force of Arrows & Products about both how the electrons move during the attraction. The negative charge on the O is attracted reaction and the electrostatic interactions that can to the partial positive charge on the C in CH3Br cause this movement. allowing the reaction to begin. Study 1: Student drew all mechanistically plausible steps and a plausible product Non-Causal Mechanistic and Study 2: Student responses that do not include Plausible Product & reasoning about both how the electrons move Because of the negative charge on the oxygen and the Plausible Arrows during the reaction and the electrostatic partial positive charge on the carbon. Because the interactions that can cause this movement. oxygen attacks the carbon then the bromine leaves Study 1: Student drew a mixture of mechanistically plausible steps, incorrect arrows, and a mixture of Non-Causal plausible and incorrect product Mechanistic and Assortment of Study 2: Student responses that do not include Arrows & Products reasoning about both how the electrons move during the reaction and the electrostatic To show how the negative oxygen attacks the Carbon interactions that can cause this movement. which breaks from partial negative bromine 144 A brief discussion on the relationship between CMR and students use of all plausible arrows and a plausible product for a familiar and unfamiliar reaction. This is a more difficult task as students need to draw all plausible steps to their mechanism and a plausible product rather than comparing those who only drew, at minimum, one arrow correctly. We found that for the familiar reaction the OCLUE-OCLUE using CMR did not significantly impact students’ ability to draw all plausible arrows and a plausible product than OCLUE-OCLUE students who were Non- CM, shown in Table 7. However, Traditional- Traditional students who use CMR are significantly more likely to draw all plausible arrows than Traditional-Traditional students who were Non-CM (Traditional: χ2=7.500; p=0.006; Cramer’s V=0.245). Unfortunately, the numbers of Traditional-Traditional students who were able to complete the unfamiliar task appropriately are too small to compare statistically. Again, we see a larger percent of OCLUE-OCLUE students who use CMR and draw all plausible arrows produce and a plausible product shown in Figure 3. Though the number of OCLUE-OCLUE students who used CMR and drew all plausible arrows and product was not significantly different than the OCLUE-OCLUE students who were Non-CM and drew all plausible arrows and a plausible product, students’ reasoning did have more of an effect on the unfamiliar reaction (Cramer’s V of 0.164, small-medium) compared to the familiar reaction (Cramer’s V of 0.087, negligible). 145 Table 6.7. Comparison of OCLUE-OCLUE students to Traditional-Traditional students use of CMR and drawing all plausible arrows and a plausible across both time points. Course Time Point Cohorts Counts χ2 (df = 1) p-value Cramer’s V Causal Mechanistic (Causal Mechanistic & Some Arrows/ (46/66) = 70% End OChem 2 Causal Mechanistic total) 0.773 0.379a 0.087 Study 1: Familiar Non- Causal Mechanistic (Non- Causal Mechanistic & Some (22/36) = 61% Arrows/ Non-Causal Mechanistic total) OCLUE Causal Mechanistic (Causal Mechanistic & Some Arrows/ (21/66) = 32% End OChem 2 Causal Mechanistic total) 0.164 2.747 0.097a Study 1: Unfamiliar Non- Causal Mechanistic (small-medium) (Non- Causal Mechanistic & Some (6/36) = 17% Arrows/ Non-Causal Mechanistic total) Causal Mechanistic (Causal Mechanistic & Some Arrows/ (13/50) = 26% End OChem 2 Causal Mechanistic total) 0.245 7.500 0.006a Study 1: Familiar Non- Causal Mechanistic (small-medium) (Non- Causal Mechanistic & Some (9/75) = 12% Arrows/ Non-Causal Mechanistic total) Traditional Causal Mechanistic (Causal Mechanistic & Some Arrows/ (2/50) = 4% End OChem 2 Causal Mechanistic total) Too few students to preform analysis Study 1: Unfamiliar Non- Causal Mechanistic (Non- Causal Mechanistic & Some (1/75) < 1% Arrows/ Non-Causal Mechanistic total) aFor all Chi-square analysis, α = 0.01 was used, significant finding are bolded. Interpretation of Cramer’s V is as follows: small effect size between 0.1 and 0.3, medium effect size between 0.3 and 0.5, and large effect size greater than 0.5. Figure 6.6. Percent of organic chemistry student’s use of CMR and drawing all plausible arrows and a plausible across both time points. 146 APPENDIX B: STUDY 2 – CAUSAL MECHANISTIC REASONING Table 6.8. Causal Mechanistic Reasoning Student Examples. Characterization Code Description Examples of Student Responses No Response (NR) Student does not provide an answer. Jessica: “I don’t know what to say.” Explanations are unreadable or incomprehensible. Student does not even attempt to answer. Non-Normative Student provides a non-normative or Emmy: “While oxygen accepted a proton and (NN) unrelated explanation. formed O−H bond and Br leaves.” Student attributes the mechanism to other Mason: “The Br is attracted to the H.” types of reactions or other types of macroscopic observations. The response is discussing incorrect entities and/or incorrect processes. Descriptive Student provides a scientifically simplistic Calvin: “The nucleophile attacks the electrophile General (DG) description of bond formation and bond which makes the leaving group leave.” breaking. Phyllis: “First the OH attacks the carbon center and the Br leaves (carbon−bromine bond breaks) this happens in one step.” Descriptive Causal Student discusses the electrostatic attraction Barbara: “These reactants interact because the OH (DC) between the species. group has a negative charge and is therefore Student gives evidence that they understand nucleophilic. It wants to attack a carbon center (or that there is an attraction between the OH- something with a positive charge even if its partial) and partial positive carbon atom. the Br is a good leaving group (better than OH) so Students do not need to justify why the OH is able to come in and take its place.” carbon is partial positive. They just need to Ryan: “The carbon is slightly positive because the demonstrate an understanding of the bromine is pulling the electrons away from the intermolecular electrostatic attraction. carbon. The negative oxygen attracts the partially positive carbon and the bromine is pushed off and a new bond is made between the carbon and the oxygen.” Descriptive Student identifies electrons as the entities Wanda: “The electrons from the negatively charged Mechanistic (DM) responsible for the reaction mechanism and OH are going to attack the carbon. This will push off explains their activities that lead to bond the bromine and the bromine will get the electrons formation/bond breaking. from the bond between the carbon−bromine Student gives evidence that they understand bond.” that electron movement from source to sink is Morgan: “The lone pair on OH is forming a bond how the reaction occurs. with carbon.” Response may only explicitly discuss the movement of the lone pair of electrons on the OH− or the electrons in the C−Br bond. This is still considered mechanistic. Causal Student provides both the causal and the Megan: “The carbon has a partial positive on it due Mechanistic (CM) mechanistic account of the reaction. to the Br and so the negatively charged O attacks Evidence that the student understands that positive carbon with its lone pair breaking the bond the lone pair of electrons on OH− is attracted of C−Br and those electrons go to the Br.” to the carbon on methyl bromide and the Travis: “The bromine leaves and takes the C−Br electrons in the C−Br bond go to the Br to electrons with it. This leaves a carbocation which become Br−. then attracts the lone pair on the oxygen to make the bond.” 147 Figure 6.7. Causal Mechanistic Reasoning Prompt. Figure 6.8. Causal Mechanistic Reasoning Full Characterization. 148 CHAPTER VII- THE BEST-LAID PLANS OFT' GO AWRY: HOW CHANGING SCAFFOLDING MAY HAVE UNFORSEEN CONSEQUENCES IN STUDENT RESPONSES Preface This work would not have been possible without the help of Jacob Starkie (JS) and Robby McKay (RM), two high school science teachers and post baccalaureate researchers who helped with coding and data collection for this chapter. Introduction In this chapter we will explore and extension of the prior studies in the context of a new, and more difficult phenomena of an intramolecular reaction. The background and literature underpinning of this chapter follows the proceeding chapters. As discussed previously, a large body of literature has been dedicated to students’ difficulties with using mechanisms, ranging from students focusing only on the product of a reaction1–4, relying on heuristics to explain a reaction5–7, rote memorization of reaction steps8–10, and a disconnect between students’ mechanisms and reaction energy diagrams.11–13 Furthermore, when researchers investigated how students understand more difficult phenomena, such as intramolecular reactions14,15, and E1/ SN1, E2, or SN2 mechanisms16–18 they found that students are not as successful nor able to achieve the most sophisticated reasoning for the given task. Lieber et al. investigated students’ arguments about whether an intramolecular reaction was possible when given the reaction of 4-chlorobutanol and hydroxide.15 They found without additional prompting, very few students were able to identify an intramolecular reaction as a plausible reaction pathway and most students choose an SN2 reaction or acid-base. When prompted with example products of the 4- chlorobutanol and hydroxide reaction more students claimed an intramolecular reaction was plausible however, most students could not support their claim with evidence or reasoning.15 Thus, further investigation into students’ understanding of more complex reactions, such as intramolecular reactions is necessary. 149 To investigate students’ understanding of a phenomenon such, as an intramolecular reaction, a prompt requires the appropriate amount of scaffolding, and it may take many iterations and reevaluations before a proper balance is reached. This process of prompt design frequently happens behind the scenes, and rarely do we get to see the trials and tribulations of prompt development. One example from Noyes et al. explored the types of scaffolding necessary to elicit students’ understanding of protein-ligand binding.19 The initial version of the task asked students to indicate where the charges are in a glucose molecule and then where it would bind to a protein that had a polar and non-polar region.19 They found that many students referenced hydrogen bonding but were not able to link their descriptions to the interaction with charge. In the subsequent prompt revision, they instead gave students two different binding sites, one of which had more polar amino acids, and asks students to predict the binding site glucose would most likely bind to and to explain their reasoning. The authors found that by removing the initial question about charge far fewer students discussed the role charge has on the interaction. In the next two iterations of the prompt the authors changed the ligand to a magnesium ion to reduce the extraneous information and students could no longer rely on “hydrogen bonding” as their explanation. The authors found that this appeared to be the right amount of scaffolding to elicit students’ understanding of the electrostatic involved in protein-ligand binding.19 There are others that have outlined their prompt revision process,20–22 and the work presented in this chapter seeks to build on this transparency of prompt development. This work will provide a detailed account of how a prompt was developed, iterated, and evaluated for students’ understanding of an intramolecular organic reaction. We discuss both the prompt development and the coding schemes designed to evaluate the prompt developments, which also provides insights into the resources students use to reason about the intramolecular reaction. 150 Research Questions Here we report on how various types of scaffolding can influence how students respond to a question about an intramolecular reaction. This work is guided by ECD to revise the questions over time. The guiding questions for this work are: 1. What impact does additional scaffolding have on students’ use of mechanistic arrows and predicting a product for an intramolecular ring closure? 2. What are the different types of resources students use to explain the phenomenon of an intramolecular ring closure? Methods The intramolecular organic reaction shown in Figure 1 was the starting point for investigating students’ understanding of intermolecular reactions and will be iterated on throughout this chapter. This type of more complex intramolecular reaction requires students to understand the distribution of charges in the molecule, and to predict the reactivity within the molecule. To do this student must call upon numerous resources that they have gathered throughout their time in general and organic chemistry to make a chain of inferences about the molecule. An ideal student response would identify where the charges in the molecule are and would predict and explain how this molecule would react. However, this requires students to make many links between their reasoning for this type of reaction which is shown in Figure 1. Though entropy does play a role in intramolecular reaction, we chose to focus on students’ understanding of electrostatics of this reaction. During the development of this prompt, several iterations were made to explore how students responded to different types of scaffolding. Thus, the student participants are described in the methods section while the coding schemes emerged as we reviewed and iterated on each version of the prompt. The response from students guided these iterations and will be discussed in depth as part of the results of this work. 151 Figure 7.1. Initial intramolecular reaction used as the foundation for all prompt iterations. The diagram shown the chain of inferences that are necessary to fully understand why the reaction is occurring. Student Participants Institutional Context This research was conducted at a large research-intensive university located in the Midwest from Fall 2019 to Spring 2022. During this time, there were several different instructors who taught OChem 1 and OChem 2. Each semester, instructors of these courses were contacted for permission to send their students consent forms and the prompt for that given year. Between Fall 2019 and Spring 2022 there were six different instructors that taught either OChem 1, OChem 2, or both. This can lead to many different combinations of instructor’s students may have for OChem 1 and OChem 2. For example, in the Spring 2020 (Fall 2019-Spring 2020 academic year), there were 1120 students who enrolled in OChem 2, and there were 15 different instructor paths these students could take through OChem 1 and OChem 2. The list of different instructor paths is listed in the Appendix A. At this institution the organic sequence has an agreed upon set of topics that instructors will cover in OChem 1 and OChem 2 to allow for students to change instructors from one class to the next. However, the way in which the material is taught can widely differ. As reported on in Chapter V and IV, there are two primary manners of instruction, the first being Traditional, which uses a published textbook and functional groups to guide the instructional sequence, and the second being Organic Chemistry, Life, the Universe, and Everything (OCLUE), which uses a transformed organic curriculum.23 152 Thus, not only are there many different instructors for the organic sequence at the institution, but there are two different approaches of teaching the material to students. To compound this, students also sign up to take courses long before the instructor is typically listed for a given section, which can also contribute to students having different instructors between OChem 1 and OChem 2, as well as, the differences between the Traditional and Transformed teaching environment. Although there is a number of paths students can take through the organic chemistry sequence, there were only two instructor paths that were consistent through the 3 academic years in this study. These paths are described as (OChem 1-OChem 2), which indicates the instructor they had in the fall for OChem 1 and then the instructor they had for OChem 2. During the 3 years of the study there were only two instructor combinations (Appendix A for full list of combinations) that we were able to collect responses for at each time point, this was Instructor A-Instructor A and Instructor A-Instructor B. However, in the Fall 2019-Spring 2020 academic year, there were only 22 students who took the Instructor A-Instructor B path and only 15 completed the assessment. Thus, the scope of this research was narrowed to only students whose organic sequence path was Instructor A-Instructor A. This instructor taught using the transformed curriculum OCLUE. While an in depth description of the courses and the cultures has previously been published23 a brief description is warranted here as some elements were changed for online instruction due to the Covid-19 pandemic. The OCLUE course consists of an open access textbook written for the OCLUE curriculum,24 required online homework problems using beSocratic,25 required recitation attendance, instruction is via synchronous in person lectures (pre-Covid-19) or asynchronous video lectures (during Covid-19), and exams. During Covid-19 students were sent prerecorded video lectures and an accompanying homework set to complete before the next synchronous homework review session. The homework was administered via beSocratic, an online homework platform, which allows open-ended drawn and written responses. In these sessions the instructor would hold an online class that would 153 cover the answer for the homework set, as well as, answer any students’ questions about the asynchronous lecture videos. Students’ beSocratic homework was graded only for completion, rather than correctness, to encourage students to share what knowledge they had and what they could do with that knowledge. The same philosophy was also applied to students’ recitation sections where students worked in groups to complete worksheets that corresponded with the lecture videos for the week and again were graded on students’ effort instead of correctness. The beSocratic homework and recitation comprised 40% of students’ overall grades and predominantly focused on constructing explanations, developing and using models, and engaging in arguments from evidence. These formative assessments are intentionally designed to support students making connections between the ideas within the course rather than test their knowledge which is reserved for the summative exams in the course. Prompt Timing and Administration Data Collection The different versions of the prompt were administered using beSocratic25 approximately one week before the final class day of OChem 2, with the exception of version 3, which was administered approximately one week before the final class day of OChem 1. Students were given a small amount of extra credit for completing the prompts. Each version of the prompt was multiple slides long, which meant students would see one question and, after constructing a response, they would click next to see the next related question. Table 2 shows the time the prompt was administered, the number of students who were sent the prompt, and the response rate for each prompt. 154 Table 7.1. Summary of data collection and response rate. Prompt Total How many students Responses Response Time Point Version students were solicited analyzed rate End OChem 2 1a 66 65 98% 135 Spring 2020 1b 69 67 97% End OChem 2 2a 99 63 64% 192 Spring 2021 2b 93 54 58% End OChem 1 3a 68 55 81% 154 Fall 2021 3b 86 73 85% End OChem 2 4a 68 41 60% 154 Spring 2022 4b 86 39 45% Mitigating Bias While this chapter will only be discussing the students who had Instructor A for the entire OChem sequence, there were other groups of students collected from other instructors. To reduce bias in coding, these other students’ responses were also included when coding, and all responses were deidentified from their course type and coded anonymously. In the Fall 2021-Spring 2022 academic school year, because only instructor A consented to contacting their students, these students’ drawings and explanations were mixed in with the previous year’s responses when coded to again support anonymity of their responses. Data Analysis To begin coding, student responses were removed from the beSocratic system and compiled such that a student’s entire response (from each section of the question) was viewed as a whole. I began by open coding the responses to capture the different resources students were using while answering the question. For students’ drawings, coding where students started and ended their arrows as individual parts of their mechanisms, this is in contrast to previous approaches where students’ arrows were coded as correct or incorrect as a whole. For students’ explanations, I captured the various resources students used when responding to the question, rather than comparing directly to the CMR framework. This was done to identify other patterns of explanations students might be using to answer the question. After these initial passes of coding were complete, I then evaluated the drawing with 155 respect to their complete mechanism and product. The explanations were then viewed through the lens of causal mechanistic reasoning, that is, whether students explained the phenomenon by describing how the particular components of a system give rise to its behavior.26 After the initial coding was done by myself (SH) and two trained high school science teachers and post baccalaureate researchers (JS and RM) blind-coded the anonymized student responses. Sets of 20 student responses were coded and discussed until agreement was reached, making slight changes to the coding scheme for both the drawing and explanations along the way. Finally, drawings were coded by both myself and JS and explanations were coded by myself, JS, and RM. I then coded the remaining responses for the prompt. Cohen’s Kappa values for the drawing tasks were between 0.806 and 0.944, and kappa values for the explanation tasks were between 0.773-0.793. Kappa values above 0.6 indicate substantial agreement and above 0.8 indicate almost perfect agreement found in Appendix A. Covid-19 One limitation of this work is that all data collected and presented in this study were collected during the global Covid-19 pandemic. The pandemic and its impact on education cannot be understated, and it will indeed take time to understand the full effect this has had on students. Though it is not in the scope of this research study, there is emerging literature noting how student learning was impacted during this time. A review of laboratory literature during the pandemic found many studies reported student difficulties in generating evidence-supported conclusions, predicting experimental outcomes, and evaluating and analyzing data.27 Another study surveying general and organic chemistry students, found that many students decreased time spent with student-organized study groups, peer-led team learning groups, and review sessions.28 Additionally, the organic students saw the highest increase in feelings of loss of focus and productivity. Students expressed they felt that they had lost learning momentum compared to before the Covid-19 lockdown when they were in person.28 In fact, at the institution in the current study, a survey that has been conducted every other year since the year 2000 156 published a report in 2021 outlining the pandemic’s influences on academic success.29 In this survey, 70% of students reported feeling tremendous/more than average stress while in the four previous surveys, 50% of students felt this way.29 Additionally, 55% of respondents said that their academics were traumatic or very difficult in the last year, while it was 45% the previous three years.29 The reason for mentioning this limitation in the methods is to acknowledge that there may be more occurring in the following results than can be explained by the discussion and current literature base. Students and instructors experienced a vastly different world than in previous years, and most of the past three school years were anything but “business as usual”. This also may be why the response rate for this task (that was worth a small amount of extra credit) was much lower during the Covid-19 pandemic. That being said, it is still necessary to engage with students regardless of circumstance and thus, we shall continue to build on the understanding of undergraduate chemical education within the given context. Results and Discussion Heterocyclic Ring Prompt Development This section will introduce each iteration of the prompt and analysis of student responses. In each revision of the prompt there will always be a version a and b which represent the two similar, but different versions of each prompt that were given out to students. Table 7.3 shows a description of each iteration of the prompt, the important findings from each version, and the changes that were made to the prompt to address these findings. The prompts will be discussed in the subsequent chapters however, a brief description will be given to guide the in-depth exploration of each of these findings and changes. These prompts all focused on the same reaction shown above in Figure 1. 1. Version 1 was designed to elicit students’ understanding of charge and relate that to the intramolecular reaction. 2. Version 2 was designed to elicit students’ understanding of charge and connect their mechanism to an electrostatic explanation 157 3. Version 3 was designed to elicit students’ understanding of an analogous SN2 reaction and to make connections to the intramolecular reaction. 4. Version 4 was designed to elicit students’ understanding of an analogous SN2 reaction while explicitly telling the students that the intramolecular reaction occurred similarly to the SN2. The prompt was also designed to elicit students’ explanations for the intramolecular reaction. 158 Table 7.2. Summary of prompt revisions that will be discussed. How the prompt was changed for Time Point Prompt Prompt description Findings the next version Version 1a 1) Asked students to draw the partial charges on the molecule 1) Students drew mechanistic arrows n = 65 2) Asked students to draw the mechanism of the reaction starting at the oxygen and ending on 1) The sodium counterion was removed End OChem 2 the sodium counterion or the carbon 2) Asked students to predict a product Spring 2020 Version 1b bonded to oxygen 3) A written explanation portion was 1) Asked students to draw the mechanism of the reaction 2) Some students drew arrows but did added n = 67 not draw a product 1) Asked students to draw the partial charges on the molecule 1) Students drew arrows starting on the Version 2a 2) Asked students to draw the mechanism and predict the product of the oxygen and ending on the adjacent n = 63 1) The additional question about charge reaction and explain their reasoning carbon much more frequently than in was removed End OChem 2 version 1a and 1b 2) An analogous SN2 reaction was Spring 2021 2) Students were not cued into the Version 2b 1) Asked students to draw the mechanism and predict the product of the included in place of the charge implicit hydrogens in the structure n = 54 reaction and explain their reasoning question 3) Students’ explanations focus on charge and stability 1) Asked students to draw the mechanism and predict the product of an 1) Half of the students were given full Version 3a SN2 reaction and explain their reasoning 1) Students who received the SN2 Lewis Structures and the other half n = 55 2) Asked students to draw the mechanism and predict the product of the reaction drew the correct product given line structures End OChem 1 intramolecular reaction and explain their reasoning more frequently that those who did 2) Students were asked to number their Fall 2021 not carbons in all reactions Version 3b 1) Asked students to draw the mechanism and predict the product of the 2) Students were still not cued into the 3) Students were explicitly told the n = 73 intramolecular reaction and explain their reasoning implicit hydrogens in the structure intramolecular reaction reacted with itself (All Lewis structure) 1) Asked students to number their carbons and to draw the mechanism and predict the product of an SN2 reaction and explain their reasoning Version 4a 1) Students who were given line 2) Ask students to explain why the intramolecular reaction reacts n = 41 structures were more successful at similarly to the SN2 reaction 3) Asked students to number their carbons and to draw the mechanism drawing the mechanism and product and predict the product of an SN2 reaction and explain their reasoning for the intramolecular reaction than End OChem 2 those who received a Lewis structure Spring 2022 (All line structures) 2) More students identified the role of 1) Asked students to number their carbons and to draw the mechanism electrons, charges, and attractions in and predict the product of an SN2 reaction and explain their reasoning their explanations of the Version 4b 2) Ask students to explain why the intramolecular reaction reacts n = 39 intramolecular reaction similarly to the SN2 reaction 3) Asked students to number their carbons and to draw the mechanism and predict the product of an SN2 reaction and explain their reasoning 159 Version 1: Focus on Charge The first iteration of the prompt focused on engaging students with charge distribution in the molecule. In the context of an intramolecular reaction, the initial knowledge statements included understanding charge distribution, because this is necessary to predict where the molecule will interact with itself. From here, the knowledge statements built from knowing the impact of charge distribution to predicting the outcome of an intramolecular reaction shown in Table 1. Thus, a prompt was created where half of the students were given additional questions about where the charges were in the molecule, and the other half of the students were only asked to predict the outcome of the reaction, shown in Figure 2. Giving half of the students the additional scaffolding (version 1a) was an opportunity to see if this activated the resource of charge and if they used this to help them draw a plausible mechanism for this reaction, compared to students who did not have this additional scaffolding (version 1b). Figure 7.2. Example student responses for prompt version 1a and 1b. 160 While coding, we did use the knowledge and evidence statements to guide our evaluation of student responses; however, we also maintained an open code book to give room for additional codes if there were patterns recognized in the data. This gave us the opportunity to see what resources students were leveraging to answer the question, without focusing solely on the knowledge and evidence statements. We began to explore the types of arrows and products students were drawing for both versions of the prompt, and rather than only focusing on the correct arrows students drew, we coded where students’ arrows started and ended to better understand how they were using mechanistic arrows. For example, we noted whether students were starting their arrows on the lone pairs on oxygen or the sodium counterion. Additionally, we noted whether students were ending their arrows on the carbon bonded to bromine, the bond between carbon and oxygen, the carbon bonded to oxygen, or the sodium counterion; examples are shown in Table 3 below. We found that 17% (N = 11) of the students with the additional scaffold (version 1a) and 18% (N=12) of students without the scaffold (version 1b) were ending their arrow at the sodium counterion. Additionally, 17% (N=11) of version 1a and 22% (N=15) of version 1b students drew arrows starting on the electrons on oxygen and ending on the carbon – resulting in a carbonyl group. In both of these instances, students started their arrows correctly on the oxygen lone pairs but ended their arrows on two places that are positive, but incorrect. The sodium counterion was depicted with its positive formal charge, and students may have leveraged the idea that the negative oxygen is attracted to a positive charge; however, they misapplied this idea by ending their arrow on the sodium. When students started their arrows on the oxygen and ended them on the partial positive carbon to form a carbonyl, students may again be leveraging the idea of negative attracts positive, but students seemingly ignored the hydrogen that is implicit in the line structure, resulting in an incorrect use of the mechanistic arrow. Arrows ending on either the sodium or the carbon bonded to oxygen, accounted for roughly 40% of mechanisms for this reaction by both versions 1a 161 (N=22) and 1b (N=27). To potentially mitigate students drawing their mechanism involving the sodium, we removed the counterion in the next iteration of the prompt. Table 7.3. Initial coding scheme for where students started or ended their arrows. Where the arrow starts Name Description Student Example Starts On Oxygen Starts an arrow on an oxygen Starts on Na+ Starts on the Na+ Where the arrow ends Name Description Student Example Arrow ends on the carbon Ends on C-Br bonded to the bromine Arrow ends on the carbon Ends on C-O bonded to the oxygen (or the bond between them) Ends on Na+ Arrow ends on the Na+ Ends on Oxygen Arrow ends on the Oxygen Arrow shows the bond breaking Second Arrow between the bromine and carbon We then evaluated students’ predicted products, from which several patterns emerged. Shown in Table 4, student products included the correct product, some type of heterocyclic ring, a ring structure, a structure with a carbonyl, a non-normative product, or no product. We were puzzled by the students who drew arrows for the reaction but then drew no product, the “no product” group. However, upon further reflection, we had only asked the students to “Draw the arrow pushing mechanism for this reaction”. While we assumed that asking students to draw the arrow pushing 162 mechanism meant to also draw the product, the lack of explicit mention in the question resulted in some students drawing a mechanism, but not a product. Shown in Figure 3, 14% (N=9) of students who received version 1a and 13% (N=9) of students who received version 1b drew arrows without a final product (dark blue). This is addressed in future iterations of this prompt to explicitly ask students to predict a product for their mechanism. Additionally, 14% (N=9) of version 1a and 7% (N=5) of version 1b students drew a heterocyclic product that was missing a carbon(s), the “Any Heterocycle group”. We wanted to capture these students in their own distinct group, because a response that is close, but incorrect, means that the students have useful resources that they are leveraging, such as the negative oxygen being attracted to the positive carbon bonded to bromine but are not able to fully use their arrows to predict the correct product, or they may simply have counted the number of carbons in the molecule incorrectly. Table 7.4. Coding scheme for the products draw by students. Product drawn Name Description Student Example Correct Product Correct Product drawn A ring that included oxygen as Any Heterocycle part of the main ring structure Some type of ring that does Ring not have and Oxygen within the ring Carbonyl Has a carbonyl Non-normative product that Non-Norm does not fit the above Normative categories Student drew arrows but then No Product drew no product 163 We also assessed where students in version 1a were drawing the partial charges on the molecule; the coding scheme is found in Table 5. We found that 63% (N=41) of students correctly drew a partial positive charge at the carbon bonded to bromine, 62% (N=40) of students correctly drew a partial positive charge on the carbon bonded to oxygen, and 57% (N=37) correctly identified both partial positives. However, when evaluating if drawing partial charges supported students drawing the correct arrows and correct product for this reaction, a different story emerged. Shown in Figure 3, we can see that there is little to no difference between the types of products students are drawing and the number of students drawing a correct product for this reaction between versions 1a and 1b. This indicates that adding the additional question about identifying charge, did not impact students’ ability to draw the correct product for this version of the prompt and possibly for this reaction. Table 7.5. Coding scheme for stundets drawn partial charges. Charges Drawn Name Description Student Example The partial charge being referred to in each row is highlighted in red Student draws a partial Partial Positive C-O positive symbol near the carbon bonded to oxygen Student draws a partial Partial Positive C-Br positive symbol near the carbon bonded to bromine Student draws a partial Partial negative Br negative symbol near the bromine Finally, we looked at the student responses as a whole to see how students were using their arrows to predict their product. Students who predicted the correct product 29% (N=19) of version 1a and 27% (N=18) of version 1b were also likely to draw all arrows correctly and drew the correct product at the same rates, 26% (N=17) and 24% (N=16) respectively. However, because roughly 25% of students were successful in drawing all plausible arrows and product for this reaction, there may be more ways to 164 elicit what students know about this type of reaction with a different prompt structure. Interestingly, of the students who did draw the correct product, 53% (N=10/19) of version 1a and 50% (N=9/18) of version 1b numbered the atoms in their molecule unprompted. This is in contrast with the students in the Any Heterocycle group, where only one student from each version numbered their atoms unprompted. Thus, prompting students to number their atoms may serve as one way to support students with this reaction mechanism, which is addressed in a later version of this prompt. Figure 7.3. Distribution of students drawn product for prompt version 1a and 1b. Version 1a and 1b presented an opportunity to reevaluate what we were asking the students. The overabundance of students starting at the negative oxygen but then ending at the sodium led us to believe that this positive sodium was leading students to incorrectly focus on that positive charge. Additionally, because so many students were drawing their arrows ending at the sodium or carbon 165 bonded to oxygen, we were unsure if the scaffolding on charge could help the students respond to the question or if it was masked by the high volume of students drawing the incorrect ending point of their arrow. To better understand students’ reasoning, we asked for a written explanation in the next iteration of the prompt. Finally, the increased number of students who were drawing their arrows but not drawing a product, also led us to believe we needed to be more explicit with our question if we wanted students to draw a product. Version 2: Explicit Wording and removal of the Counterion Based on the students’ responses to the first version of the prompt, changes were made to the prompt to better elicit students’ understanding of this reaction, Figure 4 shows the new version of the prompt. The first we modified the question to say, “Draw the arrow pushing mechanism for this reaction and predict the product.” Next, the sodium counterion was removed to reduce the extraneous information that may have been resulting in misapplied resources from students in the previous version. Finally, students were asked to explain the sequence of events for this reaction, and in a separate box, explain why they drew their arrow(s) as indicated. This was done to better elicit evidence of students’ reasoning about the reaction and if it aligned with what they drew. This prompt was also sent to students at the end of OChem 2 in 2021, like version 1, and again half of the students were asked to draw the partial charges before they were asked to draw their mechanism and explain the reaction 166 Figure 7.4. Example student responses for prompt version 2a and 2b. Evaluating these responses was done in a similar fashion to version 1, where students responses were coded based on where they began and ended their arrows. No new drawing codes were added because in version 2 of the prompt the establish coding scheme was sufficient for capturing students drawing responses. We also evaluated student explanations and compiled a coding scheme that focused on electrostatics and CMR while also capturing the additional resources students referred to in their explanations which is shown in Table 7. 167 Table 7.6. Coding scheme for students written explanations. Electrostatic Reasoning Name Description Student Example 1) Since the electrons on O are negatively charged, it is attracted to the partially positive C-Br. Student explains correctly Causal the negative oxygen being 2) This reaction occurs because the oxygen will be attracted to the carbon Mechanistic attracted (attack) the attached to the Br because the Br has an electronegative charge that will positive carbon. pull electrons from the carbon it is attached to which will make the carbon have a partial positive charge which is what makes the oxygen attracted to that Student explains correctly 1) The negative oxygen is attracted to the positive carbon the negative oxygen being Causal attracted (attack) the 2) The negative charge on the oxygen will then attack the partial positive positive carbon. does not carbon and force the Br- to act as a leaving group and get kicked off of the mention e- molecule. Student describes the 1) The oxygen attacks the carbon and kicks out bromine correct sequence of event Mechanistic and does not mention (Descriptive) 2) Oxygens lone pair attacks the C-Br and kicks out the Br and forms a charges of the atoms cyclo structure. The Br takes the lone pair and leaves when the O attacks involved Students is off topic or 1) The negative bromine is the leaving group and leaves for the carbon Non-Normative unclear in their explanation bond because there is resonance occurring. Additional Reasoning Name Description Student Example 1) The electron on O- was unstable, the negative oxygen is less stable Student explicitly refers to 2) Because there is a negative charge on the oxygen atom which will move the molecule being Stability to stabilize the atom. more/less stable as the cause of the reaction. 3) The negative charge wants to become more stable so it pushes itself onto the C-O bond to form a double bond and thus a more stable product. 1) …this can not hold any more electrons so they keep getting pushed by resonance down the line until it is able to get rid of the br as a leaving Student explicitly refers to Resonance group. resonance when explaining Explanation their arrow movement 2) The resonance is done to stabilize the oxygen because it has a negative charge and too many lone pairs. When evaluating students’ use of arrows, one drastic difference was apparent: the number of students who started their first arrow on oxygen and ended their arrow on the bond between carbon and oxygen to produce a carbonyl group greatly increase for both version 2a and 2b compared to version 1. With the revised version 2, 70% (N=44) of version 2a and 43% (N=23) of version 2b students drew their first arrow from the oxygen to the carbon to make a carbonyl. The student explanations gave us insight into why so many students took this approach. Many of these students who drew a carbonyl referenced stability as their primary reason for their mechanistic arrows, 64% (N=11/17) of version 2a 168 and 65% (N=18/28) of version 2b. Bryant’s example response is shown in Figure 5. It appeared that many students recognized that the oxygen is unstable or reactive, and referenced this in their explanations, however, like the students in version 1, they did not recognize the implicit hydrogen on the line structure. Figure 7.5. Bryant’s example response for prompt version 2a. Additionally, one group of students drew what we are describing as “resonance like arrows”, which resemble arrows moving down a molecule like resonance; however, it is not a plausible sequence of electron movement. Figure 6 shows part of Edgar’s response to version 2b where they explicitly refer to the arrows they have drawn as resonance arrows. Version 2a resulted in 17% (N=11) of students drawing and explaining these “resonance like arrows” while 9% (N=5) of version 2b did the same. This again points to the likelihood that students were not aware of or inferring the implicit hydrogen in the line structure when engaging with version 2. 169 Figure 7.6. Edgar’s example response for prompt version 2b. In version 2a, 67% (N=42) of students correctly drew a partial positive charge at the carbon bonded to bromine, and 75% (N=47) of students correctly drew a partial positive charge on the carbon bonded to oxygen. Students were readily able to identify the partial charges on the molecule; however, as researchers we may have cued them in to the wrong partial positive necessary for the intramolecular reaction. Furthermore, we had removed the positive sodium ion, but this seems to have only removed a distraction rather than support students focusing on the correct partial positive carbon bonded to bromine. There are many more carbonyl products drawn than any other category, particularly for version 2a students. One improvement was that far fewer students in version 2 drew no product after drawing arrows when compared to version 1. This also indicates that, if we want students to predict a product, we need to explicitly ask them to do so. However, while 16% (N=10) version 2a and 20% (N=11) version 2b drew a correct product, only 5% (N=3) of students from 2a and 15% (N=8) of students from 2b drew all arrows correctly and drew the correct product. 170 Figure 7.7. Distribution of students drawn product for prompt version 2a and 2b. The explanations students constructed helped to find ways to revise the prompt further. As mentioned above, many students leveraged the resources of stability and reactivity. Though it is reasonable that the oxygen is reactive, students incorrectly reacted this oxygen with other parts of the molecule. Beyond stability, some students did use electrostatics to justify why they drew their mechanism, 22% (N=12) version 2a and 14% (N=9) version 2b. These students correctly explained the negative oxygen being attracted to the positive carbon bonded to bromine. However, this finding is overshadowed by the abundance of students making a carbonyl and can serve as an area for improvement to understand how students’ reason about this reaction. These themes and responses indicated that substantial changes needed to be made to the prompt to better elicit students’ understanding. Version 2 of the prompt indicated (1) that the 171 additional scaffolding about charge may be doing more harm than good by activating inappropriate resources for students, (2) that students were not cued into the implicit hydrogens in the line structure, and (3) that students’ explanations did not invoke attraction between the oxygen and carbon bonded to bromine. Version 3: An Analogous Reaction Version 3 was administered at the end of the Fall 2021 semester rather than waiting until the end of spring 2022. This was done because of the substantial changes that were made to the prompt, to cue students more appropriately into the intramolecular nature of this reaction. We changed the prompt to better prime students to think about the types of reactivity within the molecule. To do this, we asked half of the students to also draw the arrow pushing mechanism and predict the product for a similar SN2 reaction in addition to the intramolecular reaction, shown in Figure 8. Students were also asked to explain their reasoning for both reactions to note any similarities or differences in their explanations. 172 Figure 7.8. Example student responses for prompt version 3a and 3b. With the adapted version 3, we saw a reduction in the number of students drawing an arrow from the oxygen to the adjacent carbon. Of the version 3a students, 22% (N=12) of students drew their first arrow from the oxygen to the carbon while 36% (N=26) of version 3b students did the same. This indicates that the additional SN2 scaffolding helped reduce the number of students drawing an arrow from the oxygen to the adjacent carbon, and there are far fewer students than the previous year drawing this arrow (70% version 2a and 43% version 2b). Version 3 was administered at the end of OChem 1 and observed 55% (N=30) of version 3a students drawing the correct product of the SN2 reaction. In turn, more students were drawing the correct product for the intramolecular reaction (35%, 173 N=19) than the students who did not receive the analogous SN2 reaction (18%, N=13) as shown in Figure 9. Additionally, 29% (N=16) of version 3a and 15% (N=11) of version 3b drew all arrows correctly and drew the correct product for this reaction. Considering this is a very difficult intramolecular reaction it is encouraging to see that even at the end of OChem 1 more students are starting to draw the correct arrows and product for the intramolecular reaction than either of the previous two versions that were administered at the end of OChem 2. Figure 7.9. Distribution of students drawn product for prompt version 3a and 3b. Version 3a of the prompt seemed to be activating more appropriate resources, as more students were drawing the correct arrows and correct product for the intramolecular reaction, and fewer students were attempting to form a carbonyl with the lone pair on the oxygen. That being said, there are still a fourth of version 3a students who did not recognize the implicit hydrogen in the line 174 structure (22% who formed carbonyls). Furthermore, only 29% (N=16) of Version 3a students drew all of their arrows correctly and formed a correct product, while 54% (N=30) of them drew the correct arrows and product for the analogous SN2 reaction. This made it unclear if students truly understand the attraction between the oxygen and the carbon bonded to bromine, or if the resource of resonance and carbonyl formation sits at the forefront of student’s minds, making it difficult to focus on other potential reactions. Version 4: Different Structural Representations Version 4 of the prompt was administered to students at the end of OChem 2 in the spring of 2022. The changes made in version 4 are: 1. version 4a is shown as the Lewis structure with all atoms written explicitly, while 4b is shown with the line structure. 2. The prompt first asks students to number the carbons before they draw the arrow pushing mechanism to predict the product for a similar SN2 mechanism. 3. students are shown the same SN2 mechanism they had just responded to and the structure of the intramolecular reactant and told that this reactant undergoes a reaction with itself. We then asked students to describe structural similarities between the SN2 reaction and the intramolecular reactant. This connection to the similar SN2 reaction was done in hopes to activate students’ ideas about oxygen as a nucleophile and carbon as an electrophile. The structures used in version 4a are included in Figure 1o, the full version 4b is shown in Figure 11, while the full version 4a is shown in Appendix B. Figure 7.10. This shows the Lewis structures that were used in the full version 4a prompt. The same wording was used as shown in Figure 10. The full Version 4a is in Appendix B. 175 Figure 7.11. Example student responses for prompt version 4b. There was little to no difference between version 4a and 4b and success at drawing the product for the similar SN2 reaction 71% (N= 29) and 79% (N=31), respectively, (the green bar in Figure 12). This is a similar observation to a study of a simple SN2 reaction with the same curriculum (pre covid), where 82% of those students correctly drew an SN2 mechanism and product.30 176 Figure 7.12. Distribution of students drawn product for prompt version 4a and 4b. However, differences emerged between version 4a and 4b for the intramolecular reaction. Students drawing an arrow from the oxygen to the adjacent carbon still occurred, with 24% (N=10) of version 4a and 15% (N=6) of version 4b drawing these types of arrows. Interestingly, a quarter of the students who were provided with the full Lewis Structure drew an arrow from the lone pair on oxygen to the adjacent carbon, even though the hydrogen was drawn explicitly for the students in the starting structure. Upon further investigation, only one of these students ignored the bonded hydrogen, while the rest of the students also fell into the group of “resonance like arrows”, an example is shown in Figure 13. These students acknowledged there was a hydrogen bonded to the carbon but continued using their arrows to “stop carbon from having more than four bonds [Emil version 4a]”. 177 Figure 7.13. Martyn’s example response for prompt version 4a. This disparity between versions 4a and 4b lead us to believe that the full Lewis structure may be adding too many details to the drawing. Rather than help students notice the hydrogen bonded to carbon, it led students to use these additional atoms and bonds in their erroneous mechanisms. This indicates that (1) explicitly telling the students the reaction occurs within the molecule and (2) the additional prompting with the SN2 reaction, focused the students on the electrophilic carbon bonded to bromine, rather than the electrophilic carbon bonded to oxygen. This, along with the fact that 37% (N=15) of the full Lewis structure students drew the correct heterocyclic product, while 64% (N=25) of the line structure students drew the correct heterocycle suggests that the full Lewis structure may have been a distraction to students rather than a support. 178 Figure 7.14. Distribution of students CMR for prompt version 4a and 4b. With this version of the prompt, more students identified the role of electrons, charges, and attractions in their explanations of the intramolecular reaction, 29% of version 4a and 41% of version 4b, shown in Figure 14. Additionally, 13% of version 4b students had causal responses and identified just an attraction between the oxygen and carbon. Though this version of the prompt explicitly told the students that an intramolecular reaction occurred, students were still able to explain why the reaction occurred, using their understanding of electrostatics. Furthermore, the students who responded to version 4b used electrostatics in their responses more often than version 4a, which again indicates that giving students the full Lewis structure may have caused a distraction when drawing and explaining their mechanism. 179 Summary of Prompt Versions Thus far we have established the various responses students have provided while iterating on the prompt over the course of 3 years. Figure 15 shows the combined Figures 3, 7, 9, and 12 intramolecular reaction product results to guide the discussion of student’s responses over time. From the first iteration of the prompt, we had envisioned using a scaffold about charge to see if it helps students. However, there was no difference between the products students drew for version 1a and 1b. We noted however that many students drew arrows but then no product, so we adjusted the prompt to better elicit students’ entire mechanism and product. We also had noted many students drawing arrows to the sodium counterion and thus was removed in the next iteration of the prompt. Version 2a and 2b saw improvement in the number of students drawing a product for the reaction however, many students began drawing the mechanism for a carbonyl for the reaction (light blue in Figure 15). This was troublesome because the students were not noticing or misunderstanding the role of the implicit hydrogens in the structure. Additionally, it seemed that if anything the additional scaffolding about charge was leading students astray from the correct interactions. Version 3 took a different approach to scaffolding and asked students about a similar SN2 reaction before the intramolecular reaction rather than asking students about charge. This approach led to more students drawing the correct arrows and product more often than the previous versions of the prompt however, there was still roughly a fifth of students who were successful at the SN2 mechanism but not successful at the intramolecular reaction. To address this a final iteration of the prompt was done to explicitly link the SN2 reaction to the intramolecular reaction for the students. This did support students using their arrows to predict their product more than the previous versions, shown in Table 8. Students also were able to explain how and why this reaction was occurring using CMR. This final iteration of the prompt also highlighted than using the line structures was more beneficial to the students than drawing a full Lewis structure. 180 Figure 7.15. Distribution of students drawn product for prompt versions 1-4. Table 7.7. Percent of students who drew all arrows correctly and correct product for prompt versions 1- 4. Prompt Total All correctly drawn arrows Time Point Version students and correct product End OChem 2 1a 65 26% (17/65) Spring 2020 1b 67 24% (16/67) End OChem 2 2a 63 5% (3/63) Spring 2021 2b 54 15% (8/54) End OChem 1 3a 55 29% (16/55) Fall 2021 3b 73 15% (11/73) End OChem 2 4a 41 34% (14/41) Spring 2022 4b 39 62% (24/39) Discussion The goal of this study was to build a prompt that was able to elicit students’ use of mechanistic arrows and reason about an intramolecular ring closure. To do this we reviewed student responses and then made changes to the design and scaffolding of the prompt to better elicit students’ resources 181 about the phenomena of an intramolecular reaction. Such changes may indeed be small; however, these changes can have large ramifications on the types of resources students’ leverage to answer the question. For example, we changed the wording from “Draw the arrow pushing mechanism for this reaction” to “Draw the arrow pushing mechanism for this reaction and predict the product” and found that more students drew the product for the reaction. While this may seem an obvious point it still emphasizes the fact that if we want students to show us what they know and can do we must explicitly ask them to do so. A second, seemingly small change, was to remove the sodium ion in version 2, this was done to remove extraneous information, however, as the responses showed us, this was undoubtedly a misstep on our part as researchers. Though students could no longer draw their mechanisms with the sodium counterion, they instead began to draw arrows from the oxygen to the adjacent carbon forming a carbonyl. We found that students could readily label the charges on the molecule however were drawing implausible attractions between other positive atoms. Student's explanations gave us insight as to why they had drawn these arrows; most students acknowledged that the oxygen was unstable, that there was a partial positive carbon bonded to oxygen, and by making a carbonyl the students would be stabilizing the molecule. When reflecting on this finding it makes sense that students would focus on the carbon bonded to oxygen because we had explicitly asked them to draw in the partial charges (in the case of version 2a), and we had removed the sodium counterion that was labeled with a formal charge of plus one. By removing this counterion we as researchers, changed the types of resources students were leveraging to solve this mechanism and rather than support students focusing on the attraction between the carbon bonded to bromine and the negative oxygen; we instead simply removed one option for students to choose to end their arrow. These finding suggest that students do indeed adapt to the type of scaffolding we present to them, and thus careful consideration must be made when updating or changing questions given to students. 182 In designing the 3rd and 4th version of this task the goal was shifted from removing what seemed like distractions for students, to instead making connections to an analogous SN2 reaction. By design the task this way we found that students who were given the additional SN2 reaction were better able to draw the appropriate mechanism for the reaction but were also able to explain how and why this reaction was occurring. Interestingly students who received a fully drawn Lewis structure correctly drew their mechanistic arrows and product for the similar SN2 reaction at the same frequency as students who were given only the line structures. However, the students who received the full Lewis structure drew correct arrows and product less frequently than those who received the line structure. This is a somewhat interesting observation, while some literature points to full or partial Lewis structures being distracting for students from extracting relevant information from a task,31 others found making some or all atoms explicit helped students draw their mechanisms.6 While this does warrant further exploration in a future study it is plausible that when given the entire Lewis structure students are overwhelmed with the possibilities shown and thus giving students partial line and partial Lewis structures may be more beneficial. By designing the intramolecular reaction around an analogous SN2 reaction we called forth the resources students likely already had about a reaction they are more familiar with and helped them to make the connection to the intramolecular reaction. One reason for this may be linked to both resource activation and what is known as the priming effect. The priming effect emerged in the early 90s in lexical psychology and is the facilitation of the processing of one stimuli (the intramolecular reaction) by presenting an individual with a stimuli they may be more familiar with (such as an SN2 reaction).32,33 By priming the students with a reaction they are familiar with, the students mental resources were activated within that context and were primed to be used in a different but related context. Version 4 of the prompt was also where students most frequently demonstrated their understanding of electrostatics. By asking students to explain how and why this reaction occurred we 183 found that many were able to leverage causal mechanistic reasoning in their explanations. Which was evidence for the knowledge we hoped to elicit from students at the outset of this investigation which was, (1) for students to be able to correctly draw the reaction mechanism and product and (2) explain why this reaction occurs at the molecular level. Designing task like the one described in this study are necessary if we want students to have more robust connections between their knowledge by providing students with opportunities to predict and explain their reasoning in words. There is growing evidence that suggests that students who are given ample opportunities to construct explanations, make and use models, or construct arguments from evidence can support students’ outcomes on topics such as drawing Lewis structures,34 using mechanistic arrows,3,4,21,35 relate a mechanisms path to energy,13 and constructing causal mechanistic accounts for various phenomena.20,22,30,36 Implications and Future Work This work was presented as a rich description and analysis of a prompt’s development cycle with the hope of describing in detail to others how and why decisions were made to adjust the wording and structure of a prompt about an intramolecular reaction. When constructing assessments to evaluate students’ understanding, subtle changes and prompt cues may have drastic impact on the responses students provide. It is important to use student responses to make informed decisions about the type of changes you make to a prompt. This paper hoped to establish one way of using CMR to evaluate student responses while also exploring the additional resources students used to answer the questions, such as stability, or energy. By using this prompt future administration with different groups of students, such as biochemistry students or other sections of organic chemistry, may lead to a better understanding of students reasoning. Given the circumstance surrounding the Fall 2019-Spring 2022 academic year the amount of scaffolding may be altered for future administrations of this prompt. It is often difficult to ascertain how much may be too much cueing for students, and when to remove the additional scaffolding37, thus it is important to iterate on the questions being asked of students. Furthermore, this 184 study was exploring students’ understanding of a complex reaction that requires many connections of knowledge from students, and thus warrants more investigation. Many students now take organic chemistry as a prerequisite for biochemistry courses which often include topics such as intramolecular ring closures of glucose and fructose and intramolecular aldol reactions in steroid formations. This research hopes to be a starting point for future investigations into students’ understanding of these types of complex reactions. Limitation While the limitations of administering research questions to students during the Covid-19 pandemic was stated in the above-mentioned section it is worth reiterating again here. The response rate dwindled from what has been historically strong response rates (typically >80%) from the same course, with the same instructor, for the same amount of extra credit.4,30,38 To better understand the effect the Covid-19 pandemic had on student responses this prompt may be administered again in the future. Furthermore, no statistically analysis were used to characterize this data because were took a mole holistic approach to understanding the student response. Also, because of the lower response rate and the number of prompt versions sent to students coincided with lower numbers of students in each group. For this reason, we deemed it inappropriate to perform statistical comparisons and instead use only descriptions of the students responses to provide an initial understanding of this phenomena and deter from false positive or false negatives that are prone to occur with low population numbers (low power).39 Finally, this discussion has only been surrounding the students who took OCLUE for both semester of organic chemistry, and though the responses of other instructors were included while coding to support anonymity, a greater investigation including other instructors and teaching environments is necessary to gain a larger scope of student understanding. 185 REFERENCES (1) Bhattacharyya, G.; Bodner, G. M. “It Gets Me to the Product”: How Students Propose Organic Mechanisms. J. Chem. Educ. 2005, 82 (9), 1402. https://doi.org/10.1021/ed082p1402. (2) Caspari, I.; Weinrich, M. L.; Sevian, H.; Graulich, N. This Mechanistic Step Is “Productive ”: Organic Chemistry Students’ Backward-Oriented Reasoning. Chem. Educ. Res. Pract. 2018, 19 (1), 42–59. https://doi.org/10.1039/C7RP00124J. (3) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844–849. https://doi.org/10.1021/ed2003934. (4) Houchlei, S. K.; Bloch, R. R.; Cooper, M. M. Mechanisms, Models, and Explanations: Analyzing the Mechanistic Paths Students Take to Reach a Product for Familiar and Unfamiliar Organic Reactions. J. Chem. Educ. 2021, 98 (9), 2751–2764. https://doi.org/10.1021/acs.jchemed.1c00099. (5) Galloway, K. R.; Leung, M. W.; Flynn, A. B. Patterns of Reactions: A Card Sort Task to Investigate Students’ Organization of Organic Chemistry Reactions. Chem. Educ. Res. Pract. 2019, 20 (1), 30– 52. https://doi.org/10.1039/C8RP00120K. (6) Flynn, A. B.; Featherstone, R. B. Language of Mechanisms: Exam Analysis Reveals Students’ Strengths, Strategies, and Errors When Using the Electron-Pushing Formalism (Curved Arrows) in New Reactions. Chem. Educ. Res. Pract. 2017, 18 (1), 64–77. https://doi.org/10.1039/C6RP00126B. (7) McClary, L.; Talanquer, V. College Chemistry Students’ Mental Models of Acids and Acid Strength. J. Res. Sci. Teach. 2011, 48 (4), 396–413. https://doi.org/10.1002/tea.20407. (8) Bhattacharyya, G. From Source to Sink: Mechanistic Reasoning Using the Electron-Pushing Formalism. J. Chem. Educ. 2013, 90 (10), 1282–1289. https://doi.org/10.1021/ed300765k. (9) Bhattacharyya, G. Trials and Tribulations: Student Approaches and Difficulties with Proposing Mechanisms Using the Electron-Pushing Formalism. Chem. Educ. Res. Pract. 2014, 15 (4), 594– 609. https://doi.org/10.1039/C3RP00127J. (10) Anderson, T. L.; Bodner, G. M. What Can We Do about ‘Parker’? A Case Study of a Good Student Who Didn’t ‘Get’ Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9 (2), 93–101. https://doi.org/10.1039/B806223B. (11) Popova, M.; Bretz, S. L. Organic Chemistry Students’ Challenges with Coherence Formation between Reactions and Reaction Coordinate Diagrams. Chem. Educ. Res. Pract. 2018, 19 (3), 732–745. https://doi.org/10.1039/C8RP00064F. (12) Popova, M.; Bretz, S. L. Organic Chemistry Students’ Interpretations of the Surface Features of Reaction Coordinate Diagrams. Chem. Educ. Res. Pract. 2018, 19 (3), 919–931. https://doi.org/10.1039/C8RP00063H. 186 (13) DeGlopper, K. S.; Schwarz, C. E.; Ellias, N. J.; Stowe, R. L. Impact of Assessment Emphasis on Organic Chemistry Students’ Explanations for an Alkene Addition Reaction. J. Chem. Educ. 2022, 99 (3), 1368–1382. https://doi.org/10.1021/acs.jchemed.1c01080. (14) Lieber, L.; Graulich, N. Thinking in Alternatives—A Task Design for Challenging Students’ Problem-Solving Approaches in Organic Chemistry. J. Chem. Educ. 2020, 97 (10), 3731–3738. https://doi.org/10.1021/acs.jchemed.0c00248. (15) Lieber, L.; Graulich, N. Investigating Students’ Argumentation When Judging the Plausibility of Alternative Reaction Pathways in Organic Chemistry. Chem. Educ. Res. Pract. 2022, 23 (1), 38– 54. https://doi.org/10.1039/D1RP00145K. (16) Eckhard, J.; Rodemer, M.; Langner, A.; Bernholt, S.; Graulich, N. Let’s Frame It Differently – Analysis of Instructors’ Mechanistic Explanations. Chem. Educ. Res. Pract. 2022, 23 (1), 78–99. https://doi.org/10.1039/D1RP00064K. (17) Bodé, N. E.; Deng, J. M.; Flynn, A. B. Getting Past the Rules and to the WHY: Causal Mechanistic Arguments When Judging the Plausibility of Organic Reaction Mechanisms. J. Chem. Educ. 2019, 96 (6), 1068–1082. https://doi.org/10.1021/acs.jchemed.8b00719. (18) Eckhard, J.; Rodemer, M.; Bernholt, S.; Graulich, N. What Do University Students Truly Learn When Watching Tutorial Videos in Organic Chemistry? An Exploratory Study Focusing on Mechanistic Reasoning. J. Chem. Educ. 2022, 99 (6), 2231–2244. https://doi.org/10.1021/acs.jchemed.2c00076. (19) Noyes, K.; Carlson, C. G.; Stoltzfus, J. R.; Schwarz, C. V.; Long, T. M.; Cooper, M. M. A Deep Look into Designing a Task and Coding Scheme through the Lens of Causal Mechanistic Reasoning. J. Chem. Educ. 2022, 99 (2), 874–885. https://doi.org/10.1021/acs.jchemed.1c00959. (20) Noyes, K.; Cooper, M. M. Investigating Student Understanding of London Dispersion Forces: A Longitudinal Study. J. Chem. Educ. 2019, 96 (9), 1821–1832. https://doi.org/10.1021/acs.jchemed.9b00455. (21) Crandell, O. M.; Kouyoumdjian, H.; Underwood, S. M.; Cooper, M. M. Reasoning about Reactions in Organic Chemistry: Starting It in General Chemistry. J. Chem. Educ. 2019, 96 (2), 213–226. https://doi.org/10.1021/acs.jchemed.8b00784. (22) Noyes, K.; McKay, R. L.; Neumann, M.; Haudek, K. C.; Cooper, M. M. Developing Computer Resources to Automate Analysis of Students’ Explanations of London Dispersion Forces. J. Chem. Educ. 2020, 97 (11), 3923–3936. https://doi.org/10.1021/acs.jchemed.0c00445. (23) Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96 (9), 1858–1872. https://doi.org/10.1021/acs.jchemed.9b00401. (24) Cooper, M. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe & Everything. (25) Bryfczynski, S. P. BeSocratic: An Intelligent Tutoring System for the Recognition, Evaluation, and Analysis of Free-Form Student Input. Ph.D. Dissertation, Clemson University, SC, 2010. 187 (26) Russ, R. S.; Scherr, R. E.; Hammer, D.; Mikeska, J. Recognizing Mechanistic Reasoning in Student Scientific Inquiry: A Framework for Discourse Analysis Developed from Philosophy of Science. Science Education 2008, 92 (3), 499–525. https://doi.org/10.1002/sce.20264. (27) Kelley, E. W. LAB Theory, HLAB Pedagogy, and Review of Laboratory Learning in Chemistry during the COVID-19 Pandemic. J. Chem. Educ. 2021, 98 (8), 2496–2517. https://doi.org/10.1021/acs.jchemed.1c00457. (28) Jeffery, K. A.; Bauer, C. F. Students’ Responses to Emergency Remote Online Teaching Reveal Critical Factors for All Teaching. J. Chem. Educ. 2020, 97 (9), 2472–2485. https://doi.org/10.1021/acs.jchemed.0c00736. (29) Martell, D.; Salerno, J. State of Spartan Health 2021 | Pandemic Influences on Academic Success. (30) Crandell, O. M.; Lockhart, M. A.; Cooper, M. M. Arrows on the Page Are Not a Good Gauge: Evidence for the Importance of Causal Mechanistic Explanations about Nucleophilic Substitution in Organic Chemistry. J. Chem. Educ. 2020, 97 (2), 313–327. https://doi.org/10.1021/acs.jchemed.9b00815. (31) DeCocq, V.; Bhattacharyya, G. TMI (Too Much Information)! Effects of given Information on Organic Chemistry Students’ Approaches to Solving Mechanism Tasks. Chem. Educ. Res. Pract. 2019, 20 (1), 213–228. https://doi.org/10.1039/C8RP00214B. (32) Tulving, E.; Schacter, D. L. Priming and Human Memory Systems. Science 1990, 247 (4940), 301– 306. https://doi.org/10.1126/science.2296719. (33) Psychology of Priming; Hsu, N., Schütt, Z., Eds.; Perspectives on cognitive psychology; Nova Science Publishers: Hauppauge, N.Y, 2012. (34) Cooper, M. M.; Underwood, S. M.; Hilley, C. Z.; Klymkowsky, M. W. Development and Assessment of a Molecular Structure and Properties Learning Progression. J. Chem. Educ. 2012, 89 (11), 1351–1357. https://doi.org/10.1021/ed300083a. (35) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid–Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703–1712. https://doi.org/10.1021/acs.jchemed.6b00417. (36) Ralph, V. R.; Scharlott, L. J.; Schwarz, C. E.; Becker, N. M.; Stowe, R. L. Beyond Instructional Practices: Characterizing Learning Environments That Support Students in Explaining Chemical Phenomena. J Res Sci Teach 2022, tea.21746. https://doi.org/10.1002/tea.21746. (37) Wood, D.; Bruner, J. S.; Ross, G. THE ROLE OF TUTORING IN PROBLEM SOLVING *. Child Psychology Psychiatry 1976, 17 (2), 89–100. https://doi.org/10.1111/j.1469- 7610.1976.tb00381.x. (38) Bowen, R. S.; Flaherty, A. A.; Cooper, M. M. Investigating Student Perceptions of Transformational Intent and Classroom Culture in Organic Chemistry Courses. Chem. Educ. Res. Pract. 2022, 10.1039.D2RP00010E. https://doi.org/10.1039/D2RP00010E. 188 (39) Krzywinski, M.; Altman, N. Power and Sample Size. Nature Methods 2013, 10 (12), 1139–1140. https://doi.org/10.1038/nmeth.2738. 189 APPENDIX A: INSTRUCTOR PATHS FOR OCHEM 1- OCHEM 2 AND INTERRATER RELIABILITY Table 7.8. Instructor paths list OChem 1-OChem 2 (instructor descriptions below in table 7.10). Year 1 Possible instructor sequences Number of Fall 2019-OChem 1 Spring 2020- OChem 2 (OChem 1-OChem 2) Students in group ◊ Instructor A Instructor A Instructor A-Instructor A 135 ◊ Instructor C Instructor B Instructor C-Instructor A 86 ◊ Instructor B Instructor B Instructor B-Instructor A 22 ◊ Instructor E Instructor D Instructor E-Instructor A 46 ◊ Off Sequence- Instructor A 26 ◊ Instructor B-Instructor B 296 ◊ Instructor A-Instructor B 168 ◊ Instructor C-Instructor B 152 ◊ Instructor E-Instructor B 57 ◊ Off Sequence- Instructor B 37 ‡ Instructor A-Instructor D 17 ‡ Instructor C-Instructor D 41 ‡ Instructor B-Instructor D 8 ‡ Instructor E-Instructor D 34 ‡ Off Sequence- Instructor D 25 Year 2 Possible instructor sequences Number of Fall 2020-OChem 1 Spring 2021- OChem 2 (OChem 1-OChem 2) Students in group ◊ Instructor A Instructor A Instructor A-Instructor A 192 ◊ Instructor B Instructor D Instructor B-Instructor A 136 ◊ Instructor B Instructor D Instructor D-Instructor A 30 ◊ Instructor D Off Sequence- Instructor A 35 ◊ Instructor A-Instructor D 110 ◊ Instructor B-Instructor D 486 ◊ Instructor D-Instructor D 208 ◊ Off Sequence- Instructor D 35 Year 3 Possible instructor sequences Number of Fall 2021-OChem 1 Spring 2022- OChem 2 (OChem 1-OChem 2) Students in group ◊ Instructor A Instructor A Instructor A-Instructor A 154 ◊ Instructor C Instructor D Instructor C-Instructor A 35 ◊ Instructor B Instructor D Instructor B-Instructor A 139 ◊ Instructor B Instructor F Off Sequence- Instructor A 31 ‡ Instructor A-Instructor D 144 ‡ Instructor C-Instructor D 62 ‡ Instructor B-Instructor D 379 ‡ Off Sequence- Instructor D 54 ‡ Instructor A-Instructor F 9 ‡ Instructor C-Instructor F 21 ‡ Instructor B-Instructor F 28 ‡ Off Sequence- Instructor F 7 ◊ represents students who were sent consent forms and the prompt ‡ represents students who were not sent consent forms and the prompt 190 Table 7.9. Anonymized Instructor descriptions and course descriptions. Instructor Label Instructor Description Course Description Professor Instructor A OCLUE Research Faculty Senior Teaching Specialist Instructor B Traditional following textbook Undergraduate Director of Chemistry Professor from a science focused residential college within the university Instructor C OCLUE Research Faculty Fixed Term Teaching Lecturer Instructor D Traditional following textbook Recent PhD graduate Assistant Professor Instructor E Traditional following textbook Research Faculty Senior Teaching Specialist Instructor F Traditional following textbook Organic Lab Coordinator Table 7.10. Interrater Reliability for students’ mechanism drawing. Code (N=349) Kappa Start Arrow 0.940 End Arrow 0.944 Second Arrow 0.932 Product 0.886 Table 7.11. Interrater Reliability for students’ drawn charges. Code (N=151) Kappa Charge on C bonded to Oxygen 0.806 Charge on C bonded to Bromine 0.856 Table 7.12. Interrater Reliability for students’ drawn charges. Code (N=411) Kappa Reasoning 0.773 Stability 0.793 “Resonance Like” 0.776 191 APPENDIX B: FULL PROMPT DESCRIPTION Figure 7.16. Example student responses for prompt version 4a. 192 CHAPTER VIII – CONCLUSIONS, IMPLICATIONS, AND FUTURE DIRECTIONS Conclusions This dissertation explored how students in the transformed organic chemistry course called Organic Chemistry, Life, the Universe, and Everything (OCLUE)1, use and explain their mechanistic arrows for both familiar and more complex chemical reactions, and how they differ in their responses from students in a Traditional organic course. The data was collected from organic chemistry 1 (OChem 1) and organic chemistry 2 (OChem 2) students over the course of each academic year. Students’ organic chemistry course experience impacted their use of mechanistic arrows for both familiar and unfamiliar reactions Chapters IV and V characterized how OCLUE and Traditional organic chemistry students used mechanistic arrows to predict a product for both a familiar and unfamiliar reaction. We found little difference between the cohorts’ ability to draw a plausible product for a familiar reaction; however, differences emerged when exploring students’ use of mechanistic arrows. Initially we focused only on students who were enrolled in the same course type for both semesters, OCLUE-OCLUE and Traditional- Traditional. We found that OCLUE-OCLUE students drew plausible arrows and a plausible product more often than Traditional-Traditional students for both the familiar and unfamiliar reactions. We then expanded the investigation to students who may have switched instructors between semesters: OCLUE- Traditional and Traditional-OCLUE. We found that students who switched course type performed similarly to their peers in the course type into which they switched. That is, students who switched from OCLUE to Traditional drew less plausible arrows by the end of OChem 2. Students who switched from Traditional to OCLUE began to draw plausible mechanistic arrows more frequently than they previously had. 193 There is a strong positive association between using causal mechanistic reasoning and drawing plausible arrows for an unknown reaction Chapter VI investigated the relationship between students’ use of causal mechanistic reasoning (CMR) and use of mechanistic arrows. Following the characterization from the previous chapters, and in parallel with Dr. Olivia Crandell’s investigation of a simple SN2 reaction, we found that OCLUE-OCLUE students tend to use CMR and some plausible arrows more frequently than Traditional-Traditional students. We also found that there is an association between CMR and student use of some plausible arrows for an unknown reaction. That is, students that used CMR in their explanations of a simple SN2 reaction were more likely to draw some plausible arrows for a reaction that they had not seen before (to our knowledge), and therefore could not have memorized the mechanistic arrows. Furthermore, we investigated whether students’ general chemistry background (CLUE or a Traditional general chemistry) had any impact on students use of mechanistic arrows. We found students’ general chemistry background does not impact their use of CMR and use of some plausible arrows, indicating that, when students engage with more complex reactions they would not have encountered in general chemistry, students may not retain some of the ways of thinking they acquired in the CLUE transformed curriculum. Students require significant scaffolding to draw plausible mechanisms and construct explanations about an intramolecular ring closure Chapter VII explored OCLUE student responses to an intramolecular ring closure across multiple prompt iterations. We found that students were able to identify the charges on the intramolecular molecule but, without further prompting, were not able to make the connection to an intramolecular reaction. In fact, many students drew the formation of a carbonyl or drew arrows to the counterion, rather than the appropriate reaction. We found that for students who drew the incorrect mechanism, their explanations often focused on stabilizing the molecule and disregarded pentavalent carbons they may have drawn as a product. The prompt was revised to ask students an additional question about a similar SN2 reaction before they were asked to draw the mechanism for an intramolecular reaction. By 194 priming the students to think about the SN2 reaction, more students successfully drew the mechanism for an intramolecular reaction. These students were able to explain their reaction using electrostatics more often than the students in the previous iterations of the prompt. This study was intended to create and pilot a prompt about an intramolecular reaction that warrants future administrations to analyze the influence this scaffolding has on a wider body of students. Implications Course design and implementation can impact student understanding and use of knowledge Historically, organic chemistry curricula, as well as, other science courses, have been comprised of many superfluous and disconnected topics that often have left students with a disjointed understanding of the subject.1,2 OCLUE seeks to combat this by reorganizing the curriculum and focusing on the core ideas of the discipline to best prepare students for their future courses and careers.1 We propose that the differences observed between OCLUE and Traditional students in Chapters IV though VI may come as a consequence of this change in curricular design and activities that students engage in, namely: 1) Students enrolled in OCLUE are frequently asked to link their drawings to explanations, and 2) OCLUE students have many opportunities to construct these explanations without risk of a grade penalty for incorrect explanations. Asking students to explain in words what their mechanisms mean, shows instructors how students think about the reaction. By consistently asking students to explain their reasoning throughout the course, we posit that students are more likely to link the use of mechanistic arrows with what they mean and how they can be used. We believe many more students in OCLUE attempted to draw mechanisms for an unfamiliar reaction, because they were previously given many opportunities in the course to "test out" their knowledge without a grade penalty. By asking the students to frequently construct explanations, as well as, give them the space to explain without fear of penalty, we send the message to students that this is a valued practice that is necessary to be successful in the course. If students are able to succeed in a course (receive a good grade) by simply memorizing, then we should not be surprised if students will use this approach, regardless of whether or not the 195 instructor intends for them to memorize.3 Thus, to encourage students to think about the underlying principles of a mechanism, we must ask students to explain how and why reactions occur throughout the course on multiple types of assessment. Sustained educational transformation efforts are necessary to support students There have been many calls to change the current state of science classes in higher education3–5; however, the pace with which these changes have occurred is slow.6 In the context of these studies, students may experience different course environments throughout their education, where they may move from CLUE/OCLUE to Traditional or vice versa. I have presented evidence that shows students who take OCLUE for two semesters are better able to draw mechanisms and predict products for a familiar and unfamiliar reaction than students who take Traditional organic chemistry for two semesters. However, we found that students who take OCLUE for the first semester of OChem regress in the likelihood that they will use mechanistic arrows to predict a product when they switch to a Traditional section for the second semester. We might expect that students who have learned the skill of mechanistic arrows and how to use them in a predictive way would retain this ability; however, we found that fewer students who switch from OCLUE to Traditional use them to predict the product for either the familiar or unfamiliar reaction. Conversely, students who switched from Traditional to OCLUE were more likely to draw plausible arrows and products than students enrolled in two semesters of Traditional. This indicates that students adapt to the course culture and context (which emphasize what is valued and necessary to receive a good grade in the course) in which they are enrolled. If memorizing a product is sufficient for students to be successful, then that is what they will do. This finding also indicates that systemic change cannot be accomplished without a coordinated effort to improve learning outcomes for students. Regardless of the evidence that students are supported by an educational transformation, individual faculty are unlikely to find success creating systemic change. 196 Research on systemic change in higher education indicates that for successful transformation to occur, researchers, practitioners, and administrators need to be involved to continue the transformation.7–9 Iterative question design leads to better understanding of what students know and can do The way questions are presented to students can drastically change how they respond to a phenomenon. By asking students to draw a mechanism and explain in words, we can better understand the ways students think about a reaction and adjust how the question is asked to better elicit student understanding. In the case of Chapter VII, students focused on surface level features of the reactants, which aligns with previous literature.10–13 However, by introducing a more familiar analogous reaction, more students were able to engage with the question and demonstrate their understanding. This work showed that, just because a student wasn’t able to correctly answer a question, does not mean they do not understand the underlying principles; rather, the question may not have been asked in a way that elicited students’ complete understanding. Future Directions The studies in this dissertation explored how students use and explain mechanistic arrows across both familiar and complex reactions. However, there is still more work to be done exploring student reasoning about more complex reactions and the impact OCLUE has on student understanding of organic chemistry. While I explored the scaffolding required to elicit student reasoning about an intramolecular reaction, this was done with only one cohort of students; future iterations of the prompt may be necessary to understand how other cohorts of students would respond. Furthermore, as many students in the pre professional track take biochemistry after organic, it would be fruitful to explore how students with different course backgrounds reason about the more complex reactions in biochemical systems. Ideally, we would like to see that students are able to transfer their understanding of (relatively) simple reactions such as SN2 reactions, tautomerisms, and aldol and retro-aldol condensations, into biochemical systems such as glycolysis. 197 Additionally, students reference many ideas related to energy in their explanations, and this idea was often misused or unclear if students understood what energy meant in this context. Future investigations into student understanding can illuminate the different productive and unproductive ideas students have related to energy. 198 REFERENCES (1) Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96 (9), 1858–1872. https://doi.org/10.1021/acs.jchemed.9b00401. (2) Krajcik, J.; Merritt, J. Understanding A Framework for K−12 Science Education. The Science Teacher 2012, 5. (3) Bowen, R. S.; Flaherty, A. A.; Cooper, M. M. Investigating Student Perceptions of Transformational Intent and Classroom Culture in Organic Chemistry Courses. Chem. Educ. Res. Pract. 2022, 10.1039.D2RP00010E. https://doi.org/10.1039/D2RP00010E. (4) Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering; Singer, S. R., Nielsen, N., Schweingruber, H. A., Eds.; The National Academies Press: Washington, D.C, 2012. (5) PCAST, President’s Council of Advisors on Science and Technology. Engage To Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and Mathematics.; Presidential Report; President’s Council of Advisors on Science and Technology, 2010. (6) Helping Students Make Sense of the World Using Next Generation Science and Engineering Practices; National Science Teachers Association, 2017. https://doi.org/10.2505/9781938946042. (7) Stains, M.; Harshman, J.; Barker, M. K.; Chasteen, S. V.; Cole, R.; DeChenne-Peters, S. E.; Eagan, M. K.; Esson, J. M.; Knight, J. K.; Laski, F. A.; Levis-Fitzgerald, M.; Lee, C. J.; Lo, S. M.; McDonnell, L. M.; McKay, T. A.; Michelotti, N.; Musgrove, A.; Palmer, M. S.; Plank, K. M.; Rodela, T. M.; Sanders, E. R.; Schimpf, N. G.; Schulte, P. M.; Smith, M. K.; Stetzer, M.; Van Valkenburgh, B.; Vinson, E.; Weir, L. K.; Wendel, P. J.; Wheeler, L. B.; Young, A. M. Anatomy of STEM Teaching in North American Universities. Science 2018, 359 (6383), 1468–1470. https://doi.org/10.1126/science.aap8892. (8) Henderson, C.; Beach, A.; Finkelstein, N. Facilitating Change in Undergraduate STEM Instructional Practices: An Analytic Review of the Literature. J. Res. Sci. Teach. 2011, 48 (8), 952– 984. https://doi.org/10.1002/tea.20439. (9) Sturtevant, H.; Wheeler, L. The STEM Faculty Instructional Barriers and Identity Survey (FIBIS): Development and Exploratory Results. Int. J. STEM Educ. 2019, 6 (1), 35. https://doi.org/10.1186/s40594-019-0185-0. (10) Reinholz, D. L.; Apkarian, N. Four Frames for Systemic Change in STEM Departments. Int. J. STEM Educ. 2018, 5 (1), 3. https://doi.org/10.1186/s40594-018-0103-x. (11) Graulich, N. The Tip of the Iceberg in Organic Chemistry Classes: How Do Students Deal with the Invisible? Chem. Educ. Res. Pract. 2015, 16 (1), 9–21. https://doi.org/10.1039/C4RP00165F. (12) Graulich, N.; Bhattacharyya, G. Investigating Students’ Similarity Judgments in Organic Chemistry. Chem. Educ. Res. Pract. 2017, 18 (4), 774–784. https://doi.org/10.1039/C7RP00055C. 199 (13) Flynn, A. B.; Featherstone, R. B. Language of Mechanisms: Exam Analysis Reveals Students’ Strengths, Strategies, and Errors When Using the Electron-Pushing Formalism (Curved Arrows) in New Reactions. Chem. Educ. Res. Pract. 2017, 18 (1), 64–77. https://doi.org/10.1039/C6RP00126B. (14) Bhattacharyya, G.; Bodner, G. M. “It Gets Me to the Product”: How Students Propose Organic Mechanisms. J. Chem. Educ. 2005, 82 (9), 1402. https://doi.org/10.1021/ed082p1402. 200