INVESTIGATION OF STUDENTS’ CAUSAL MECHANISTIC REASONING IN UNDERGRADUATE ORGANIC CHEMISTRY By Olivia Marie Crandell A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Chemistry – Doctor of Philosophy 2020 INVESTIGATION OF STUDENTS’ CAUSAL MECHANISTIC REASONING IN UNDERGRADUATE ORGANIC ABSTRACT CHEMISTRY By Olivia Marie Crandell The undergraduate organic chemistry course is a prerequisite course for many students who plan to pursue careers in chemistry and chemical engineering. It also serves those students who wish to pursue professional careers in medicine, dentistry, and veterinary sciences. Previous research on student learning in organic chemistry shows that students struggle to understand ideas such as acid- base reactions and structure-property relationships which are foundational concepts on which more complex concepts are built. Furthermore, the typical organic chemistry course emphasizes students use of the electron-pushing formalism to represent how bonds are formed and broken in chemical reactions. Expert organic chemists use this formalism to represent predicted reaction mechanisms that explain the formation of products. Numerous studies have characterized student difficulties using electron-pushing mechanisms in an expert-like way as well as associating underlying chemical principle with the representations. We suggest that deep understanding of chemical reactions and their underlying chemical principles can be developed by engaging students in causal mechanistic explanation as part of a transformed organic chemistry course that emphasizes students using their knowledge of electrostatics, structure-property relationships, and energy to engage in explanation of chemical phenomena. Our goal is to engage students in as specific type of explanation called in casual mechanistic explanation which includes reasoning about the underlying causal factors in conjunction with the underlying entities and their activities that bring the phenomenon about. The studies reported here use a qualitative approach to elicit student’ written explanations and drawn reaction mechanisms for various chemical reactions. Students were sampled at multiple time points over the course of their two-semester organic course to investigate how student reasoning changes overtime. Students participants were enrolled in either the beforementioned transformed organic chemistry course or were enrolled in an untransformed course that we refer to as the traditional context. This traditional context served as a control group for which to compare possible changes in reasoning for students enrolled in the transformed course sequence. Findings suggest that student engagement in causal mechanistic reasoning varies depending on students’ general chemistry and organic chemistry course experience as well as the nature of the prompt eliciting the reasoning. Findings also suggest that students are generally capable of drawing mechanistic arrows that would generally be considered correct, however triangulating student reasoning with a detailed analysis of students’ drawings, we found that typical organic chemistry assessment items that lack a reasoning component may overestimate student understanding. Our investigations also revealed student difficulties invoking the correct nucleophilic substitution process for a given reaction. Students often invoked an SN1 mechanistic process incorrectly, despite their engagement in casual mechanistic reasoning. Implications of these findings for organic chemistry instruction and assessment are discussed along with implications for future research. I dedicate this dissertation to my loving fiancé Quinn Kemerer and my supportive parents, Scott and Elizabeth Crandell. Quinn comforted me when I had doubts about my desire to pursue graduate school, he drove me to my first day of entrance exams when I was too nervous to find my way, held my hand through every difficult hurdle and celebrated my every triumph. I am grateful for his patience, support, sacrifice and encouragement throughout this season of our lives. I would also like to dedicate this dissertation to my parents. Their love, encouragement, support, and presence for every important moment in my life gave me the confidence to pursue anything I wanted to be. I am and always will be fueled by their constant reassurance that everything will be “Okay” and no matter what, I am loved. I am so blessed to have a support system full of unconditional love. iv ACKNOWLEDGEMENTS A dissertation is not completed in isolation. My journey has been influenced and shaped by many profound advisors and colleagues. I am so grateful for their investment of their time and energy into my training and my future career. First, and most importantly, I would like to thank Dr. Melanie Cooper for her guidance, support, and understanding. My confidence has grown immensely because of her example and I am proud to have her in my corner. She expected the very best from me and invested the very best in me in return. I am proud to be included in her academic family tree. I would also like to thank my second reader, Dr. Lynmarie Posey for all of her guidance through my graduate career. Dr. Posey was present and available to assist with anything I needed or answer any questions I had not just at the end of every step along the way. I would also like to thank my other committee members, Dr. Joseph Krajcik and Dr. James McCusker for their guidance starting in my second year. I am grateful for the support and guidance of the numerous postdoctoral researchers who have crossed my path. First and foremost, Dr. Justin Carmel for his support in preparing my literature seminar and Second Year Oral Examination materials. He is a dear friend and a trusted role model I will continue to rely on as I begin my career. I would like to thank Dr. Sonia Underwood and Dr. Hovig Kouyoumdjian for their collaboration on my first publication. Their prior work laid the groundwork for the studies reported in this dissertation. Later in my career, I had the pleasure of working with Dr. Ryan Stowe, Dr. Kinsey Bain, Dr. Elizabeth Day, Dr. Ashling Flaherty, and Dr. Erin Duffy. I have gained so much by working alongside each of them and look forward to working with them in the future. I am thankful to my fellow graduate students for their mentorship, collaboration and friendship: Katie Paris Kohn, Oscar Judd, Chris Minter, Keenan Noyes, Samantha Houchlei, Sewwandi Abeywardana. I would also like to thank my undergraduate researchers with whom I have had the pleasure of working: Macy A. Lockhart and August Jarzambek. I would not have been able to analyze all this data without their help and feedback. I would v also like to thank Dr. Amy Pollock for her mentorship and friendship during my time at MSU. She has touched so many with her kind heart and encouraging spirit and I am lucky to be one of them. vi TABLE OF CONTENTS LIST OF TABLES ....................................................................................................................................... ix LIST OF FIGURES .................................................................................................................................... xiii CHAPTER I: INTRODUCTION ..................................................................................................................... 1 Study Goals and Research Questions ................................................................................................... 2 REFERENCES ............................................................................................................................................ 5 CHAPTER II: THEORETICAL FRAMEWORKS ............................................................................................... 8 Knowledge Construction: Information Processing Model ..................................................................... 8 Development of Expert Knowledge ...................................................................................................... 9 Changing Prior Knowledge: Conceptual Change and Intellectual Resources ....................................... 11 Heuristics and Dual-Processing .......................................................................................................... 19 Summary ........................................................................................................................................... 20 APPENDIX .............................................................................................................................................. 22 REFERENCES .......................................................................................................................................... 25 CHAPTER III: LITERATURE REVIEW ......................................................................................................... 28 Three-Dimensional Learning .............................................................................................................. 28 Causal Mechanistic Reasoning ........................................................................................................... 36 Mechanistic Reasoning in Organic Chemistry ..................................................................................... 37 Summary ........................................................................................................................................... 43 REFERENCES .......................................................................................................................................... 44 CHAPTER IV: REASONING ABOUT REACTIONS IN ORGANIC CHEMISTRY: STARTING IT IN GENERAL CHEMISTRY ........................................................................................................................................... 49 Preface .............................................................................................................................................. 49 Introduction ...................................................................................................................................... 49 Defining Causal Mechanistic Reasoning ............................................................................................. 51 Why Engage Students in Causal Mechanistic Reasoning in Organic Chemistry? .................................. 54 The Value of Longitudinal Studies in CER............................................................................................ 55 Polytomous Assessments Can Provide Longitudinal Information about Student Reasoning ................ 56 Methods ............................................................................................................................................ 57 Results ............................................................................................................................................... 68 Implications for Teaching and Further Research ................................................................................. 81 Limitations ......................................................................................................................................... 84 Future Work ...................................................................................................................................... 85 Final Thoughts ................................................................................................................................... 85 APPENDIX .............................................................................................................................................. 86 REFERENCES .......................................................................................................................................... 99 CHAPTER V: ARROWS ON THE PAGE ARE NOT A GOOD GAUGE: EVIDENCE FOR THE IMPORTANCE OF CAUSAL MECHANISTIC EXPLANATIONS ABOUT NUCLEOPHILIC SUBSTITUTION IN ORGANIC CHEMISTRY ............................................................................................................................................................ 104 vii Preface ............................................................................................................................................ 104 Introduction .................................................................................................................................... 104 Importance of scaffolding to activate resources ............................................................................... 105 Causal Mechanistic Reasoning in Organic Chemistry ........................................................................ 108 Research Questions ......................................................................................................................... 111 Methods .......................................................................................................................................... 111 Results and Discussion ..................................................................................................................... 128 Summary ......................................................................................................................................... 137 Implications for Instruction .............................................................................................................. 138 Implications for Research................................................................................................................. 139 Limitations ....................................................................................................................................... 140 APPENDIX ............................................................................................................................................ 142 REFERENCES ........................................................................................................................................ 154 CHAPTER VI: “WHAT ABOUT THE STUDENTS WHO SWITCHED COURSE TYPE?”: AN INVESTIGATION OF INCONSISTANT ORGANIC COURSE EXPERIENCE ................................................................................... 159 Introduction .................................................................................................................................... 159 Research Question ........................................................................................................................... 160 Methods .......................................................................................................................................... 161 Results ............................................................................................................................................. 165 Discussion and Conclusions.............................................................................................................. 171 Implications for Instruction .............................................................................................................. 172 Implications for Research................................................................................................................. 173 Limitations ....................................................................................................................................... 173 APPENDIX ............................................................................................................................................ 175 REFERENCES ........................................................................................................................................ 179 CHAPTER VII: THE EFFECT OF SCAFFOLDING ON CAUSAL MECHANISTIC REASONING ........................... 183 Introduction .................................................................................................................................... 183 Research Questions ......................................................................................................................... 185 Methods .......................................................................................................................................... 185 Results and Discussion ..................................................................................................................... 197 Summary ......................................................................................................................................... 209 Implications for Instruction .............................................................................................................. 210 Implications for Research................................................................................................................. 211 Limitations ....................................................................................................................................... 211 REFERENCES ........................................................................................................................................ 213 CHAPTER VIII: CONCLUSIONS, IMPLICATIONS, AND FUTURE WORK ..................................................... 215 Conclusions ..................................................................................................................................... 215 Implications for organic chemistry instruction and assessment ........................................................ 217 Future Work .................................................................................................................................... 221 REFERENCES ........................................................................................................................................ 222 viii LIST OF TABLES Table 4.1. Research Design Comparing Data from Students in Four Cohorts on Several Demographic and Academic Measures. ............................................................................................................................. 58 Table 4.2. Publisheda Characterization Scheme for the Reaction of HCl and H2O. ................................... 64 Table 4.3. Characterization Scheme for the Reaction of NH3 with BF3. ................................................... 67 Table 4.4. Comparison of Percentage of Correct Mechanistic Arrow Drawings for the Reaction of H2O + HCl. ....................................................................................................................................................... 72 Table 4.5. Comparison of Percentage of Correct Mechanistic Arrow Drawings for the Reaction of NH3 + BF3......................................................................................................................................................... 73 Table 4.6. Comparison of Percentage of Correct Mechanistic Arrow Drawings for the Reaction of H2O + HCl. ....................................................................................................................................................... 78 Table 4.7. Comparison of Percentage of Correct Mechanistic Arrow Drawings for the Reaction of NH3 + BF3......................................................................................................................................................... 79 Table 4.8. Published Characterization Scheme for Student Reasoning about the Reaction of HCl with H2O. .............................................................................................................................................................. 88 Table 4.9. Published Distribution of Students' Incorrect and Correct Mechanism Drawings and the Ratio of Correct to Incorrect Drawings by Each Type of Student Response for the reaction of HCl and H2O..... 89 Table 4.10. Characterization Scheme for the Reaction of NH3 with BF3. ................................................. 90 Table 4.11. Statistical analysis of academic measures for the four cohorts. ............................................ 91 Table 4.12. Chi-Square Analyses. ........................................................................................................... 92 Table 4.13. Chi-square analysis of student reasoning for HCl and H2O. ................................................... 94 Table 4.14. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for HCl + H2O........................ 94 Table 4.15. Chi-Square Analysis of Student Reasoning for HCl + H2O Comparing Non-Lewis Causal codes to Lewis Causal codes. ........................................................................................................................... 95 Table 4.16. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for HCl + H2O........................ 95 Table 4.17. Student Response Percentages for HCl and H2O................................................................... 95 Table 4.18. Chi-Square Analysis of Student Reasoning for NH3 + BF3 Comparing Non-Causal Mechanistic to Causal Mechanistic. ........................................................................................................................... 97 ix Table 4.19. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for NH3 + BF3. ....................... 97 Table 4.20. Chi-Square Analysis of Student Reasoning for NH3 + BF3 Comparing Non-Causal Mechanistic to Causal Mechanistic. ........................................................................................................................... 97 Table 4.21. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for NH3 + BF3. ....................... 97 Table 4.22. Student Response Percentages for NH3 + BF3. ...................................................................... 98 Table 5.1. Causal Mechanistic Characterization Scheme. ..................................................................... 123 Table 5.2. Summary of Tags Assigned to a Response When Warranted. ............................................... 124 Table 5.3. Classifications Used to Compare Explanations to Arrow Drawings. ...................................... 127 Table 5.4. Comparative Percentage of Students Whose Response Became Mechanistic after Explaining Their Drawn Mechanistic Arrows. ........................................................................................................ 130 Table 5.5. Chi-Square Comparisons of the Proportions of Non-Causal Mechanistic Responses versus Causal Mechanistic Responses. ............................................................................................................ 134 Table 5.6. Comparison between Reaction Process Explanation and Mechanistic Arrow Use as Characterized in Table 5.3. .................................................................................................................. 136 Table 5.7. Chi-Square Comparison of OCLUE and Traditional Cohorts. ................................................. 136 Table 5.8. Summary of course types during the three years of this study. Each section has ~300-360 students. ............................................................................................................................................. 144 Table 5.9. Year 1 Participants. .............................................................................................................. 144 Table 5.10. Year 1 Participants Descriptive Statistics. ........................................................................... 145 Table 5.11. Mann-Whitney Comparison of Year 1 – Original SN2 Prompt to Year 1 – Modified SN2 Prompt. ............................................................................................................................................................ 146 Table 5.12. Chi – Square Analysis of Gender for Year 1 Participants. .................................................... 146 Table 5.13. Students who had ACT (or SAT equivalent), GC1 course, GC2 course, OC1 course that was either Year 2 - OCLUE or Year 2 – Traditional, OC2 course that was either Year 2 – OCLUE or Year 2 – Traditional, took the Year 2 – Time Point 1 and the Year 2 – Time Point 2. ........................................... 146 Table 5.14. Year 2 Participants Descriptive Statistics. ........................................................................... 147 Table 5.15. Mann – Whitney Comparison of Year 2 – Traditional and Year 2 – OCLUE. ......................... 148 Table 5.16. Chi – Square Analysis of Gender for Year 2 Participants. .................................................... 148 x Table 5.17. Year 3 Participants Organic Chemistry Enrollment. ............................................................ 148 Table 5.18. Year 3 Participants Descriptive Statistics. ........................................................................... 149 Table 5.19. Mann – Whitney Comparison of Year 3 – Traditional and Year 3 – OCLUE. ......................... 150 Table 5.20. Chi – Square Analysis of Gender for Year 3 Participants. .................................................... 150 Table 5.21. Chi – Square Analysis for Year 3 Participants. ..................................................................... 151 Table 5.22. Chi – Square Analysis of Gender for Year 2 and Year 3 Participants. ................................... 151 Table 5.23. Analysis of terminology users and non-users. .................................................................... 153 Table 6.1. Summary of cohorts. ........................................................................................................... 163 Table 6.2. Summary of comparisons of academic measures. For brevity, only differences that were found to be significantly different are reported here. The full statistical outputs are reported in the Appendix. ............................................................................................................................................................ 164 Table 6.3. Distribution of reasoning characterizations for all cohorts at each relevant time point. NR-No Response, NN-Non-normative, DG-Descriptive General, DC-Descriptive Causal, DM-Descriptive Mechanistic, CM-Causal Mechanistic. .................................................................................................. 170 Table 6.4. Chi-Square tests of Homogeneity of the proportions between cohorts. A Bonferroni correction was used to determine significance to help reduce the chance of Type 1 error. a An α = 0.017 was used for the series of 3 pair-wise comparisons. b An α = 0.008 was used for the series of 6 pair-wise comparisons. ....................................................................................................................................... 171 Table 6.5. Descriptive statistics for the OCLUE-OCLUE, Traditional-Traditional, OCLUE-Traditional, and Traditional-OCLUE cohorts................................................................................................................... 176 Table 6.6. Non-parametric comparisons between OCLUE-OCLUE, Traditional-Traditional, OCLUE- Traditional, and Traditional-OCLUE cohorts. ........................................................................................ 177 Table 7.1. Causal Mechanistic Reasoning Characterization Scheme for the Reduced Scaffolding Prompt and Expanded Scaffolding Prompt. ...................................................................................................... 191 Table 7.2. Summary of tags assigned to an explanation response when warranted. These tags are assigned in addition to the Causal Mechanistic Characterization codes................................................ 192 Table 7.3. Classification scheme for non-static coding of mechanistic arrow drawings for the Reduced Scaffolding prompt with the intramolecular reaction of 6-bromohexan-2-olate. .................................. 193 Table 7.4. Distribution of reasoning characterizations at each time point for three types of prompt. NR- No Response, NN-Non-normative, DG-Descriptive General, DC-Descriptive Causal, DM-Descriptive Mechanistic, CM-Causal Mechanistic. .................................................................................................. 200 xi Table 7.5. Chi-Square test comparing the proportion of Descriptive Causal responses to other characterizations for the Reduced Scaffolding Prompt. All other characterizations include No Response, Non-Normative, Descriptive General, Descriptive Mechanistic and Causal Mechanistic. ...................... 200 Table 7.6. Reduced Scaffolding static arrows results. ........................................................................... 202 Table 7.7. Consistency of explanations and arrows for all three reaction prompts. .............................. 203 Table 7.8. Chi-Square comparisons of causal mechanistic responses verses non-causal mechanistic response for the Expanded Scaffolding prompt. .................................................................................. 204 Table 7.9. Percent of students who constructed consistent reasoning at the start and end of OC2 for each prompt type. ............................................................................................................................... 207 Table 7.10. Percent of students who constructed consistent reasoning for multiple prompts at a given time point. .......................................................................................................................................... 209 xii LIST OF FIGURES Figure 2.1. Information Processing Model. Reproduced with permission from ref 2. Copyright 1997 American Chemical Society. ..................................................................................................................... 9 Figure 2.2. The process for connecting structure and properties for a simple molecule. Reproduced with permission from ref 23. Copyright 2012 The Royal Society of Chemistry. ............................................... 19 Figure 2.3. Permissions to use Information Processing Model figure. ..................................................... 23 Figure 2.4. Permissions to use Structure/Property figure. ...................................................................... 24 Figure 3.1. A visual representation for how core ideas might theoretically overlap and how pieces of knowledge might be connected to those ideas. The core ideas of electrostatic and bonding interactions abbreviated as ES & B Interactions. ....................................................................................................... 31 Figure 3.2. List of Scientific and Engineering Practices Identified in A Framework for K-12 Science Education.1 ............................................................................................................................................ 32 Figure 3.3. 3D-LAP criteria required to elicit evidence of student engagement in the practice of Constructing Explanations and Engaging in Argumentation for Evidence.14 ............................................ 32 Figure 3.4. Sample organic chemistry assessment item that meets the criteria for the practice Constructing Explanations and Engaging in Argumentation from Evidence. ........................................... 33 Figure 3.5. Crosscutting Concepts identified in A Framework for K-12 Science Education.1 .................... 35 Figure 4.1. Assessment prompts administered using beSocratic for the reaction BF3 with NH3. An identical prompt structure was used for the reaction of HCl with H2O. .................................................. 61 Figure 4.2. The characterization of student explanations for HCl + H2O for Cohorts A, B, and C at the start and end of organic chemistry. Exact percentages are listed in S4 in the Supporting Information. No Response (NR), Non-Normative (NN), General Descriptive (GD), Brønsted Descriptive (BD), Brønsted Causal (BC), Lewis Mechanistic (LM), Lewis Causal Mechanistic (LCM). .................................................. 69 Figure 4.3. The characterization of student explanations for NH3 + BF3 for Cohorts A, B, and C, at the start and end of organic chemistry. Exact percentages are listed in S10 in the Supporting Information. No Response (NR), Non-Normative (NN), Descriptive General (DG), Descriptive Causal (DC), Descriptive Mechanistic (DM), Causal Mechanistic (CM). ......................................................................................... 70 Figure 4.4. The classification of student explanations for the reaction of H2O + HCl. These students were enrolled in a CLUE–GC2 course but were given the assessment item at different times. ........................ 72 Figure 4.5. The classification of student explanations for the reaction of NH3 + BF3. These students were enrolled in a CLUE–GC2 course but were given the assessment item at different times. ........................ 73 xiii Figure 4.6. The classification of student explanations for the reaction of H2O + HCl. These students had different GC2 experiences but were given the assessment item at the start of OC1 and the end of OC2.76 Figure 4.7. The classification of student explanations for the reaction of NH3 + BF3. These students had different GC2 experiences but were given the assessment item at the start of OC1 and the end of OC2.77 Figure 4.8. An example of a correct arrow drawing for the reaction of NH3 + BF3. .................................. 78 Figure 4.9. Permissions to reproduce manuscript in its entirety. ............................................................ 87 Figure 4.10. Comparison of the Cohorts A, B, and C for the reaction of HCl with H2O at the start of OC1. This figure shows the similar trends of Cohort B and Cohort C. Cohorts B and C were combined to simplify data visualization in the chapter. .............................................................................................. 93 Figure 4.11. Comparison of the Cohorts A, B, and C for the reaction of HCl with H2O at the start of OC1. This figure shows the similar trends of Cohort B and Cohort C. Cohorts B and C were combined to simplify data visualization in the manuscript. ........................................................................................ 94 Figure 4.12. Comparison of Cohorts A, B, and C at the Start of OC1 for NH3 and BF3............................... 96 Figure 4.13. Comparison of Cohorts A, B, and C at the End of OC1 for NH3 and BF3. ............................... 96 Figure 5.1. A: Original SN2 Prompt structure administered using beSocratic. B: Modified SN2 Prompt administered using beSocratic.33 ......................................................................................................... 112 Figure 5.2. Summary of data collections over the three years of this study. ......................................... 114 Figure 5.3. Example of a correct mechanistic arrow drawing (top) and an incorrect mechanistic arrow drawing (B). ......................................................................................................................................... 125 Figure 5.4. Comparison of causal mechanistic reasoning between the Original SN2 Prompt (A) and the Modified SN2 Prompt (B) at the end of Year 1 – Time Point 2. These characterizations for each prompt type are further separated by student use of polarity. No Response (NR), Non-Normative (NN), Descriptive General (DG), Descriptive Causal (DC), Descriptive Mechanistic (DM), Causal Mechanistic (CM). ................................................................................................................................................... 128 Figure 5.5. Distribution of Causal Mechanistic reasoning characterizations for OCLUE and Traditional cohorts for Year 2 – Time Point 1 (A), Year 2 – Time Point 2 (B), and Year 3 – Time Point 2 (C). NR=No Response, NN=Non-Normative, DG=Descriptive General, DC=Descriptive Causal, DM=Descriptive Mechanistic, CM=Causal Mechanistic. ................................................................................................. 131 Figure 5.6. Distribution of organic terminology use for Non-Causal Mechanistic and Causal Mechanistic responses. ........................................................................................................................................... 132 Figure 5.7. Explanations and arrow drawing comparison compared to Causal Mechanistic reasoning for Year 2 – Time Point 1 (A), Year 2 – Time Point 2 (B), and Year 3 – Time Point 2 (C). NR=No Response, NN=Non-Normative, DG=Descriptive General, DC=Descriptive Causal, DM=Descriptive Mechanistic, CM=Causal Mechanistic. ...................................................................................................................... 137 xiv Figure 5.8. Permissions to reproduce manuscript in its entirety. .......................................................... 143 Figure 5.9. Analysis of causal mechanistic reasoning with polarity tags at Y2 Time Point – 1. ............... 152 Figure 5.10. Analysis of causal mechanistic reasoning with polarity tags at Y2 Time Point – 2. ............. 152 Figure 5.11. Analysis of causal mechanistic reasoning with polarity tags at Y3 Time Point – 2. ............. 153 Figure 6.1. Characterization of Causal Mechanistic Reasoning for the reaction of CH3Br with OH-. For simplicity of representation, the No Response and Non-Normative bins are removed from this representation. The proportions for No Response and Non-Normative can be found in the Supplemental Information. DG-Descriptive General, DC-Descriptive Causal, DM-Descriptive Mechanistic, CM-Causal Mechanistic. ........................................................................................................................................ 168 Figure 7.1. Reduced scaffolding prompt administered via beSocratic9.................................................. 186 Figure 7.2. Structure of the Expanded Scaffolded Prompt administered across four slides via beSocratic.9 ............................................................................................................................................................ 188 Figure 7.3. Examples of correct static mechanistic arrow drawing (A) and incorrect static mechanistic arrow drawings (B and C) for the Reduced Scaffolding Prompt. ........................................................... 193 Figure 7.4. Examples of various mechanistic pathways observed in student drawings. These examples were recreated in ChemDraw for clarity with student work represented in blue. A: Example of canonically correct mechanistic pathway for the formation of t-butyl iodide. B: Example of canonically correct mechanistic pathway for the formation of an ether product. C: Example of response that omitted a final product. D. Example of response that did not include any mechanistic arrows. E: Example of response representing an SN2 process. F and G: Examples of non-normative responses. ..................... 196 Figure 7.5: Characterization of Causal Mechanistic Reasoning for the Reduced Scaffolding Prompt. NR- No Response, NN-Non-Normative, DG-Descriptive General, DC-Descriptive Causal, DM-Descriptive Mechanistic, CM-Causal Mechanistic. .................................................................................................. 199 Figure 7.6. Visualization of how data were analyzed to address RQ 3. .................................................. 206 Figure 7.7. Visualization of how data were analyzed to address RQ 4. .................................................. 208 xv CHAPTER I: INTRODUCTION A primary focus of the undergraduate organic chemistry course is developing students’ knowledge of chemical reactions. This includes the ability to represent how reactions proceed using curved arrows to represent electron movement called reaction mechanisms. There are numerous studies characterizing student difficulties drawing reaction mechanisms and their difficulties understanding how and why processes occur. Indeed, many of these studies suggest that the curved arrow notation does not function as a model of reactivity for students in the same way it does for experts.1-4 We suggest that organic chemistry students can develop a robust understanding of chemical reactivity by engaging in causal mechanistic reasoning – a specific type of explanation in which the underlying entities and causes are invoked to explain how an effect occurred.5 In the context of chemical reactivity, a causal mechanistic explanation draws on chemical principles to explain why atomic and molecular species interact (the cause) and gives a detailed account of electron movement that transforms reactants to products (the mechanism). Constructing an explanation of this nature requires students to draw on their knowledge of atomic and molecular structures and the electrostatic interactions that give rise to structure-property relationships. These concepts, among others, have been identified as core ideas of chemistry because they underpin a wide variety of chemical phenomena and are the “big ideas” around which experts build their knowledge.6 The act of pulling ideas together, reasoning with them and applying them to explain phenomena in various contexts is within itself a means to developing deep understanding.7 A Framework for K-12 Science Education developed a vision for science learning in which students develop expert-like knowledge structures grounded in core ideas and use their knowledge by engaging in 1 scientific practices, such as explanation to reason about crosscutting concepts of science such as patterns or cause and effect in a system.8 Cooper and Klymkowsky have developed a transformed general chemistry curriculum9 and subsequent organic chemistry curriculum10 that incorporates many elements of the Framework’s vision for how student knowledge should be developed and used over time. Chemistry, Life, the Universe and Everything9 (CLUE) and the subsequent Organic Chemistry, Life, the Universe and Everything10 (OCLUE) are transformed chemistry curricula designed to help students develop a connected framework of knowledge that can be used to predict and explain phenomena. Evidence supporting the efficacy of the design and implementation of CLUE has been reported extensively elsewhere.11-14 This dissertation reports on studies aimed at understanding how students construct causal mechanistic responses for various reactions and with various prompt structures. Student participants in these studies were enrolled in OCLUE or a non-transformed organic chemistry course which we refer to as the traditional course context. Students were sampled multiple times over the two-semester course to better understand how their reasoning changed over time. The studies presented in this dissertation were designed to gather a chronological series of data Study Goals and Research Questions on organic student engagement in causal mechanistic reasoning. They build on prior work that characterized general chemistry students’ engagement in causal mechanistic reasoning for a simple acid-base reaction.15 The four studies reported in this dissertation are summarized below. Study 1: Reasoning about Reactions in Organic Chemistry: Starting It in General Chemistry The first study investigates students who were enrolled in a traditional organic chemistry course. Students were prompted to reason about a simple Bronsted acid-base reaction of HCl and H2O and a Lewis acid-base reaction of NH3 with BF3. We investigated the impact of their prior general 2 chemistry course experience on their reasoning during organic chemistry. Our research questions guiding this study included: 1. How does student reasoning change over time from the end of general chemistry to the end of organic chemistry for both Bronsted and Lewis acid-base reactions? 2. What is the effect of students’ prior general chemistry experience on their reasoning and ability to draw mechanistic arrows? Study 2: Arrows on the Page Are Not a Good Gauge: Evidence for the Importance of Causal Mechanistic Explanations about Nucleophilic Substitution in Organic Chemistry In this study, we experimented with a different chemical reaction and modified prompt phrasing in order to elicit causal mechanistic reasoning from students enrolled in OCLUE as well as those enrolled in a traditional course. Students were asked to reason about a simple nucleophilic substitution reaction. The following questions guided this study: 1. How does the nature of the prompt affect student responses about a simple nucleophilic substitution reaction? 2. How does the type of organic chemistry course affect student ability to engage in causal mechanistic reasoning? 3. How does the reasoning about a reaction change over the course of two semesters? 4. How do student written explanations of reaction type compare to their mechanistic arrow drawings? Study 3: “What About the Students Who Switched Course Type?”: An Investigation of Inconsistent Organic Course Experience This study closely follows the investigation in Study 2. The previous study investigated students who were enrolled in either OCLUE or a traditional course consistently for both semesters. Due to 3 scheduling constraints, some students switch between course types and are identified as “switcher” students. This study investigates how “switcher” students reasoned about the nucleophilic substitution reaction. We studied these students longitudinally sampling them multiple times over two semesters to detect changes in reasoning. This study was guided by the following research question: 1. What is the impact of changing between a transformed organic chemistry course and a traditional organic chemistry course on students’ causal mechanistic reasoning? Study 4: The Effect of Scaffolding of Causal Mechanistic Reasoning This final study investigates OCLUE and traditional students who were selected based on their participation in two data collections in which students responded to multiple prompts with varying levels of scaffolding. In doing so, we were able to characterize the variability in student reasoning with different prompting structures and for different chemical reactions. These data are particularly interesting as they were collected from the same participants overtime. This study was guided by the following research questions: 1. How does reduced scaffolding affect student engagement in causal mechanistic reasoning and mechanistic arrow drawings for students enrolled in transformed and traditional organic chemistry courses? 2. How does expanded scaffolding affect student engagement in causal mechanistic reasoning and mechanistic arrow drawings for students enrolled in transformed and traditional organic chemistry courses? 3. How consistent are students in engaging in causal mechanistic reasoning within each prompt from the start to the end of OC2? 4. How consistent are students in engaging in causal mechanistic reasoning across multiple prompt structures? 4 REFERENCES 5 REFERENCES (1) (2) (3) (4) (5) (6) (7) (8) (9) Bhattacharyya, G.; Bodner, G. M. “It Gets Me to the Product”: How Students Propose Organic Mechanisms. J. Chem. Educ. 2005, 82 (9), 1402. https://doi.org/10.1021/ed082p1402. Ferguson, F.; Bodner, G. M. Making Sense of the Arrow-Pushing Formalism among Chemistry Majors Enrolled in Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9 (2), 102–113. Anderson, T. L.; Bodner, G. M. What Can We Do about ‘Parker’? A Case Study of a Good Student Who Didn’t ‘Get’ Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9 (2), 93–101. https://doi.org/10.1039/B806223B. Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844–849. https://doi.org/10.1021/ed2003934. Russ, R. S.; Scherr, R. E.; Hammer, D.; Mikeska, J. Recognizing Mechanistic Reasoning in Student Scientific Inquiry: A Framework for Discourse Analysis Developed from Philosophy of Science. Sci. Educ. 2008, 92 (3), 499–525. https://doi.org/10.1002/sce.20264. Cooper, M. M.; Posey, L. A.; Underwood, S. M. Core Ideas and Topics: Building Up or Drilling Down? J. Chem. Educ. 2017, 94 (5), 541–548. https://doi.org/10.1021/acs.jchemed.6b00900. Pashler, H.; Bain, P. M.; Bottge, B. A.; Graesser, A.; Koedinger, K.; McDaniel, M.; Metcalfe, J. Organizing Instruction and Study to Improve Student Learning; National Center for Education Research, Insitute of Education Sciences, U.S. Department of Education: Washington, D.C., 2007. A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas; National Academies Press: Washington, D.C., 2012. Cooper, M.; Klymkowsky, M. Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform. Journal of Chemical Education 2013, 90 (9), 1116–1122. https://doi.org/10.1021/ed300456y. (10) Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96, 1858–1872. (11) Cooper, M. M.; Underwood, S. M.; Hilley, C. Z.; Klymkowsky, M. W. Development and Assessment of a Molecular Structure and Properties Learning Progression. J. Chem. Educ. 2012, 89 (11), 1351– 1357. (12) Williams, L. C.; Underwood, S. M.; Klymkowsky, M. W.; Cooper, M. M. Are Noncovalent Interactions an Achilles Heel in Chemistry Education? A Comparison of Instructional Approaches. Journal of Chemical Education 2015, 92 (12), 1979–1987. https://doi.org/10.1021/acs.jchemed.5b00619. 6 (13) Underwood, S. M.; Reyes-Gastelum, D.; Cooper, M. M. When Do Students Recognize Relationships between Molecular Structure and Properties? A Longitudinal Comparison of the Impact of Traditional and Transformed Curricula. Chem. Educ. Res. Pract. 2016, 17 (2), 365–380. https://doi.org/10.1039/C5RP00217F. (14) Becker, N.; Noyes, K.; Cooper, M. Characterizing Students’ Mechanistic Reasoning about London Dispersion Forces. J. Chem. Educ. 2016, 93 (10), 1713–1724. https://doi.org/10.1021/acs.jchemed.6b00298. (15) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid–Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703–1712. https://doi.org/10.1021/acs.jchemed.6b00417. 7 CHAPTER II: THEORETICAL FRAMEWORKS The research presented in the forthcoming chapters was conceived from several perspectives about how people learn. I will begin by reviewing a model of information processing that i) explains and predicts how new information is perceived, ii) how multiple pieces of information can be manipulated, and iii) how and why certain information is stored in and retrieved from long term memory. Then, I will review various perspectives about the nature of the construction of this knowledge and how prior knowledge is changed. Knowledge Construction: Information Processing Model Before a student even enters a classroom, they have already developed ideas about the complex world around them through their experiences. By the time a student enters college, their knowledge of the natural world has been molded by years of intuition and instruction.1 The Information Processing Model (Figure 2.1) suggests that students’ sensory system (i.e. sight, hearing, etc.) takes in a wealth of information that is then filtered by the perception filter based on relevant prior knowledge and experiences.2 “The learner attends to what is familiar, stimulating, interesting, surprising, or exciting. To do this, the filter will be controlled, to a large extent, by what is already held in long term memory. Something cannot be familiar, interesting, or surprising unless it is being compared with some previous experience or expectation” (p. 55).3 The model suggests that information that is passed through the perception filter is brought into the working memory space where information is interpreted, manipulated, and used. Working memory is the hypothetical “operating room” where new information is integrated with information retrieved from long term memory. Evidence suggests that there is a limit to the amount of information that can be held and manipulated in the working memory at any given time4 and not all information is stored into long term memory and instead is quickly forgotten.3 8 Figure 2.1. Information Processing Model. Reproduced with permission from ref 2. Copyright 1997 American Chemical Society. The Information Processing model emphasizes the importance of prior knowledge for recognizing the relevance of new knowledge (via the perception filter), but also for the storage of new knowledge. Johnstone suggests that knowledge can be stored in long term memory as “connected, misconnected, and/or unconnected” relative to other prior knowledge.3 Learners make connections to prior knowledge in ways that make sense and are operational to them and therefore, the “misconnections” Johnstone was referring to are connections that are not aligned with an expert-like connection.3 Additionally, not all new information is tightly integrated with other information when it is stored in long term memory – what Johnstone referred to as “unconnected”3 and Ausubel termed “rote memorization.”5 The nature of these connections in long term memory explain key differences between the structure of expert knowledge compared to that of novices.1 Development of Expert Knowledge The consensus document on the nature of knowledge, learning, and teaching titled How People Learn reports on the nature of expert knowledge and the development of expertise.1 This resource begins by identifying some defining characteristics of the structure and utility of expert knowledge: 1. “Experts notice features and meaningful patterns of information that are not noticed by novices.” 9 2. “Experts have acquired a great deal of content knowledge that is organized in ways that reflect a deep understanding of their subject matter.” 3. “Experts’ knowledge cannot be reduced to sets of isolated facts or propositions but, instead, reflects contexts of applicability: the knowledge is “conditionalized” on a set of circumstances.” 4. “Experts are able to flexibly retrieve important aspects of their knowledge with little attentional effort.” 5. “Experts have varying levels of flexibility in their approach to new situations.” These characteristics of expert knowledge can be explained using the Information Processing Model.3 It is not expected that students will become experts in chemistry as a result of taking a few college chemistry courses or even completing a bachelor’s degree in chemistry. However, it is expected that students are building toward an expert-like understanding of the discipline, and therefore, understanding how expert knowledge is constructed and used is key for forming a theory of cognition underpinning research on teaching and learning. Experts have a great deal of knowledge about their discipline that is well connected in their long-term memory.1 Expert knowledge structures contain fewer “misconnections”, meaning non- canonical connections, and few “unconnected” pieces of knowledge as experts’ knowledge is not a mere collection of facts.1 When experts add to their knowledge base, they pull relevant information from long term memory into working memory and carefully integrate the new information or deem it irrelevant and reject it and do not incorporate it into their knowledge structures. This process leads to highly contextualized knowledge structures. This process posed in the Information Processing Model explains how expert knowledge develops over time. Once these expert knowledge structures are developed, they serve the expert well for recognizing meaningfully features and patterns in new scenarios via the feedback loop between the perception filter and long-term memory. For a novice lacking these well- pruned knowledge structures, they might struggle to recognize relevant information and meaningful 10 patterns in new scenarios as their perception filter is influenced by loosely organized knowledge in their long-term memory. The path from novice to expert is not direct but rather a slow progression overtime. Learners progressing from novice to expert are forced to draw upon relevant prior knowledge that might contain misconnected and unconnected knowledge and prune their knowledge structures to create meaningfully connected structures that are highly contextualized. This process of reorganizing one’s prior knowledge structures has been one of great debate and interest. The next section will present various perspectives from the science education literature about the nature of prior knowledge and how that knowledge might change. Changing Prior Knowledge: Conceptual Change and Intellectual Resources Much of the earlier research investigating theories of the mechanisms of students’ conceptual change has been conducted in the context of physics6,7 with a lesser amount in organismal biology.8 Physics content has served well in these studies because, from a very young age, students experience and observe macroscopic physical phenomena and form ideas and expectations about how the world works. Some call this naïve physics7, others have called them phenomenological primitives (or p-prims).6 This wealth of intuitive prior knowledge about the physical world provides a landscape for researchers to probe via interviews to understand students’ thinking. For example, a common phenomenon explored in this literature is a ball that is thrown into the air, reaches the peak of its flight and then falls back down again. Students are asked to explain the phenomenon in terms of the forces acting on the ball. Student explanations encompass a range of ideas including thinking about the upward force from the initial toss “wearing out” and then gravity taking over or possibly these forces being unbalanced until the top of the flight path where forces become balanced and then unbalanced again when the ball begins to fall.6,9,10 11 To make sense of student ideas such as these, science education scholars have taken influence from the philosophy of science scholars such as Thomas Kuhn and Stephan Toulmin and drawn parallels to conceptual change in novice learners.11 Two perspectives emerge from the philosophy of science – assumed coherence in scientific theories by Thomas Kuhn and rejection of this assumed coherence by Stephan Toulmin. Though, Kuhn and Toulmin’s arguments were about the nature of scientific theories and how they change and advance, these scholars’ views have influenced the endpoints for a hypothetical spectrum of theories about conceptual change in student learning.11 On one hand, there is an assumption of coherence in students’ thinking comparing them to coherent theories held by scientists; the other end rejecting this coherence in student thinking and instead assuming student knowledge to be fragmented in nature. I will first discuss conceptual change theories that are grounded in the assumption that students’ naive ideas are theoretically coherent and function much like a scientists’ coherent theories about the natural world. Student theories are assumed to be coherent because they are logical to the student and adequately function to help the student make sense of the world around them. Students use their naively logical theories to predict what will happen when an object is dropped, pushed, thrown, etc. McCloskey noticed that students’ naïve ideas about physics (what he terms their intuitive physics) closely resembled those of pre-Newtonian scholars and are perhaps as coherent as those of pre-Newtonian scholars.10 Before Newton, the belief was that an object stayed in motion because of some sort of internal impetus force that keeps it moving in a given direction. McCloskey found that the impetus theory manifested when investigating students’ thinking about a thrown ball, dropped ball, or a puck pushed across a table. He concluded that instruction was only partially effective at changing their ideas and suggested that instructors identify these prior intuitions so they can directly address how they differ from canonical laws of physics.10 12 McCloskey’s findings are only one example of the existence of recurring, inaccurate ideas in student thinking. In fact, so many incorrect ideas have been identified, not only in physics but in other STEM disciplines as well12, that they have been termed more broadly as misconceptions.13 Scholars who assume coherence in students’ theories, like McCloskey, posed approaches to “fix” students’ misconceptions. Posner and Strike were deeply influential in this area by identifying conditions under which students might discard an existing conception and hypothetically replace it with a more useful and canonically accurate one. To do this, Posner and Strike suggested that the learner must be dissatisfied with their existing conception and its power to make sense of this new phenomenon. In this way, students experience cognitive dissonance when trying to use their existing theory in an instance where it no longer has the same explanatory power. An instructor can then, present the learner with the new idea that must be intelligible and plausible, and the learner can exchange this new idea for the old one.13 Posner and Strike’s perspective of conceptual change inspired many research programs aimed at identifying misconceptions and attempting to confront and fix them12, including the work reported above by McCloskey.10 There are other examples of conceptual change theories that assume some level of coherence in student thinking. For example, Vosniadou found that young children struggled to reconcile their ideas about the “roundness” of the Earth with their observation that the Earth appeared flat to them, and this led students to construct many different models of the shape of the Earth.7 From this, Vosniadiou suggested “…that children find it difficult to construct the mental model of the Earth because this model violates certain entrenched presuppositions of the naïve framework theory of physics within which the concept of the Earth is embedded”(p. 54).7 “Entrenched presuppositions” elude to coherent structures that are assembled from birth and form a child’s naïve physics. Vosniadiou argued that the rigidity and stability of these presuppositions is what makes conceptual change so difficult as these frameworks constrain thinking when considering new information just as a child’s idea that the Earth is flat might 13 constrain their thinking when introduced to the idea that the Earth is round.7 Chi et al.14 and Carey15 have argued for a mechanism of change that involves a “re-categorization” of ideas in a student’s mental model while still assuming their mental organizational structures to be coherent and consistent regardless of the phenomenon. Just as Toulmin provided an alternative perspective to Kuhn’s assumption of coherence in scientific theory11, Minstrell9, diSessa6, and Hammer16 offer contrasting perspectives to coherence that collectively make up the fragmentation perspectives of conceptual change.11 The fragmentation perspectives assume just that – student knowledge is not necessarily coherent and is best modeled as a collection of knowledge fragments. In Minstrell’s work with high school physics students, he found that unpacking the simple phenomenon of a book sitting on a table elicited students’ naïve ideas such as “only gravity acts on the book”, “air pressure pushes it down”, and “the table is getting in the way of the book being pulled down”; similar ideas have been reported many of the coherence studies described previously.9 Minstrell’s instruction differed in that he offered many different examples of the book sitting on different surfaces. He placed the book on a student’s hand, on the ground, and on a spring and encouraged students to think about the different forces acting on the book in each example. He also added more books to form a stack when the student was holding the book in their hand. Minstrell found that half the class acknowledged that there were two forces acting on the book (gravity and the table) and the other half only acknowledged gravity. However, as more books were stacked on the students’ hands, they came to realize that perhaps there was an upward force at work as it became harder to hold the books up with the hand. Students felt that they needed to change their reasoning when thinking about the stack of books on the hand, but some were reluctant because they felt they needed to be consistent between their reasoning with the table and their reasoning with the hand.9 Theorists who assume coherence in student thinking might interpret Minstrell’s observations as 1) evidence of coherence in student thinking as the student tried to apply the same set of ideas and 14 reasoning across what appear to be different contexts and 2) might interpret these various examples as successful efforts to confront and overcome students’ misconceptions about force. However, when the students were holding heavy books and then experienced heavy books on the spring, they were cued into the idea that there was clearly an upward force. The class was only 50% sure about the presence of an upward force when first thinking about a book sitting on the table.9 This example suggests that the context in which students were asked to reason mattered a great deal in the type of answer a student gave. diSessa built on this work by introducing the theory that students’ intuitive ideas are composed of many different phenomenological primitives (p-prims) meaning “simple abstractions from common experiences that are taken as relatively primitive in the sense that they generally need no explanation; they simply happen”(p. 5).6 diSessa argues that these p-prims are fragments of knowledge that are not tightly associated with other pieces of knowledge and therefore, do not necessarily form coherent “theories” about the physical world. diSessa also argues that p-prims and other pieces of knowledge acquired over years of experience and instruction make up students’ knowledge not as a coherent structure but as fragments of knowledge in pieces that are activated and used in different contexts.6 For example, simple intuition and years of experience form the idea that more effort in a situation leads to a bigger effect and if there is any resistance, more effort is needed. diSessa names this specific intuitive piece of knowledge the Ohm’s Law p-prim as it relates directly to concepts of voltage and current.6 This label is simply a name assigned to this concept by diSessa, but students probably invoke this concept long before they learn anything about electricity because the same principle applies when pushing a box across the floor. The heavier the box, the harder it will be to push or if something blocks the path, this too presents resistance requiring more effort to move the box. Hammer et al. have contributed to this vein of theory by suggesting pieces of knowledge are intellectual resources existing in a “manifold” that are not inherently connected coherently.17 15 Conceptual change theories on the fragmentation end of the spectrum suggest that students’ resources be rewoven in meaningful ways to yield more canonical scientific understanding. This theoretical picture of “reweaving” of useful and productive resources for given contexts contrasts with the theoretical perspective of confronting and “fixing” or replacing incorrect ideas. Hammer et al. leverage “reweaving” by considering how productive resources can be “activated” in appropriate contexts.17 They argue that “thinking in terms of a manifold of cognitive elements allows models of the mind that can respond differently in different moments. But the variability is not haphazard; resources do not activate and deactivate randomly” (p. 9).17 Simply put, small groups of these resources might be activated by a given context or situation to be used to understand or reason about that phenomenon. As these groups of resources are activated together time and time again, they might become a more connected unit becoming “locally coherent” meaning the resources logically reinforce each other in that context. However, this “local coherence” is not to be assumed in other contexts for novice students. Thus, it is vital to consider how an individual’s interpretation of the context influences the resources they invoke. Indeed, invoking relevant knowledge in appropriate contexts is a key distinction between novices and experts.1 Experts have vast disciplinary knowledge that is organized in such a way that it is connected in meaningful ways rather than fragmented in the form of isolated facts. Their knowledge is closely associated with specific contexts, and thus the relevant knowledge in a given situation is easily retrieved.1 These features of expert knowledge suggest a sophisticated level of “local coherence” in expert-level knowledge as it relates to the expert’s discipline, but this does not necessarily mean we should assume the same about students’ (novices’) knowledge.6,17 Despite the wealth of misconceptions research, there is little evidence to support the idea that confronting students’ incorrect ideas will contribute to the desired “local coherence” suggested by Hammer et al. Thus, it might be more fruitful to think of students’ mechanism of conceptual change using the theories lying on the fragmentation end of the conceptual change spectrum. 16 As previously discussed, the majority of research on students’ conceptual change has been conducted in the context of physics specifically exploring student understanding of Newtonian mechanics, because, regardless of age, all people have experience with these macroscopic, observable phenomena. But what about phenomena that students can not readily observe or experience? Here lies one of the great challenges of teaching and learning chemistry – as a discipline, chemists are concerned with understanding and making predictions about activities at the atomic and sub-atomic level to understand macroscopic phenomenon.18 As the atomic level is not observable with the naked eye, chemists invoke an entire language of symbols and representations that students must also learn to navigate.18 To make matters worse, activities at the atomic and sub-atomic level are not necessarily intuitive and students’ prior knowledge about the macroscopic world might lead them astray when thinking about chemical phenomenon. One might find it encouraging to think that students are nearly “blank slates” when they enter a chemistry course since they have little to no experience with the atomic level. Nothing could be further from the truth as no student is ever a blank slate1, rather students will use whatever intellectual resources they have at their disposal when approaching a new scenario in chemistry, such as those identified as p-prims by diSessa.6 Taber suggests “P-prims are a hypothetical way of explaining both how people can provide answers to questions where they have no pre-existing answer in place, and for explaining the origins of more complex and stable conceptual structures”(p. 1033).19 They might bring ideas such as “more means more” (diSessa’s Ohm’s Law p-prim6), but these ideas might be useful in some instances and completely inappropriate in others. Understanding chemical phenomena requires students to invoke ideas about energy, chemical bonding, and chemical structure.20 Boo identified numerous misconceptions students hold about energy changes in chemical reactions, one of which being the incorrect idea that energy is required to make bonds.21 Boo hypothesizes that “…the notion that bond making requires energy input may be the result of extrapolating views about events in the 17 macroscopic world into the microscopic world – in the macroscopic world, energy is needed to make things…”(p. 574).21 The idea that energy is needed to “build” something is not inherently right or wrong but can be incorrectly applied to when thinking about energy in chemistry. Another example of misapplied resources might be a student predicting that a bigger compound has a higher boiling point relative to a smaller compound simply because it is bigger.22 In fact, predicting a relative boiling point requires multiple intellectual resources to be used in concert.22,23 The learner would need to activate resources that include: 1) interpreting the structure in 2D and/or 3D space, 2) considering atom electronegativities to determine electron density distributions and bond polarities, 3) determining molecular polarities to predict which intermolecular forces could be present, and finally 4) relating this information to the physical property of boiling point (see Figure 2.2).23 Most of these resources cannot be derived from a students’ prior intuitive knowledge of “more means more”6, however “more means more” is not a useless resource as more intermolecular forces would lead to a higher boiling point. In a study of structure-property relationships, “more means more” was invoked as a rule of thumb or a heuristic – an oversimplification to attempt to circumvent this entire cognitive exercise.22 18 Figure 2.2. The process for connecting structure and properties for a simple molecule. Reproduced with permission from ref 23. Copyright 2012 The Royal Society of Chemistry. Heuristics and Dual-Processing Heuristics have been defined as “shortcut reasoning procedures” or ways of thinking that reduce cognitive effort when approaching a problem.24 For example, students have been found to use the “octet” rule to determine the stability of atoms rather than considering net charge, ionization energies, and/or conditions under which the elements are stored.25 Maeyer and Talanquer found that students are more likely to resort to heuristics when ranking compounds by relative acidity, boiling/melting point, and solubility rather than use chemical principles to make predictions.24 Students made predictions based on a single factor without weighing multiple scientific principles or worse, made predictions based on their simple recognition of the reagent.25 Use of heuristics to make sense of phenomenon can be explained by considering the dual processing nature of human thinking. Dual processing theory identifies two cognitive processes called System 1 and System 2, although other terminology such as Implicit and Explicit have also been used in cognitive science.26 System 1 refers to instinctual thinking and behaviors that occur quickly and automatically, necessarily, to make decisions. Both humans and animals are believed to instinctually 19 invoke automated thinking processes by coordinating several cognitive subsystems. System 2 thinking is slow and intentional rather than fast and automatic.26 “System 2 provides the basis for hypothetical thinking that endows modern humans with unique potential for higher level rationality in their reasoning and decision-making” (p. 458).26 Though both systems co-exist, System 1 dominates until intentional efforts are made to override System 1 with System 2.26 Studies show that when asked to evaluate logic statements based on the string of logic presented rather than the believability of the conclusions, participants found it very difficult to override their prior beliefs.27 Participants were specifically asked to follow the logic statements (i.e. invoke System 2) rather than evaluate the believability of the statements (i.e. allow System 1 to predominate). Invoking System 2 is taxing on working memory as several pieces of information must be grappled with systematically and concurrently.27 Evidence suggests invoking heuristics is a natural response of System 1 thinking to avoid overwhelming working memory when faced with a decision. However, as evidenced in the previously mentioned studies22,24,25, use of heuristics in chemistry offers only surface-level reasoning. Learners must choose to pull multiple pieces of relevant knowledge into their working memory, coordinate them systematically to make a prediction or craft an explanation. Summary In the chapter, I have presented several perspectives about the nature of human knowledge, how it acquired and changed overtime, and how it might be used in different contexts. The Information Processing model suggests that new information is brought in through a perception filter that is mediated by knowledge already stored in long term memory. Information that passes through this filter is held in the working memory where it is used, integrated with other knowledge, or rejected from memory all together. Once knowledge is committed to long term memory, it can be organized (or disorganized) in infinite ways. Experts’ knowledge surrounding their discipline is well organized and contextualized, so it is useful and accessible. Novices knowledge is less organized and less connected. 20 The nature of novices’ prior knowledge can be modeled as coherent and theory-like, but it has also been modeled as pieces that are assembled in situ depending on the phenomenon at hand. Common to everyone (experts and novices) is our use of heuristics (or shortcut rules-of-thumb) to reduce cognitive load on the working memory when thinking about a problem. However, many heuristics do not give deep explanatory power of a phenomenon, and this has been found to be particularly true for chemistry phenomena where multiple chemical principles must be coordinated together to yield an explanation.25 The human brain can be thought of in a two-system model: System 1 is responsible for quick, intuitive thinking and behavior and System 2 is responsible for slower, intentional reasoning that might contradict that of System 1 and therefore, must be intentionally and deliberately overcome. These models lay the foundation for the research herein on how undergraduate organic chemistry students construct causal mechanistic explanations about chemical reactions. 21 APPENDIX 22 Figure 2.3. Permissions to use Information Processing Model figure. 23 Figure 2.4. Permissions to use Structure/Property figure. 24 REFERENCES 25 REFERENCES (1) National Research Council. How People Learn: Brain, Mind, Experience, and School.; National Academies Press: Washington, D.C., 1999. (2) (3) Johnstone, A. H. Chemistry Teaching - Science or Alchemy? J. Chem. Educ. 1997, 74 (3), 262–268. Johnstone, A. H. Chemical Education Research in Glasgow in Perspective. Chem. Educ. Res. Pract. 2006, 7 (2), 49–63. (4) Miller, G. A. The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. Psychol. Rev. 1956, 63, 81–97. (5) (6) (7) Ausubel, D. P. Educational Psychology: A Cognitive View; Holt, Rinehart and Winston: New York, 1968. diSessa, A. A. Knowledge in Pieces. In Constructivism in the computer age; Lawrence Erlbaum Associates: New Jersey, 1988; pp 49–70. Vosniadou, S. Capturing and Modeling the Process of Conceptual Change. Learning and Instruction 1994, 4, 45–69. Carey, S. Conceptual Change in Childhood; MIT Press/Bradford Books: Cambridge, MA. (8) (9) Minstrell, J. Explaining the “at Rest” Condition of an Object. The Physics Teacher 1982, 20 (10). (10) McCloskey, M. Intuitive Physics. Scientific American 1983, 248 (4), 122–131. (11) diSessa, A. A. A History of Conceptual Change Research: Threads and Fault Lines. In The Cambridge Handbook of the Learning Sciences; Cambridge University Press, 2014; pp 88–108. (12) National Research Council. Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering; Singer, S. R., Nielsen, N. R., Schweingruber, H. A., Eds.; National Academies Press: Washington, D.C., 2012. (13) Posner, G. J.; Strike, K. A.; Hewson, P. W.; Gertzog, W. A. Accommodation of a Scientific Conception: Toward a Theory of Conceptual Change. Sci. Education 1982, 66 (2), 211–227. (14) Chi, M. T. H.; Slotta, J. D.; de Leeuw, N. From Things to Processess: A Theory of Conceptual Change for Learning Science Concepts. Learning and Instruction 1994, 4, 27–43. (15) Carey, S. Cognitive Science and Science Education. American Psychologist 1986, 41 (10), 1123– 1130. (16) Hammer, D. Epistemological Beliefs in Introductory Physics. Cognition and Instruction 1994, 12 (2), 151–183. 26 (17) Hammer, D.; Elby, A.; Scherr, R. E.; Redish, E. F. Resources, Framing, and Transfer. In Transfer of Learning: Research and Perspectives; Information Age Publishing: Greenwich, CT, 2004. (18) Johnstone, A. H. Why Is Science Difficult to Learn? Things Are Seldom What They Seem. Journal of Computer Assisted Learning 1991, 7, 75–83. (19) Taber, K. S. Conceptual Resources for Learning Science: Issues of Transience and Grain-Size in Cognition and Cognitive Structure. Int. J. Sci. Educ. 2008, 30 (8), 1027–1053. (20) Cooper, M. M.; Underwood, S. M.; Hilley, C. Z.; Klymkowsky, M. W. Development and Assessment of a Molecular Structure and Properties Learning Progression. J. Chem. Educ. 2012, 89 (11), 1351– 1357. (21) Boo, H. K. Students’ Understandings of Chemical Bonds and the Energetics of Chemical Reactions. J. Res. Sci. Teach. 1998, 35 (5), 569–581. (22) Cooper, M. M.; Corley, L. M.; Underwood, S. M. An Investigation of College Chemistry Students’ Understanding of Structure–Property Relationships. Journal of Research in Science Teaching 2013, 50 (6), 699–721. https://doi.org/10.1002/tea.21093. (23) Cooper, M. M.; Underwood, S. M.; Hilley, C. Z. Development and Validation of the Implicit Information from Lewis Structures Instrument (IILSI): Do Students Connect Structures with Properties? Chem. Educ. Res. Pract. 2012, 13, 195–200. (24) Maeyer, J.; Talanquer, V. The Role of Intuitive Heuristics in Students’ Thinking: Ranking Chemical Substances. Sci. Educ. 2010, 94 (6), 963–984. https://doi.org/10.1002/sce.20397. (25) Taber, K. S. College Students’ Conceptions of Chemical Stability: The Widespread Adoption of a Heuristic Rule out of Context and beyond Its Range of Application. Int. J. Sci. Educ. 2009, 31 (10), 1333–1358. (26) Evans, J. St. B. T. In Two Minds: Dual-Process Accounts of Reasoning. Trends in Cognitive Sciences 2003, 7 (10), 454–459. (27) Evans, J. St. B. T. On the Conflict between Logic and Belief in Syllogistic Reasoning. Memory & Cognition 1983, 11, 295–306. 27 CHAPTER III: LITERATURE REVIEW Three-Dimensional Learning In 2012, the National Research Council released a consensus study that synthesized the literature on teaching and learning in science and engineering and proposed a vision for science education based on this available evidence.1 The result of this study was a conceptual framework intended to guide curriculum, instruction, assessment, and professional development for K-12 science education. This framework laid out three dimensions for science learning: 1) disciplinary core ideas for physical sciences, life sciences, earth and space sciences, and engineering and technology, 2) scientific and engineering practices, and 3) crosscutting concepts.1 Core ideas are concepts that are central to the discipline – the content we want students to know. Scientific practices are the “things” scientists do – they are the ways we want students to use their knowledge. Crosscutting concepts are lenses through which to think about different phenomena. Though originally designed for pre-college students, this framework, informed by evidence and theory about how people learn science, is also appropriate for post-secondary teaching and learning. Disciplinary core ideas are the “big ideas” that are central to the discipline. They are the Core Ideas foundational scientific principles that provide the explanatory underpinnings for a range of phenomena and, when understood deeply, provide predictive power when approaching new phenomena. As discussed in Chapter II, expert knowledge is not a disorganized collection of knowledge fragments and isolated facts but rather is highly integrated and contextualized in such a way that knowledge can be easily retrieved and appropriately applied in various scenarios.2 Core ideas serve as the foundation on which experts build additional knowledge. When faced with new information, that knowledge is carefully integrated into the experts’ knowledge framework, deepening their knowledge of core ideas 28 rather than standing in isolation to other prior knowledge. It is not expected that students will become experts after a few courses, but building knowledge around core ideas is step toward helping students develop expertise. The core ideas identified for the physical sciences in A Framework for K-12 Science Education1 are: 1) matter and its interactions, 2) motion and stability: forces and interactions, 3) energy, and 4) waves and their interactions in technologies. Gaining deep and useful understanding of these concepts cannot be accomplished in single lesson nor a single course. As such, the Framework lays out a hypothetical learning progression for each core idea building in sophistication over time starting from elementary school up through high school.1 The core ideas identified in A Framework for K-12 Science Education were amended to underpin the specialized disciplinary knowledge necessary for chemistry students at the college level as part of a large-scale transformation effort at Michigan State University.3 These modified core ideas are: 1) electrostatic and bonding interactions, 2) atomic/molecular structure/property relationships, 3) stability and change in chemical systems, and 4) energy. Core ideas differ from simple topic listings in that core ideas underpin many phenomena including many ideas that would traditionally be identified as a topic. For example, many organic chemistry textbooks contain a chapter on alkenes and alkynes which, as a topic, can be explained by understanding electrostatic interactions and atomic/molecular structure/property relationships. However, a deep understanding of electrostatic interactions and structure/property relationships is useful for understanding and predicting a wide range of other chemical phenomena other than just alkene reactivity.4 These modified core ideas are central to the design of the transformed undergraduate general chemistry course (Chemistry, Life, the Universe and Everything, CLUE)5 and organic chemistry course (Organic Chemistry, Life, the Universe and Everything, OCLUE)6 curricula. The CLUE and OCLUE curricula explicitly connect fine-grain knowledge pieces to these “big ideas” in both instruction and assessment.5,6 For example, student knowledge of structure/property relationships (a core idea) is explicitly assessed 29 by asking students to predict relative melting/boiling points from given structures and explain their reasoning.4 This use of explanation to engage students in the use of core ideas will be discussed in the next section under Scientific and Engineering Practices. Other projects have defined “big ideas” in chemistry7,8 and biology.9 For example, the American Chemical Society Exams Institute defined ten “big ideas” and hundreds of fine-grained content items under these “big ideas”.8 Together, these fine-grain knowledge statements, called content details, comprise comprehensive content maps undergraduate for general chemistry10, organic chemistry11, inorganic12 and physical chemistry.13 The ACS Exams Institute’s approach for using these curriculum maps and “big ideas” has been to design assessment items that elicit students’ knowledge of the fine-grain knowledge statements without explicitly connecting back to the “big idea” from which the fine-grain knowledge was derived. Cooper et al. argue that assessments (and curricula) that emphasize the finer-grain items as distant derivatives of the “big ideas” might not be facilitating the connected framework characteristic of expert-like knowledge.3 Figure 3.1 provides a visual representation for how core ideas might theoretically overlap, and knowledge might be connected to form an integrated knowledge framework. This figure is modified from Cooper et al.3 to represent how fine-grained knowledge should be directly connected to the core ideas. Figure 3.1 also represents knowledge fragments that are not grounded within the core idea framework but might be relevant in another context, for example a skill such as balancing a chemical equation or a memorized fact such as a pKa value. However, for experts, a pKa might not be represented as an isolated fact as experts understand pKa values as a manifestation of structure/property relationships and electrostatic interactions within a compound and therefore, pKa values might be deeply integrated in their knowledge structure of the discipline making it highly contextualized and easily retrieved in relative contexts. Experts may even be able to retrieve many pKa values at once without overwhelming their working memory because the information is so well organized and consolidated in the knowledge framework. 30 Figure 3.1. A visual representation for how core ideas might theoretically overlap and how pieces of knowledge might be connected to those ideas. The core ideas of electrostatic and bonding interactions abbreviated as ES & B Interactions. Scientific and Engineering Practices Another dimension of the Framework for K-12 Science Education is the scientific and engineering practices – the ways in which students use their knowledge to understand and investigate the world.1 The Framework identifies eight scientific and engineering practices listed in Figure 3.2. Scientific practices differ from simple skills such that “…engaging in scientific investigation requires not only skill but also knowledge that is specific to each practice” (p. 30).1 Additionally, engagement in the scientific practices gives students a deeper understanding of the processes of science by grappling with phenomena in similar ways to how scientists do. The Framework’s vision was for knowledge to be used meaningfully in tandem with scientific practices such as constructing explanations or developing and using models. The Framework established a consensus for the individual elements of authentic scientific investigation. Prior to this, the science education literature referred generally to the “things” scientists do as “inquiry” for which there was no generally agreed definition. 31 Scientific and Engineering Practices identified in A Framework for K-12 Science Education 1. Asking questions (science) and defining problems (engineering) 2. Developing and using models 3. Planning and carrying out investigations 4. Analyzing and interpreting data 5. Using mathematics and computational thinking 6. Constructing explanations (science) and designing solutions (engineering) 7. Engaging in argument from evidence 8. Obtaining, evaluating, and communicating information Figure 3.2. List of Scientific and Engineering Practices Identified in A Framework for K-12 Science Education.1 As part of the above-mentioned transformation effort at Michigan State University, a protocol was developed to bring much needed clarity to the discussion about how students might be assessed on their engagement in these scientific practices at the college level. This protocol, named the Three- Dimensional Learning Assessment Protocol (3D-LAP), gives criteria for evaluating formative or summative assessment items for their potential to engage a learner in scientific practices, core ideas, and/or crosscutting concepts.14 The protocol lists criteria that must be met by the assessment item if it is to have the potential to elicit evidence of student engagement in that practice, core idea, and/or crosscutting concept. As a demonstration of the utility of the 3D-LAP, we have provided the criteria for the practice of Constructing Explanations and Engaging in Argumentation from Evidence is shown in Figure 3.3. These criteria are applied to a sample assessment item shown in Figure 3.4. 3D-LAP criteria for the Scientific and Engineering Practice – Constructing Explanations and Engaging in Argumentation from Evidence Student is asked to provide reasoning based on evidence to support a claim. 1. Question gives an event, observation, or phenomenon. 2. Question gives or asks student to make a claim based on the given event, observation, or phenomenon. 3. Question asks student to provide scientific principles or evidence in the form of data or observations to support the claim. 4. Question asks student to provide reasoning about why the scientific principles or evidence support the claim. Figure 3.3. 3D-LAP criteria required to elicit evidence of student engagement in the practice of Constructing Explanations and Engaging in Argumentation for Evidence.14 32 Figure 3.4. Sample organic chemistry assessment item that meets the criteria for the practice Constructing Explanations and Engaging in Argumentation from Evidence. Engaging in Explanation and Constructing and Argument were two distinct practices listed separately in the Framework but were consolidated for the purpose of the 3D-LAP.1,14 Authors of the 3D- LAP found significant overlap between the criteria necessary to elicit evidence of Constructing Explanations and Engaging in Argument from Evidence. Osborn asserts that there is a clear distinction between the fundamental nature and ontology of explanation and argumentation. He suggests that explanations are elicited when the claim about the phenomenon is not in dispute and the explanation contains evidence and reasoning to support the given claim. An argument is fundamentally different in that the claim about the phenomenon is in question and the quality of evidence and reasoning invoked to support one claim or another comprises the act of argumentation.15 However, in the contexts of student engagement with an assessment item, since both explanation and argument demand coordinating evidence and reasoning together to either explain or argue for a given claim, the combination of these practices into a single set of criteria that accounts for these differences was fruitful for the purposes of this instrument. The criteria in Figure 3.3 first require that the assessment item either gives an event, observation, or phenomenon. In the context of an organic chemistry question, this might be a reaction presented in chemical structures or a story problem about an observation of a chemical phenomenon. Next, the assessment item must ask the student to make a claim (for argumentation) or the question 33 gives a claim (for explanation). Third, the question asks the student to provide the relevant scientific principles or describe relevant data that could support the claim. This would be true for the construction of an explanation or an argument. Finally, the assessment item must demand that the student provide reasoning linking their cited scientific principles or evidence that supports the claim. These last criteria demanding explicit elicitation of scientific principles, evidence, and reasoning are what made these two practices so complementary in the context of this instrument. These last two criteria are also the commonly omitted elements when characterizing a traditional organic chemistry question and evaluating its potential to engage students in explanation.16 In the example question in Figure 3.4, criterion 1 is met by presenting the student with a phenomenon – two reactants being added together. Figure 3.4A asks the student to make a claim – predict the product for the reaction. Many instructors would assume that simply drawing mechanistic arrows to predict a product would be evidence enough of student understanding of the mechanism by which a reaction proceeds.16 However, Figure 3.4 B and C explicitly prompts for evidence of student ability to identify scientific principles and use them to reason about the reaction. The prompt example in Figure 3.4 would also meet the criteria for Developing and Using Models as outlined in the 3D-LAP.16 While the Framework1 does not place one scientific practice above another, I have emphasized the practice of constructing explanations because “the goal of science is the construction of theories that can provide explanatory accounts of features of the world” (p. 52).1 Certainly scientists strive to construct explanations for phenomena, however, it is not just the explanation itself that is of value for students to know but also the physical act of constructing explanations. The Institute of Educational Sciences17 found convincing evidence to recommend that instructors should “selectively ask students to try to answer ‘deep’ questions that focus on underlying causal and explanatory principles” (p. 29).17 In doing so, students must identify relevant information and connect it together. Connecting information together in meaningful ways is the mechanism by which expert-like knowledge is built. 34 The third dimension outlined in the Framework’s three-dimensional vison is crosscutting Crosscutting concepts concepts (CCC).1 They can be thought of as lenses to approach and understand a phenomenon. The list of CCCs identified in the Framework are listed in Figure 3.5. One might notice that these CCCs do not make up a homogeneous set of knowledge statements like the core ideas. For example, Patterns and Scale are general features that can be considered in a system, but they are far less specific than considering how energy is conserved in a system or how structure relates to function. The CCCs were intended to highlight the different ways in which phenomena can be investigated and understood regardless of discipline. Studying patterns of causes and subsequent effects give way to theories about how phenomena occur – the mechanism of change. Knowing how something occurs allows scientists to make predictions in new situations. This begs the question: what is a mechanism, and why is it so valuable to reason about causes, effects, and the mechanisms that link them together? The Framework identifies Cause and Effect and other crosscutting concepts as important pieces of three-dimensional learning. The next section reviews additional literature exploring student engagement in mechanistic explanations. Crosscutting Concepts as identified in the Framework for K-12 Science Education 1. Patterns 2. Cause and Effect: Mechanism and Prediction 3. Scale, Proportion and Quantity 4. Systems and System Models 5. Energy and Matter: Flows, Cycles, and Conservation 6. Structure and Function 7. Stability and Change Figure 3.5. Crosscutting Concepts identified in A Framework for K-12 Science Education.1 35 Science education scholars have not reached consensus for a definition of mechanistic Causal Mechanistic Reasoning explanation. Russ et al. point out, “The ambiguity of the objective makes it difficult for researchers, curriculum developers, and teachers to pursue it systematically” (p. 500).18 Some studies have identified that students’ reasoning can be teleological in nature meaning the students reasoned that the phenomenon occurs because in doing so, it is fulfilling its purpose.19,20 These studies identified examples of what should and should not characterized as mechanistic reasoning. Russ et al. argued the need for a more definitive framework to identify the elements of mechanistic reasoning in student responses.18 Drawing on work by Machamer, Darden, and Craver21, Russ et al. define a mechanism as “[the explanation of] how phenomena are produced by tracing the productive changes continuously from setup conditions through intermediate stages to termination conditions” (p. 511).18 To do this, one must identify the entities underlying the phenomenon, their properties, organizations, and activities. Russ et al.’s study with middle school children elicited mechanistic examples such as “water molecules are little hard balls that bounce off everything.” This passage identifies entities of change (water molecules as balls), properties (small and hard), and their activities (bouncing around).18 Krist et al. specified that these entities must be identified at one scalar level below the phenomenon of interest as the entities must be underlying the phenomena.22 Russ et al. also argue that mechanistic reasoning is inherently causal in nature meaning “the why” behind a given phenomenon is naturally built into a mechanistic explanation as “mechanism both accounts for the causal law governing physical behavior and is more than the causal law” (p. 506).18 Finally, Russ et al. argue that a mechanistic explanation that reasons about the underlying entities, their properties and activities will encompass causal reasoning and, in turn, a causal-only explanation is not sufficient for mechanistic reasoning. Prior work by Cooper, et al.23, Becker et al.24 and Noyes and Cooper25 identified student reasoning that is causal-only and mechanistic- only in the context of acid-base reactions and London Dispersion Forces and have used the term causal 36 mechanistic reasoning to clearly identify reasoning that explicitly includes both elements. Other researchers have treated mechanism and causality as independent dimensions of explanation in their analysis of student reasoning about chemical reactions26,27 and colligative properties.28 The work presented in this dissertation will use the term causal mechanistic reasoning to clearly identify a response that reasons about the causal factors contributing to the phenomenon and the underlying entities/activities (the mechanism). Organic chemists utilize the term “mechanism” and mechanistic reasoning to mean a step-by- step account of electron movement based on established patterns of reactivity due to electrostatic interactions.29 That is, a chemical mechanism identifies entities a scalar level below the phenomenon of interest (electrons underlying the structure of atoms) and uses their activities to explain the effect. For example, student understanding of intermolecular forces underpinning phase change has been studied.4 However, intermolecular forces are a phenomenon within themselves with an underpinning mechanism explained by electrostatic interactions between subatomic particles. Talanquer identified fourteen different types of chemical mechanisms that exist at the molecular/atomic and subatomic levels to explain matter transformation processes, energy transfer and transformation, activation, stabilization, and equilibration.30 Mechanistic Reasoning in Organic Chemistry Chemistry students are introduced to a wide variety of representations such as Lewis structures, chemical formulas, and mathematical equations just to name a few. Students who advance to organic chemistry are then introduced to the electron pushing formalism to represent how a chemical reaction occurs. The formalism is used by expert chemists to represent movement of electrons responsible for bond breaking and bond formation. The goal is not for students to merely reproduce these representations but instead for students to understand why the electrons move in such a way – to illustrate their knowledge of a chemical phenomenon. There are numerous studies identifying 37 undergraduate and graduate student difficulties using the electron pushing formalism (mechanistic arrows) in an expert-like way.31,32,33,34,35 Many of these studies have explored student reasoning in tandem with their use of the electron pushing formalism in think-aloud interview protocols and will be reviewed below. Bhattacharyya and Bodner presented organic chemistry graduate level students with various reaction conditions and subsequent products and asked participants to pose mechanisms.31 These authors found that students often attempted to reproduce mechanisms from memory and as such posed improbable intermediates. The proposed mechanisms “got them to the product” rather than following scientific principles. Students explained their approach to proposing mechanistic arrows as “just playing around” and “forcing it to fit”. Bhattacharyya et al. characterized these strategies as an exercise in “connecting the dots”.31 These findings were particularly concerning as these participants were otherwise identified as successful students earning acceptable grades in the graduate level course. Anderson and Bodner found similar trends in a study of undergraduate organic students who had been successful in their prior general chemistry course.36 In this case study, they highlighted the experience of one student for whom these authors studied over the course of the entire semester. This student, Parker, strived to understand why reactions occurred and why mechanisms occurred but felt forced to resort to memorization and rule-based reasoning rather than relying on chemical knowledge. The trouble was, Parker often forgot the rules and had struggled to apply them appropriately. However, the authors of the study observed the lectures and felt that the instructor was explaining why reactions occurred and valued students’ understanding. Anderson et al. concluded that Parker’s difficultly associating chemical meaning to mechanistic arrows resulted in his struggle to understand why reactions occur.36 To an expert, a mechanistic arrow communicates a wealth of information about a chemical system, an electron rich area that is attracted to an electron deficient area, but a novice such as Parker may not interpret this information in a meaningful way. 38 Kraft et al. characterized student modes of reasoning to determine which modes of reasoning led to successful mechanisms.37 In this study, graduate students were presented with a reagents and products and asked to propose mechanisms and explain their reasoning. Participants were most likely to invoke case-based reasoning with medium success in constructing the correct mechanism. Case-based reasoning is characterized by attempts to draw a parallel between the current problem and a prior problem that might somehow resemble the problem at hand but lacks an overarching generalization for how or why those cases are related and the knowledge that might be inappropriately applied. Less successful students relied on rule-based reasoning strategies where they invoked rules and heuristics, usually incorrectly, as their rules were so vague and lacked additional knowledge needed to reason through different types of problems. More successful students invoked a model-based mode of reasoning where they were able to correctly relate the problem at hand to a larger concept in their knowledge structure and generalize that knowledge to appropriately apply it.37 For example, identifying that a process proceeds via an SN2 mechanism and then applying their knowledge of an SN2 mechanism to reason about the new problem was considered a model-based reasoning approach. However, this strategy was invoked least often by students but often times by experts in coordination with other types of reasoning at appropriate times. This finding emphasizes key features of the structure of expert knowledge in contrast to novices. Experts’ knowledge is not a mere collection of facts and rules (such as those invoked in rule-based reasoning) but reflects a deep understanding of the discipline.2 That is not to say that experts do not have rules of thumb and/or and sometimes invoke rule-based reasoning, but they do so in contexts that are relevant and useful.37 Graulich investigated the various ways organic chemistry students’ approach multiple choice questions presenting starting material and product and the student must choose the correct reagent set.38 This author identifies several strategies and heuristics used by students to predict the reagent with various success. For example, students made associations with memorized material from their 39 instruction rather than considering how a reagent would interact mechanistically. Additional studies show students tend to use surface features when categorizing organic reactions in contrast to experts who tend to consider mechanistic pathways.39,40 Russ et al.’s mechanistic reasoning framework18 has influenced the analysis of many studies of mechanistic reasoning in organic chemistry. A series of studies by Talanquer aimed to characterize the landscape of student reasoning about chemical reaction mechanisms, causality, and chemical control.26,27 These studies were conducted with a variety of student participants enrolled in general chemistry and organic chemistry up through first year graduate students and advanced graduate students. These authors identified several “conceptual modes” in student reasoning. Conceptual modes are defined as “the different manners in which a given entity, system, or phenomenon seem to be conceptualized by an individual in different situations or by different individuals with diverse backgrounds” (p. 562).27 In doing so, this study found that advanced graduate students invoked more sophisticated conceptual modes than less experienced students.26 Weinrich and Talanquer conducted additional interviews with a similar student sample to elicit conceptual modes about chemical mechanism, causality, and chemical control in forming specific products.27 These studies provide rich descriptions of the knowledge students invoke when considering how and why reactions occur. In a follow-up analysis, Weinrich and Talanquer characterized reasoning not in terms of the knowledge invoked (conceptual modes) but instead looked at the modes of reasoning itself ranging from descriptive reasoning to multi-component reasoning modes.41 The authors re-analyzed interview data where students were asked to reason about how and why various reactions happen and to explain how to control various reactions. These authors identified four characterizations of reasoning with increasing sophistication: descriptive, relational, linear-causal, and multi-component. Descriptive reasoning was defined as reasoning that focused on surface features and it was very rare that students reasoning was limited to this level. A quarter of students in this study reasoned with relational reasoning meaning they 40 identified correlations between properties and behaviors of implicit and explicit factors but did not go the extra step to justify them. Nearly half of the participants invoked a linear-casual reasoning mode in which the phenomenon was explained in terms of simple cause and effect relationships and students invoke implicate and explicit features. Finally, a quarter of the participants invoked multi-component reasoning which invokes several causal factors and their complex relationships and activities to build a “causal story”. However, when broken down by education level (general chemistry ranging to advanced graduate level), it was found that simple modes of reasoning (descriptive and linear) was the dominate reasoning mode for general chemistry students and the reasoning became more sophisticated overtime as students became more advanced.41 Similarly, Bodé et al. used this framework to analyze students’ modes of reasoning when engaging in argumentation about contrasting cases of SN1 reactions and found that ~60% of correct arguments (i.e. chose the correct case) also discussed causal connections between carbocation stability and activation energy.42 In study of student reasoning of colligative properties in the context of freezing point depression, Moreira et al.28 amended the reasoning modes analysis framework described above (descriptive, relational, linear-causal, and multi-component) to incorporate Russ et al.’s18 definition of mechanism to explicitly identify entities, their activities and properties. However, in this study of 10th grade students, most of the explanations were limited to descriptive or relational modes of reasoning meaning they did not identify the causal links between entities or mechanistic activities suggesting that “advancing student thinking about colligative properties such as freezing point depression requires the construction of mechanisms that recognize the probabilistic nature of change at the particulate level” (p. 129). These authors also concluded that the explanations elicited about this phenomenon lacked causal sophistication, but students were still capable of reasoning about mechanisms at the particulate level.28 The studies presented in this dissertation utilize very simple systems/phenomena to better elicit student causal mechanistic reasoning at the particulate level. 41 Caspari et al. investigated student reasoning about the activation energies for contrasting cases of single mechanistic steps (i.e. different leaving groups leaving different substituted substrates).43 They found students mostly approached reasoning using a static approach meaning students invoked properties of the structure of the product but did not invoke any reasoning about the actual mechanistic process (i.e. a dynamic approach. The authors argued that students were only able to make the causal connection between activation energy and charge if their reasoning invoked process-oriented reasoning (i.e. a dynamic approach).43 Bongers et al. also studied students’ static vs. dynamic reasoning and found that students engaged in both types of reasoning but suggested their dynamic models (i.e. mental models of the particles in motion) were not well connected to their static mental models but this improved overtime.44,45 In another study, Caspari et al. also incorporated Russ et al.’s mechanistic framework into their analysis of organic students’ mechanistic reasoning.46 In this study, they found that students often invoke backward-oriented reasoning meaning students justified mechanistic steps based on “information about subsequent parts in the mechanism [to make] a decision about prior parts.” For example, students would pose an initial step to get to a structure that they recognize as demonstrated in the following quote: “And I knew that I wanted this positive charge to end up over here to make a better leaving group, so I just did a hydrogen shift” (p. 53).46 Flynn et al. have suggested that students should first learn the electron-pushing formalism as a representation before learning any specific reactions to reduce cognitive load demands.47 They have designed a transformed organic chemistry curriculum in which reactions are organized by patterns of mechanism rather than by functional group, however, studies show that this does not necessarily mean students organized reactions this way when asked to engage in a card sorting task48 although students’ organizations do become more expert-like overtime.45 Bodé and Flynn have identified a number of strategies students use when solving synthesis problems49 but later found that no participants used the 42 electron-pushing formalism incorrectly in a set of interviews probing students’ understanding of familiar and unfamiliar reactions.50 Grove, Cooper, and Cox investigated student use of mechanistic arrows for familiar and unfamiliar reactions throughout two semesters of organic chemistry.33 Data were collected via an online drawing tool that allowed researchers to replay the student inputs. Students were asked to draw a mechanism to predict the product for a range of reactions. Between 30-60% of students did not draw mechanisms at all for the multiple reactions. Of the those that did, 20% drew the mechanistic arrows after they predicted a product. Grove et al. also reported that students who used mechanisms were only more successful at predicting a correct product when the reaction was unfamiliar leading the authors to conclude that many students not only memorized the product of the reaction, but also the position of the mechanistic arrows.34 Summary In this chapter, I have summarized various areas of literature investigating student engagement in explanation. There is strong evidence to support engaging students in explanation to deepen their understanding of science. The three-dimensional learning framework described in the Framework for K- 12 Science Education was conceived of the best available evidence for teaching and learning science. It provides a vision for how student engagement in scientific practices such as explanation might be integrated with deep, meaningful knowledge of a discipline. Some researchers have explicitly defined a specific type of explanation called a mechanistic explanation which includes reasoning about underlying entities and causes to explain how and why a phenomenon occurs. Several studies have identified organic students’ difficulties understanding how and why reaction occur. This chapter reviewed studies probing student reasoning in organic chemistry, many of which specifically investigating students’ understanding of mechanism. We add to this literature by eliciting and characterizing students’ casual mechanistic explanations in various course contexts, using various prompt structures, and across time. 43 REFERENCES 44 REFERENCES (1) A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas; National Academies Press: Washington, D.C., 2012. (2) National Research Council. How People Learn: Brain, Mind, Experience, and School.; National Academies Press: Washington, D.C., 1999. (3) (4) (5) (6) (7) Cooper, M. M.; Posey, L. A.; Underwood, S. M. Core Ideas and Topics: Building Up or Drilling Down? J. Chem. Educ. 2017, 94 (5), 541–548. https://doi.org/10.1021/acs.jchemed.6b00900. Cooper, M. M.; Corley, L. M.; Underwood, S. M. An Investigation of College Chemistry Students’ Understanding of Structure–Property Relationships. Journal of Research in Science Teaching 2013, 50 (6), 699–721. https://doi.org/10.1002/tea.21093. Cooper, M.; Klymkowsky, M. Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform. Journal of Chemical Education 2013, 90 (9), 1116–1122. https://doi.org/10.1021/ed300456y. Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96, 1858–1872. College Board. Advances in AP: AP chemistry https://apcentral.collegeboard.org/courses/ap- chemistry/course. (8) Murphy, K.; Holme, T.; Zenisky, A.; Caruthers, H.; Knaus, K. Building the ACS Exams Anchoring Concept Content Map for Undergraduate Chemistry. J. Chem. Educ. 2012, 89 (6), 715–720. (9) American Association for the Advancement of Science. Vision and Change in Undergraduate Biology Education: A Call to Action; Washington, D.C., 2011. (10) Holme, T.; Murphy, K. The ACS Exams Institute Undergraduate Chemistry Anchoring Concepts Content Map I: General Chemistry. J. Chem. Educ. 2012, 89 (6), 721–723. (11) Raker, J. R.; Holme, T.; Murphy, K. The ACS Exams Institute Undergraduate Chemistry Anchoring Concepts Content Map II: Organic Chemistry. J. Chem. Educ. 2013, 90 (11), 1443–1445. (12) Marek, K. A.; Raker, J. R.; Holme, T.; Murphy, K. The ACS Exams Institute Undergraduate Chemistry Anchoring Concepts Content Map III: Inorganic Chemistry. J. Chem. Educ. 2018, 95 (2), 233–237. (13) Holme, T.; Reed, J. J.; Raker, J. R.; Murphy, K. The ACS Exams Institute Undergraduate Chemistry Anchoring Concepts Content Map IV: Physical Chemistry. J. Chem. Educ. 2018, 95 (2), 238–241. 45 (14) Laverty, J. T.; Underwood, S. M.; Matz, R. L.; Posey, L. A.; Carmel, J. H.; Caballero, M. D.; Fata- Hartley, C. L.; Ebert-May, D.; Jardeleza, S. E.; Cooper, M. M. Characterizing College Science Assessments: The Three-Dimensional Learning Assessment Protocol. PLOS ONE 2016, 11 (9), e0162333. https://doi.org/10.1371/journal.pone.0162333. (15) Osborne, J. F.; Patterson, A. Scientific Argument and Explanation: A Necessary Distinction? Sci. Educ. 2011, 95 (4), 627–638. (16) Stowe, R. L.; Cooper, M. M. Practicing What We Preach: Assessing “Critical Thinking” in Organic Chemistry. J. Chem Educ. 2017, 94 (12), 1852–1859. https://doi.org/10.1021/acs.jchemed.7b00335. (17) Pashler, H.; Bain, P. M.; Bottge, B. A.; Graesser, A.; Koedinger, K.; McDaniel, M.; Metcalfe, J. Organizing Instruction and Study to Improve Student Learning; National Center for Education Research, Insitute of Education Sciences, U.S. Department of Education: Washington, D.C., 2007. (18) Russ, R. S.; Scherr, R. E.; Hammer, D.; Mikeska, J. Recognizing Mechanistic Reasoning in Student Scientific Inquiry: A Framework for Discourse Analysis Developed from Philosophy of Science. Sci. Educ. 2008, 92 (3), 499–525. https://doi.org/10.1002/sce.20264. (19) Carey, S. On the Origin of Causal Understanding. In Causal Cognition: A Multidisciplinary Debate; Sperber, D., Premack, D., Premack, A. J., Eds.; Symposia of the Fyssen Foundation; Oxford University Press: New York, NY, 1995; pp 263–308. (20) Metz, K. E. Development of Explanation: Incremental and Fundamental Change in Children’s Physics Knowledge. J. Res. Sci. Teach. 1991, 28 (9), 785–797. https://doi.org/10.1002/tea.3660280906. (21) Machamer, P.; Darden, L.; Craver, C. F. Thinking about Mechanisms. Philosophy of Science 2000, 67 (1), 1–25. (22) Krist, C.; Schwarz, C.; Reiser, B. Identifying Essential Epistemic Heuristics for Guiding Mechanistic Reasoning in Science Learning. J. Learn. Sci. 2019, 28 (2), 160–205. (23) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid–Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703–1712. https://doi.org/10.1021/acs.jchemed.6b00417. (24) Becker, N.; Noyes, K.; Cooper, M. Characterizing Students’ Mechanistic Reasoning about London Dispersion Forces. J. Chem. Educ. 2016, 93 (10), 1713–1724. https://doi.org/10.1021/acs.jchemed.6b00298. (25) Noyes, K.; Cooper, M. M. Investigating Student Understanding of London Dispersion Forces: A Longitudinal Study. J. Chem. Educ. 2019, 96 (9), 1821–1832. (26) Yan, F.; Talanquer, V. Students’ Ideas about How and Why Chemical Reactions Happen: Mapping the Conceptual Landscape. Int. J. Sci. Educ. 2015, 37 (18), 3066–3092. 46 (27) Weinrich, M. L.; Talanquer, V. Mapping Students’ Conceptual Modes When Thinking about Chemical Reactions Used to Make a Desired Product. Chem. Educ. Res. Pract. 2015, 16, 561–577. (28) Moreira, P.; Marzabal, A.; Talanquer, V. Using a Mechanistic Framework to Characterize Chemistry Students’ Reasoning in Written Explanations. Chem. Educ. Res. Pract. 2019, 20, 120– 131. (29) Bhattacharyya, G. From Source to Sink: Mechanistic Reasoning Using the Electron-Pushing Formalism. J. Chem. Educ. 2013, 90 (10), 1282–1289. https://doi.org/10.1021/ed300765k. (30) Talanquer, V. Importance of Understanding Fundamental Chemical Mechanisms. J. Chem. Educ. 2018, 95 (11), 1905–1911. (31) Bhattacharyya, G.; Bodner, G. M. “It Gets Me to the Product”: How Students Propose Organic Mechanisms. J. Chem. Educ. 2005, 82 (9), 1402. https://doi.org/10.1021/ed082p1402. (32) Bhattacharyya, G. Trials and Tribulations: Student Approaches and Difficulties with Proposing Mechanisms Using the Electron-Pushing Formalism. Chem. Educ. Res. Pract. 2014, 15, 594–609. (33) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844–849. https://doi.org/10.1021/ed2003934. (34) Grove, N. P.; Cooper, M. M.; Cox, E. L. Does Mechanistic Thinking Improve Student Success in Organic Chemistry? J. Chem. Educ. 2012, 89 (7), 850–853. https://doi.org/10.1021/ed200394d. (35) Flynn, A. B.; Featherstone, R. B. Language of Mechanisms: Exam Analysis Reveals Students’ Strengths, Strategies, and Errors When Using the Electron-Pushing Formalism (Curved Arrows) in New Reactions. Chem. Educ. Res. Pract. 2017, 18, 64–77. (36) Anderson, T. L.; Bodner, G. M. What Can We Do about “Parker”? A Case Study of a Good Student Who Didn’t “get” Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9, 93–101. (37) Kraft, A.; Strickland, A. M.; Bhattacharyya, G. Reasonable Reasoning: Multi-Variate Problem- Solving in Organic Chemistry. Chem. Educ. Res. Pract. 2010, 11, 281–292. (38) Graulich, N. Intuitive Judgments Govern Students’ Answering Patterns in Multiple-Choice Exercises in Organic Chemistry. J. Chem. Educ. 2015, 92 (2), 205–211. https://doi.org/10.1021/ed500641n. (39) Graulich, N.; Bhattacharyya, G. Investigating Students’ Similarity Judgements in Organic Chemistry. Chem. Educ. Res. Pract. 2017, 18, 774–784. (40) Galloway, K. R.; Leung, M. W.; Flynn, A. B. A Comparison of How Undergraduates, Graduate Students, and Professors Organize Organic Chemistry Reactions. J. Chem. Educ. 2018, 95 (3), 355– 365. 47 (41) Weinrich, M. L.; Talanquer, V. Mapping Students’ Modes of Reasoning When Thinking about Chemical Reactions Used to Make a Desired Product. Chem. Educ. Res. Pract. 2016, 17, 394–406. (42) Nicholas E Bodé; Jacky M. Deng; Alison B. Flynn. Getting Past the Rules and to the WHY: Cusal Mechanistic Arguments When Judging the Plausibility of Organic Reaction Mechanisms. J. Chem. Educ. 2019, 96 (6), 1068–1082. (43) Caspari, I.; Kranz, D.; Graulich, N. Resolving the Complexity of Organic Chemistry Students’ Reasoning through the Lens of a Mechanistic Framework. Chem. Educ. Res. Pract. 2018, 19, 1117. (44) Bongers, A.; Northoff, G.; Flynn, A. B. Working with Mental Models to Learn and Visualize a New Reaction Mechanism. Chem. Educ. Res. Pract. 2019, 20 (3), 554–569. (45) Lapierre, K. R.; Flynn, A. B. An Online Categorization Task to Investigate Changes in Students’ Interpretations of Organic Chemistry Reactions. J. Res. Sci. Teach. 2020, 57 (1), 87–111. (46) Caspari, I.; Weinrich, M. L.; Sevian, H.; Graulich, N. This Mechanistic Step Is “Productive”: Organic Chemistry Students’ Backward-Oriented Reasoning. Chem. Educ. Res. Pract. 2018, 19, 42–59. (47) Flynn, A. B.; Ogilvie, W. W. Mechanisms before Reactions: A Mechanistic Approach to the Organic Chemistry Curriculum Based on Patterns of Electron Flow. J. Chem. Educ. 2015, 92 (5), 803–810. (48) Galloway, K. R.; Leung, M. W.; Flynn, A. B. Patterns of Reactions: A Card Sort Task to Investigate Students’ Organization of Organic Chemistry Reactions. Chem. Educ. Res. Pract. 2019, 20 (1), 30– 52. https://doi.org/10.1039/C8RP00120K. (49) Bode, N. E.; Flynn, A. B. Strategies of Successful Synthesis Solutions: Mapping, Mechanisms, and More. J. Chem. Educ. 93 (4), 593–604. (50) Webber, D. M.; Flynn, A. B. How Are Students Solving Familiar and Unfamiliar Organic Chemistry Mechanism Questions in a New Curriculum? J. Chem. Educ. 2018, 95 (9), 1451–1467. 48 CHAPTER IV: REASONING ABOUT REACTIONS IN ORGANIC CHEMISTRY: STARTING IT IN GENERAL CHEMISTRY Preface This chapter discusses our findings in an investigation of organic chemistry students’ reasoning about a simple acid-base reaction. These students were enrolled in the same non-transformed organic chemistry course but differed in their general chemistry course experience – either transformed general chemistry or more traditional general chemistry course. This research has been previously published in the Journal of Chemistry Education and is reprinted with permission from Crandell, O.M.; Kouyoumdjian, H.; Underwood, S.M.; Cooper, M.M. Reasoning about Reactions in Organic Chemistry: Starting it in General Chemistry. J. Chem. Educ. 2019, 96(2), 213-226. Copyright 2019 American Chemical Society. A copy of permissions obtained is included in the Appendix. Supplemental Information for this manuscript is included in the Appendix. Introduction Acid–base chemistry is fundamental to understanding a wide range of chemical reactions: from simple Brønsted proton transfer, to nucleophilic substitutions, to the role of Lewis acids in catalysis. In organic chemistry it is generally acknowledged that to develop expertise, students must: 1. Understand the central role of acid–base chemistry 2. Be able to identify acids and bases 3. Be able to predict the products of acid–base reactions 4. Move flexibly among the various models chemists use to describe such reactions Nevertheless, there are numerous studies on the problems associated with student conceptual understanding of the nature of acid and bases.1–8 For example, it has been shown that students struggle 49 to identify acids and bases both at the high school chemistry level1–3 and the undergraduate level.4–8 We also know that students do not necessarily leave chemistry degree programs with an operational understanding of acid–base reactions.9 For example, chemistry graduate students in the midst of their dissertation research can wrestle with determining acidity of simple alcohols.9 Much of the prior work on acid–base understanding at the college level has tended to focus on the nature of acids and acidity, rather than on the acid–base reaction itself. For example, McClary and Talanquer identified types of naive heuristics, or rules of thumb, that general chemistry students may be using when reasoning about relative strengths of acidity.10,11 These findings were operationalized by Bretz and McClary in the development and validation of an instrument to help organic chemistry instructors identify incorrect ideas about acid strength that their students may hold.6 Other researchers have focused on how students use acid–base models. For example, Cartrette and Mayo found that most students tend to use the Brønsted acid–base model, even in cases where it was not appropriate.7 While many educators agree that a firm grip on the use of the Lewis acid–base model of reactivity is important for organic chemistry, there are few indications that most students who emerge from a general chemistry course are prepared to use it to reason about organic reactions. Because of this, most commercially available organic chemistry textbooks include early chapters dedicated to acid–base chemistry that are intended to refresh and build on students’ knowledge of acid–base reactions from general chemistry.12,13 These chapters typically offer definitions and examples for Brønsted acid–base theory and Lewis acid–base theory and provide examples of acid–base reactions of various types. We have previously reported our findings about how students who were enrolled in a transformed general chemistry course reason about the simple Brønsted acid–base reaction in which HCl reacts with H2O.14 In this earlier study we showed that student reasoning about acid–base reactions can be elicited by appropriately designed prompts, and that students who were able to provide causal mechanistic explanations (discussed below) for acid–base reactions were also more likely write correct 50 mechanistic arrows that correlate with electron movement during such reactions. Here we extend that study to 1) investigate the evolution of student causal mechanistic explanations and mechanistic arrow drawings of Lewis acid–base reactions over two semesters of organic chemistry, 2) investigate, via a longitudinal study, the effect of different general chemistry preparation on student causal mechanistic reasoning as they move through organic chemistry, and 3) apply this methodology to student understanding of a simple Lewis acid–base reaction. Defining Causal Mechanistic Reasoning While it has been noted that causal mechanistic reasoning is an important goal in science education, there are a number of different ideas about just what this phrase means. Some have argued that mechanistic reasoning is inherently causal and frequently use the terms “causal mechanistic reasoning” and “mechanistic reasoning” interchangeably.15–18 Russ et al. emphasized the need to identify the components of the system that are “doing” the phenomenon when they stated “…mechanisms account for observations by showing that underlying objects cause local changes in the system by acting on one another.”18 The key term in this passage is the “underlying objects”. The mechanistic piece of a causal mechanistic explanation must have defined underlying objects or entities that are at least one scalar level below the phenomenon of interest. In a similar vein, Krist, Schwarz and Reiser have proposed a framework of epistemic heuristics to support students development of mechanistic thinking that involves: (i) thinking across scalar levels; (ii) identifying and unpacking relevant factors; and (iii) checking how well the underlying mechanisms fit the observed phenomenon.19 Indeed, Talanquer states that mechanistic reasoning is necessary in chemistry as “the organization of components can take place at various levels, and properties of a system at a given level often emerge from the properties, interactions, activities, and organization of the subcomponents defined at a sublevel.”20 51 In our work on causal mechanistic reasoning to explain chemical phenomena we have separated the causal and mechanistic pieces, because our analysis shows that students can provide causal explanations, without a mechanistic piece involving objects at a scalar level below the phenomenon of interest, and vice versa.14,21 For example, in discussing how London dispersion forces arise, students may give a causal description that involves transient positive and negative charges being attracted without discussing how those transient charges arise. The mechanistic aspect of the explanation arises from a discussion of electron movement creating the transient charges.21 In this case the level below the observed phenomenon includes electrons or other subatomic particles. An explanation that involves movement of electrons to produce transient charges that results in an attraction between particles is classified (by us) as a causal mechanistic explanation of London dispersion forces.21 Similarly, in characterizing acid–base reactions, we were able to separately identify causal explanations, mechanistic explanations, as well as causal mechanistic explanations.14 Causal explanations generally invoke an electrostatic interaction between the reacting species, while mechanistic explanations include the idea that electrons are moving as bonds break and form. Causal mechanistic explanations include both these ideas. For acid–base reactions the situation is further complicated by the theories that students use to explain the reactions. For example, when asked to explain the reaction HCl + H2O → H3O+ + Cl–, some general chemistry students simply provide us with a description.14 For example, Heather writes “The HCl is the acid meaning it is a proton donor and the water is the base meaning it is a proton acceptor. At the molecular level the hydrogen from the HCl is breaking off and the water is gaining it forming H3O+.” This explanation was coded as Brønsted Descriptive because the student used the Brønsted acid–base model and simply described what happened, but did not explain why or how the reaction occurred. However, when we refined the prompt, to ask both what is happening, and then separately why is it happening, many more students were able to provide a causal mechanistic explanation.14 For example, Francis wrote “The lone pair on 52 the water molecule attracts the Hydrogen from the HCl. The H-Cl bond is broken and forms a new bond with oxygen. The reaction occurs because the partial negative charge on the oxygen attracts the partial positive charge on the hydrogen...” This description invokes the Lewis acid–base model, because the student invokes the involvement of the lone pair and provides a causal mechanistic explanation for why the reaction occurs and was therefore, classified as Lewis Causal Mechanistic. The full coding scheme that was used in the prior work,14 and in this study of the investigation of HCl and H2O, is provided in Data Analysis section below. It should be noted that causal mechanistic reasoning, in the sense described above is not the same as mechanistic reasoning in organic chemistry as exemplified by the drawing of curved arrows. Although in a national survey of 103 organic chemistry faculty, 77 experts agreed with a definition of mechanistic reasoning that requires one to represent electron movement based on previously established knowledge of chemical reactivity,22 this does not necessarily include the idea of pushing electrons from source (a region of high electron density) to sink (a region of lower electron density), but perhaps involves something more akin to pattern recognition on the part of the student. Ideally one might hope that as students draw mechanistic arrows they are mindful that the arrows represent the movement of electrons from a source of electrons to a sink. However, there is ample evidence that many students do not use their knowledge of chemical reactivity to do this but rather rely on pattern recognition and memory to answer questions about mechanisms.23,24 Triangulation of data from both written responses and drawn mechanisms can provide us stronger evidence of student understanding then either data source alone. It is somewhat problematic that the terminology, definitions, and meanings of mechanistic reasoning are easily conflated and can create confusion about what is expected of students and how we will know if they have met those expectations. For the purposes of this paper and based on our previous work with acid–base reactions and our work in other contexts, we define causal mechanistic reasoning 53 as a type of explanation of a phenomenon that identifies the causal factors and the physical entities underlying a phenomenon and uses both the causal factors and the activities of the underlying entities (electrons) to provide a step-wise account of the phenomenon from start to finish. Why Engage Students in Causal Mechanistic Reasoning in Organic Chemistry? In an investigation of the utility of mechanistic thinking for organic students at the end of second-semester organic chemistry (OC2), students were asked to draw mechanisms and predict products for both familiar and unfamiliar reactions.25 Just as in other studies,23,24 the numbers of students who were able to use mechanistic arrows productively was rather disappointing. For the unfamiliar tasks, where the students could not recall the answer from memory, students who drew mechanisms were significantly more likely to predict the correct products than the students who did not use mechanisms.26 This is certainly evidence that students’ use of reaction mechanisms can be a powerful predictive tool if used appropriately. However, it is common for organic instructors to assess student learning by asking them to draw the product of a reaction, or even a mechanism without justifying their prediction or mechanistic proposal, believing (erroneously) that this is evidence of students’ ability to reason about organic chemistry. Indeed, in a small survey of organic exams given at the nation’s elite universities, little explicit evidence of reasoning was required from students.27 There is an extensive literature on the benefits of having students answer deep explanatory questions. For example, construction of deep explanatory accounts of phenomena is cited in the IES report, Organizing Instruction and Study to Improve Student Learning: IES Practice Guide as one of only two instructional strategies that are supported by strong evidence as improving learning.28 We propose that helping students to engage in causal mechanistic thinking is an important and useful variant on this idea.29 The process requires that students reflect on and connect the sequence of events underlying a phenomenon and the causal drivers involved. That is, the act of constructing a causal mechanistic explanation should help students learn. Indeed, in our earlier study on acid–base reactions, we found 54 that students who constructed causal mechanistic explanations also had the highest success in drawing an appropriate curved arrow mechanism.14 The Value of Longitudinal Studies in CER When students advance into organic chemistry and beyond, instructors expect that they are bringing foundational general chemistry knowledge with them and expect that they will know when and how to invoke this knowledge. In fact, colleges and universities structure degree-plans in many disciplines in such a way that entry-level courses are prerequisite to the advanced upper level courses. This is especially true in chemistry, where most courses past general chemistry have prerequisites. While it makes sense that students must learn basic concepts first, so they can build on them to learn more complex ideas in later courses,30,31 there is scant research on how students carry basic ideas forward to the next set of courses. Similarly, while much work has been done on characterizing student alternate ideas or misconceptions, we know little about how these ideas change as students move throughout the curriculum. That being said, what research there is seems to indicate that even graduate students in chemistry may have persistent and problematic understanding of chemistry ideas.23,32 For example, Bodner and Bhattacharyya report that chemistry graduate students are unable to use electron pushing arrows in a predictive manner.23 In our work on drawing Lewis structures we found that organic chemistry students were little better than general chemistry students at drawing structures, and that upper-level and graduate students were no more likely to understand that structures can be used to deduce information about physical and chemical properties.32 In fact, the lack of longitudinal studies was noted in the National Academies report on discipline based education research (DBER), along with the need for more such studies.33 Well-constructed longitudinal studies have the potential to elicit evidence of long-term impacts of curriculum, interventions, or other factors since these impacts may only become apparent weeks, months, or years later.34 White and Arzi define longitudinal studies as “… a study in which two or more measures or 55 observations of a comparable form are made of the same individuals or entities over a period of at least one year.”34 The chemistry education research (CER) community is responding to the call for more longitudinal studies that follow cohorts of students through two semesters of a given course and gather data via a pretest and posttest.35–37 However, because of student enrollment patterns, it is far less common and much more difficult for researchers to continue studying a phenomenon for two or more years. Typically, one must begin with a very large initial cohort to have a chance of retaining enough students for meaningful study by the end of the project.38–41 In our own prior work, we have explored how student understanding of structure–property relationships42 and intermolecular forces (IMFs)43 changes over two years from the beginning of general chemistry to the end of organic chemistry. We were able to show that students who learned general chemistry in a transformed general chemistry curriculum were much more likely to make connections between a chemical structure and its macroscopic properties, and that this difference was maintained throughout organic chemistry.42 Similarly, in a study on student understanding of IMFs, students from the transformed courses were significantly more likely to represent IMFs as forces operating between molecules compared to students from a traditional curriculum, and this difference was maintained through another year of organic chemistry.43 Polytomous Assessments Can Provide Longitudinal Information about Student Reasoning Most assessment instruments used at the college level are dichotomous, that is the answer is scored as either right or wrong, which means that many of the nuances of student understanding are lost. Particularly when students are constructing explanations and arguments, the types of responses can vary widely, as do the ideas and mechanisms invoked. The analyses of student responses to the Brønsted–Lowry acid–base prompt used in this paper allows us to differentiate between the ideas and mechanisms used in the response and as such the characterizations represent increasingly sophisticated explanations of acid–base reactions—there are several possible codes and therefore these assessment 56 instruments are polytomous. By using this approach (rather than items that are scored right or wrong) we are able to investigate how students’ ideas change over time. The present study is an extension of our previously published research on reasoning about acid– base reactions with general chemistry students where we: (i) developed a causal mechanistic reasoning framework; (ii) developed a prompt structure to elicit causal mechanistic responses: and (iii) developed a coding scheme that allowed us to identify increasingly sophisticated responses.14 In this study, using this causal mechanistic explanation framework and the prompt structure, we investigate how organic chemistry students respond to the same prompt, and to a new prompt that asks about a Lewis acid– base reaction that does not involve the more familiar proton transfer. The research questions that were guided this study were: 1. How does student reasoning change over time from the end of general chemistry to the end of organic chemistry for both Brønsted and Lewis acid–base reactions? 2. What is the effect of students’ prior general chemistry experience on their reasoning and ability to draw mechanistic arrows? Methods Student Participants These studies were performed at a large Midwestern research intensive university. All students were informed of their rights as human research subjects and all data was obtained and handled in accordance with the Institutional Review Board. There are four groups of student participants in this study: their background and time of data collection are summarized in Table 4.1. All students were recruited via email as approved by the relevant instructor of record and completed the assignment for extra credit. 57 Identifier in Paper CLUE–GC Cohort A Cohort B Cohort C Description of Cohort—Students in These N Semester Data Gathereda Groups Completed: End of GC2 (Spring 2015) Start of OC1 (Fall 2015) End of OC2 (Spring 2016) A CLUE general chemistry 2 course A CLUE general chemistry 2 course A more selective general chemistry 2 course Transferred general chemistry 2 credit from another institution or didn’t take a general chemistry 2 course 107 92 48 54 X — — — — X X X — X X X aAll analyses were performed using SPSS. The full statistical data are provided in the Supporting Information in the Appendix. Table 4.1. Research Design Comparing Data from Students in Four Cohorts on Several Demographic and Academic Measures. CLUE–GC (N = 107) students were enrolled in a transformed general chemistry 2 course (Chemistry, Life, the Universe and Everything—CLUE44) in 2015. The data presented here were discussed in our prior work14 and are used here to show a progression of reasoning from the end of general chemistry 2 (GC2) through organic chemistry 2 (OC2). These students are representative of the whole CLUE cohort, but in spring 2015 the assessment item was only administered to these 107 students. The other three groups of students reported in this study were enrolled in, and completed both, semesters of a traditional (using a published text13 and homework) sophomore-level organic chemistry during the fall 2015 semester (OC1) and the spring 2016 semester (OC2). Out of the 674 students who completed both assessment items administered at the beginning of OC1 and at the end of OC2, 200 students were randomly selected. These students were then characterized by the nature of the GC2 course they completed, and three main GC2 course work pathways emerged for these students. Almost half of the students had completed the CLUE transformed general chemistry course for their GC2 course work (N = 92). That is, they were enrolled in the same course as the CLUE–GC group who were the focus of our previous paper.14 While these students are not the same subpopulation of students as CLUE–GC, 58 they are comparable using various academic and demographic measures (see Appendix). These students will be referred to as Cohort A (N = 92). Originally, we envisaged that the remaining students would form a single cohort, but on further analysis we saw that there were two different groups of students: Cohort B (N = 48) had been enrolled in a more selective general chemistry sequence: these students include honors students, chemistry majors, and students who were enrolled in a self-selected residential college program. Cohort C (N = 54) was composed of students who either transferred credit for GC2 (and we therefore did not know what type of general chemistry experience they had been exposed to), or students who had not taken GC2 (at this institution GC2 is not a prerequisite for OC1). There were also six students who had taken a previous version of general chemistry several years earlier. Since these six students did not readily fall into any group, we did not use their data for this study, leaving us with 194 total students in our comparison groups. Summaries of all statistical analyses of demographic and academic measures are reported in S2 and S3 of the Supporting Information. Comparisons of the background information between all the cohorts showed that except for the instances discussed below, there were no significant differences. Cohort A had earned somewhat higher grades than CLUE–GC in GC2, (means of 3.3 vs. 2.9, U = 3617.0, z = –3.202, p = 0.001, r = 0.23 [small effect size]). As one might expect, students who continued on into organic chemistry were more successful in general chemistry. A comparison of the three organic chemistry cohorts A, B, and C showed that Cohort B had slightly higher ACT scores than Cohort A (U = 1555.5, z = -2.548, p = 0.011, r = 0.22 [small effect size]) and Cohort C (U = 900.0, z = –2.145, p = 0.032, r = 0.21[small effect size]). Since Cohort A had elected to take more selective general chemistry courses, one might expect that they would have higher incoming ACT scores. Cohort C had a significantly lower GPA at the start of OC1 than both Cohort A and B, with medium effect sizes. At the end of OC2, Cohort B 59 and Cohort C also differed on their OC2 grade with a small effect size (see Appendix). Cohorts A and B did not differ on the final OC2 grade.45,46 Description of Assessment Tasks The assessment prompt was designed specifically to elicit student causal mechanistic reasoning about a given acid–base reaction. In this paper, we report on student responses to the previously reported reactions of HCl with H2O and the Lewis acid–base reaction of NH3 with BF3, which has not been previously reported. As in our earlier study, we structured each assessment task into four parts. First, students were presented with Lewis structures of the reactants and the products for the given reaction and asked to classify the reaction and explain their classification (Figure 4.1a). Next, students were asked to describe what is happening at the molecular level (Figure 4.1b). The prompt then asked students to explain why the reaction occurs using a molecular level explanation (Figure 4.1b). Finally, the students were provided Lewis structures of the reactants and product(s) and asked to draw the mechanistic arrows to indicate how the reaction occurs (Figure 4.1c). It is important to note that the prompt asking students to “describe what is happening…” and the prompt asking students to “explain why the reaction occurs…” are separated in the prompt structure and students are provided two separate boxes to respond to these prompts separately. Based on our previous work,14 we know that by asking these questions separately, students recognize that explaining why is different than describing what. 60 How would you classify this reaction? Please explain why you chose that classification. Can you describe in full detail what you think is happening on the molecular level for this reaction? Specifically, discuss the role of each reactant. Using a molecular level explanation, please explain why this reaction occurs? Specifically, why the reactants form the products shown. For the following reaction, please draw arrows in the BLUE box to indicate how the reaction occurs. Figure 4.1. Assessment prompts administered using beSocratic for the reaction BF3 with NH3. An identical prompt structure was used for the reaction of HCl with H2O. Data Collection Data analyzed in this study were collected via the online homework and research platform beSocratic.43,47–49 These data were in the form of written student explanations and drawn mechanistic arrows. The same prompt structure was used to elicit student knowledge about two acid–base reactions—the reaction of HCl with H2O, which has been previously reported for CLUE–GC14, and the reaction of NH3 with BF3 being reported here for the first time. These prompts were administered to CLUE–GC at the end of GC2 in spring 2015, and to students in the organic chemistry sequence, once at the start of OC1 in fall 2015 and once at the end of OC2 in spring 2016 as shown in Table 4.1. Both reactions were administered each time and the prompts were identical in each administration. It should be noted that while the students answered these prompts at the start of OC1 and again at the end of OC2, the answers were not provided to them. The activity was administered to 107 CLUE–GC students in spring 2015, 763 OC1 students in fall 2015 with an 83% response rate, and then again to OC2 to 674 students in spring 2016 with a 92% response rate. Data from 194 randomly chosen organic chemistry 61 students who completed both assignments were analyzed, and these students made up organic chemistry Cohorts A, B, and C. The student responses were anonymized and coded without knowledge of which cohort they belonged to. Prompt 1: HCl and H2O Data Analysis The written student responses to the HCl with H2O prompt were analyzed using the published causal mechanistic reasoning coding scheme (Table 4.2).14 Student responses that only discuss the observation that a bond was breaking or forming were characterized as General Descriptive (GD). Some descriptive explanations were closely aligned with the Brønsted acid–base definition (e.g., they identified the proton donor and/or proton acceptor and explicitly identified the reaction species) but still did not discuss electrostatic interactions or explicit electron movement. We characterized these types of responses as Brønsted Descriptive (BD). Responses that provided Brønsted acid–base causal reasoning including discussion of polarity and electrostatic interaction were characterized as Brønsted Causal (BC). Responses that provided a Lewis acid–base explanation that included discussion of electron activities was characterized as Lewis Mechanistic (LM) and often incorporated a Brønsted-like explanation as well. Ideally, students would provide Lewis acid–base casual reasoning discussing both polarity and electron movement characterized as Lewis Casual Mechanistic (LCM). Two of the authors (MMC) and (HK) who were involved in the development of the scheme and who had previously coded responses for the earlier report,14 both coded randomly chosen responses that were not part of the data set and obtained a Cohen’s Kappa of 0.9. Next, one author (HK) coded 388 explanations to the HCl/H2O prompt collected from organic students (194 for OC1 and 194 for OC2). Spot checks of 20 randomly chosen responses by other authors showed 100% agreement. As with our previous work on this reaction and coding scheme, the student responses for “describe what is 62 happening…” and “explain why…” were analyzed together since students sometimes responded to why in the what textbox and vice versa. Student responses to “classify this reaction” were only analyzed when additional context was needed to make sense of student responses to describe what and explain why. Student mechanistic arrows were reviewed separately from the written responses and were coded as described in the previous work: that is if (i) the first arrow was drawn from the lone pair on the oxygen in water to the hydrogen in HCl; and (ii) the second arrow was drawn from the bond between the hydrogen and the chlorine atom in HCl to the chlorine atom. Any other variations of arrow drawings were coded as incorrect. 63 Characterization Scheme No Response (NR): No answer or their explanations were unreadable or incomprehensible. Non-Normative (NN): Students provide non- normative or unrelated explanations. In addition, students do not recognize it is an acid–base reaction and instead attribute the mechanism to other types of reactions or other macroscopic observations. General Descriptive (GD) (what): Students provide scientifically simplistic description and may discuss bond breaking or forming. Brønsted Descriptive (BD) (what): Students provide Brønsted acid–base explanation including identification of acid and/or base and discussion of proton transfer. Brønsted Causal (BC) (what and why): Students provide Brønsted acid–base causal reasoning that includes discussion of polarity of one or both of the reactants. Lewis Mechanistic (LM) (what and how): Students provide Lewis acid–base explanation, including role of lone pair (may also encompass the Brønsted explanation). Lewis Causal Mechanistic (LCM) (what, how, and why): Students provide Lewis acid–base causal reasoning that includes discussion of polarity of one or both of the reactants (may also encompass the Brønsted explanation). Examples Viktor: “I do not really have a reasoning.” Raymond: “The hydrogen on the HCl is donating its electrons to the oxygen on the water.” Catherine: “The acid is reacting with the base and the acid is a proton donor while the base is a proton acceptor.” Heather: “The HCl is the acid meaning it is a proton donor and the water is the base meaning it is a proton acceptor. At the molecular level the hydrogen from the HCl is breaking off and the water is gaining it forming H3O+.” Claire: “The oxygen atom in water bonds to the hydrogen atom in hydrochloric acid as the hydrogen and chlorine atom break apart. The partial negative oxygen in water is attracted to the partial positive hydrogen in hydrochloric acid. When the oxygen and hydrogen form a bond the hydrogen and chlorine break their bond creating the products H3O+ and Cl-.” Jackie: “The O in the H2O gives its electrons to the H in the HCl bond, and simultaneously the HCl bond breaks, placing those electrons onto the Cl. This reaction happens because it is more favorable.” Francis: “The lone pair on the water molecule attracts the Hydrogen from HCl. The H–Cl bond is broken and forms a new bond with oxygen. The reaction occurs because the partial negative charge on oxygen attracts the partial positive charge on the hydrogen. The bond between the Hydrogen and Cl is less strong than the bond that forms between hydrogen and oxygen.” aSee ref 14. Table 4.2. Publisheda Characterization Scheme for the Reaction of HCl and H2O. 64 Prompt 2: NH3 and BF3 Since student reasoning about the reaction of NH3 with BF3 should be similar to that of HCl/H2O, a modification of the previously established HCl/H2O coding scheme was developed (Table 4.3). As with HCl/H2O, we were able to identify different ways in which students responded to the prompt, with the difference being that only the Lewis model of acid–base reactivity is appropriate for the reaction. Responses that simply described what was shown in the reaction scheme were categorized as Descriptive General (DG). These responses identified the formation of the bond between the nitrogen atom and the boron atom but did not discuss electron movement nor provide any causal reason about why the reaction occurs. Aaron displayed this type of reasoning when he wrote “Two compounds come together to make one new compound.” Responses that discussed the electrostatic attraction between the nitrogen and the boron but omitted any discussion of electron movement were characterized as Descriptive Causal (DC) as exemplified by Casey: “The boron is electron deficient and is attracted to the nitrogen.” Responses that displayed evidence of understanding that the lone pair of electrons on nitrogen form a bond with boron but did not discuss electrostatic attraction were characterized as Descriptive Mechanistic (DM) since students were invoking the Lewis acid–base model of reactivity in their reasoning. Tony’s reasoning fit this characterization when he said “I believe that is because the boron has an available space/orbital around it that will allow the lone pair from the nitrogen to bond. Because N and B are both neutral, the bonding causes the nitrogen to have a positive charge, and the boron negative charge.” Finally, responses that included reasoning about the mechanism in terms of electron movement and also explicitly discuss the attraction of the lone pair to the boron atom were characterized as Causal Mechanistic (CM). Timothy’s response “The lone pair of electrons on the nitrogen attacks the partial positive boron which creates a new shared bond between them” includes 65 both of the necessary elements. The characterization scheme of student reasoning for the reaction of BF3 with NH3 is shown in Table 4.3. Three of the authors (MMC, HK, OMC) coded a random sample of 20% of the 388 student responses (194 from OC1 and 194 from OC2) to establish inter-rater reliability, resulting in pairwise Cohen’s Kappa values above 0.7 to establish the coding scheme. To finish coding the other 80% of the data, one of the authors (OMC) worked to train two undergraduate coders. Each trained coder obtained Cohen Kappa values ranging from 0.69 to 0.93 with the author (OMC). These two trained coders coded sets of 75–100 responses and their results were compared to each other to ensure accuracy. In the case of any discrepancies between the two trained undergraduate coders, the author (OMC) and the two trained coders would discuss until consensus was reached. 66 Characterization Scheme No Response (NR): Student does not provide an answer, or explanations are unreadable or incomprehensible. Non-Normative (NN): Student attributes the mechanism to other types of reactions or other macroscopic observations. Descriptive General (DG) (what): Student provides a scientifically simplistic description of bond formation. Descriptive Causal (DC) (what and why): Student provides an explanation that discusses the electrostatic attraction of the species. Descriptive Mechanistic (DM) (what and how): Student provides a Lewis acid–base explanation that explicitly discusses electrons and their movement. Causal Mechanistic (CM) (what, why, and how): Student provides a causal and a mechanistic explanation for the reaction. Student Examples — Kate: “The acid, the NH3 is accepting electron pair from BF3 then they come together due to ionic bond.” Rachel: “The nitrogen bonds to the boron to make the new complex.” Casey: “The boron is electron deficient and is attracted to the nitrogen.” Andrew: “The partially negative nitrogen is pulled to the boron.” Tony: “I believe that is because the boron has an available space/orbital around it that will allow the lone pair from the nitrogen to bond. Because N and B are both neutral, the bonding causes the nitrogen to have a positive charge, and the boron negative charge.” Michelle: “Boron has a vacant orbital in which the lone electrons on the N can form a bond.” Mary: “The lone pair in the NH3 is able to give its electrons to the B in BF3. It acts as a nucleophile and is partially negative while the B is partially positive.” Timothy: “The lone pair of electrons on the nitrogen attacks the partial positive boron which creates a new shared bond between them.” Table 4.3. Characterization Scheme for the Reaction of NH3 with BF3. The mechanistic arrows for the reaction of NH3 with BF3 were coded in a similar way to the previously described HCl/H2O prompt. The mechanistic arrow was considered correct if the arrow began at the lone pair on nitrogen and ended at the boron. Drawings that included backwards arrows or any extraneous arrows were considered incorrect. 67 Results We have organized our findings by our two research questions: (i) How does student reasoning change over time from the end of general chemistry to the end of organic chemistry; and (ii) What is the effect of the prior general chemistry experience on student reasoning? We will address the results for each type of reaction (HCl + H2O and NH3 + BF3) within each research question. RQ 1: How Does Student Reasoning Change over time from the End of General Chemistry to the End of Organic Chemistry for Both Brønsted and Lewis Acid–base Reactions? Finding 1a: All Three Organic Cohorts Improved throughout Two-Semesters of Organic Chemistry. HCl + H2O In general, all students’ reasoning, regardless of general chemistry preparation, became more sophisticated over the course of two semesters of organic chemistry. Figure 4.2 shows the classification of student reasoning for Cohorts A, B, and C both at the start of OC1 and at the end of OC2. Since we are comparing how reasoning changed from one time point to the next for the same group of students, a McNemar test for repeated measures45 was used to analyze the change in the proportion of students who transitioned from a Non-Lewis Causal Mechanistic characterization to Lewis Causal Mechanistic from the start of OC1 to the end of OC2. For Cohort B and Cohort C there is a noticeable shift from General Descriptive (GD) responses to Lewis Causal Mechanistic (LCM) by the end of OC2. At the start of OC1, only 15% of Cohort B and 11% of Cohort C participants gave a Lewis Causal Mechanistic (LCM) response. By the end of OC2, 40% of students in both of these cohorts gave an LCM response. These shifts from a Non-LCM response to a LCM response from the start to end of organic chemistry was significant for Cohorts B and C (p = 0.012 for Cohort B and p = 0.001 for Cohort C). Students in Cohort A improved from 43% of students giving a Lewis Causal Mechanistic response at the start of OC1 to 58% at the end of OC2 (p = 0.043). Recall that these students all completed the same organic chemistry course with the same instructor and the primary difference between these three cohorts is their general 68 chemistry preparation. We will report our observed effects of general chemistry course experience in RQ 2. Figure 4.2. The characterization of student explanations for HCl + H2O for Cohorts A, B, and C at the start and end of organic chemistry. Exact percentages are listed in S4 in the Supporting Information. No Response (NR), Non-Normative (NN), General Descriptive (GD), Brønsted Descriptive (BD), Brønsted Causal (BC), Lewis Mechanistic (LM), Lewis Causal Mechanistic (LCM). NH3 + BF3 As shown in Figure 4.3, a similar but less marked pattern emerges for the reaction of NH3 + BF3. All of the cohorts shift from a descriptive explanation to a causal mechanistic explanation, but only the change for Cohort C is significant (p < 0.001). 69 Figure 4.3. The characterization of student explanations for NH3 + BF3 for Cohorts A, B, and C, at the start and end of organic chemistry. Exact percentages are listed in S10 in the Supporting Information. No Response (NR), Non-Normative (NN), Descriptive General (DG), Descriptive Causal (DC), Descriptive Mechanistic (DM), Causal Mechanistic (CM). Finding 1b: A Comparison of the pattern of responses for the two reactions shows that students are more likely to provide a mechanistic explanation for NH3 + BF3 than for HCl + H2O Even at the Start of OC1. Although the coding schemes are somewhat different for the two reactions, there are some comparisons that can be made. At the start of OC1, for Cohorts B and C the most prevalent type of explanation for HCl + H2O is simply a description of what is happening—that is, a proton is being transferred from the acid to the base (BD), followed by the BC explanation where students indicate why the interaction occurs. Together these account for over 50% of Cohort B and C explanations, whereas explanations that invoke movement of electrons (LC and LCM) account for between 20 and 30% of the explanations. For Cohorts B and C at the beginning of OC1, this reaction does not seem to activate ideas about the involvement of electrons. In contrast, for the Lewis acid–base reaction over 80% of students invoke the involvement of electrons (DM and CM) during the reaction for all three cohorts. 70 By the end of OC2 for all cohorts the explanations for HCl + H2O have shifted to between 70 and 80% mechanistic (LC and LCM), similar to the NH3 + BF3 responses. At the end of OC2 the responses for Cohorts B and C are comparable between the two reactions. That is, enrollment in organic chemistry appears to help students use mechanistic (but not necessarily causal mechanistic) thinking. We will discuss the differences between Cohort A, B, and C in the results for RQ 2. Finding 1c: CLUE students retain their reasoning ability from the end of general chemistry to the Start of Organic Chemistry. HCl + H2O We have previously reported findings on causal mechanistic reasoning for students at the end of a CLUE general chemistry course (CLUE–GC), and here we compare those findings to Cohort A. As shown in Figure 4.4, there is little difference between these two groups, this, despite the fact that there was a several-month gap between the two data collections. As shown in Figure 4.4, the major category of explanation at both time points was Lewis Causal Mechanistic (LCM), and the pattern of responses is quite similar with 41% of students in the CLUE–GC cohort and 43% of students in Cohort A providing LCM explanations (2(1) = 0.113, p = 0.737). These data from CLUE–GC students seem to belie the common complaint from faculty that student knowledge tends to decay over the summer and valuable time must be wasted at the start of OC1 to review GC material. 71 Figure 4.4. The classification of student explanations for the reaction of H2O + HCl. These students were enrolled in a CLUE–GC2 course but were given the assessment item at different times. We also compared the number of students who can draw the correct mechanistic arrows at both time points (Table 4.4). At the end of general chemistry 71% of students (CLUE–GC) were able to draw both arrows of the mechanism correctly, while at the beginning of OC1 the percentage fell slightly to 59%. The difference between these two sets of data at different timepoints is not significant (2 (1) = 3.321, p = 0.068); that is, there was little decline in students’ mechanistic arrow drawing ability. Cohort Time CLUE–GC (N = 107) Cohort A (N = 92) End of GC2 Start OC1 Answers, % Correct 71 59 Incorrect 29 41 Table 4.4. Comparison of Percentage of Correct Mechanistic Arrow Drawings for the Reaction of H2O + HCl. 72 NH3 + BF3 A comparison of the data from CLUE–GC with Cohort A for NH3 + BF3 again showed that there was little difference between the end of GC2 and the start of OC1 (2 (1) = 0.394, p = 0.530) (Figure 4.5). Students in CLUE–GC were already quite successful drawing the correct mechanistic arrow for this process (77% correct) and that percentage grew to 93% at the end of OC2 (2 (1) = 3.481, p = 0.062) (see Table 4.5). Figure 4.5. The classification of student explanations for the reaction of NH3 + BF3. These students were enrolled in a CLUE–GC2 course but were given the assessment item at different times. Cohorta Time Answers, % Correct 77 Incorrect 23 End of GC2 CLUE–GC (N = 107) Cohort A (N = 92) aThese students all took CLUE for GC2 but were given the assessment item at different times. 13 Start OC1 87 Table 4.5. Comparison of Percentage of Correct Mechanistic Arrow Drawings for the Reaction of NH3 + BF3. 73 RQ 2: What is the effect of students’ prior general chemistry experience on their reasoning and ability to draw mechanistic arrows? Finding 2a: Cohort B and Cohort C gave similar responses regardless of General Chemistry 2 Course experience. An inspection of Figures 4.2 and 4.3 show that the pattern of responses for Cohorts B and C are similar to each other. At the start of OC1, the most common response for HCl + H2O for both Cohorts B and C was Brønsted Descriptive (29 and 35%, respectively). The similarity between Cohorts B and C also extends to the other types of reasoning: Brønsted Causal (23 and 22%, respectively), Lewis Mechanistic (19 and 13%, respectively), and Lewis Causal Mechanistic (15 and 11%, respectively). Indeed, the difference in proportions of Lewis Causal Mechanistic responses compared to Non-Lewis Causal Mechanistic responses for Cohorts B and C at the start of OC1 is not significant (2(1) = 0.275, p = 0.600). These data suggest that students who took a “selective” GC2 course (Cohort B), who had higher ACT scores, and higher OC1 and OC2 grades, actually began organic chemistry with similar ability to explain a simple Brønsted acid–base reaction as students who did not take a GC2 course at all or transferred an equivalent credit into the university (Cohort C). Similarly, by the end of OC2, these cohorts did not appear different in their ability to reason about a simple proton transfer (2(1) = 0.014, p = 0.905) (Figure 4.2). It is encouraging that both cohorts improved their ability to reason about the reaction over two semesters of organic chemistry, however, Cohort B did not outperform Cohort C as one might have expected. We observed the same pattern of performance between Cohorts B and C for the reaction of NH3 + BF3 at the start (2(1) = 1.732, p = 0.188) and end (2(1) = 0.274, p = 0.601) of organic chemistry. We therefore combined Cohorts B and C to simplify data visualization and statistical comparisons from this point on. From now on we refer to this combined cohort as Cohort B + C (N = 102). 74 Finding 2b: Students in Cohort A were more likely to provide causal mechanistic reasoning than those in Combined Cohort B + C HCl + H2O At the beginning of OC1 we see the performance of Cohort A is quite different from the combined Cohort B + C (Figure 4.6). The major response category for Cohort A is Lewis Causal Mechanistic, while for Cohorts B + C the major category is Brønsted Descriptive (Figure 4.6). Combining Cohorts B and C allowed us to compare the two groups (Cohort A (N = 92) and Cohorts B + C (N = 102) using a Chi-square analysis. We first compared the percent of Lewis Causal Mechanistic codes to the sum of all the other codes (all other Non-Lewis Causal Mechanistic codes). These analyses indicate that there are significant differences between the two groups both at the start (2 (1) = 23.010, p < 0.001) and at the end (2 (1) = 5.872, p = 0.015) of two semesters of organic chemistry. By the end of OC2, the percentage of students providing a Lewis Causal Mechanistic response from Cohort A (58%) was still higher than Cohort B + C (40%). In fact, the percentage of students from Cohort B + C providing Lewis Causal Mechanistic responses is lower at the end of OC2 than the percent from Cohort A was at the beginning of OC1 (43%). Comparing the change over time from the start of OC1 to the end of OC2, a McNemar test for repeated measures was used to analyze the proportion of students who transitioned from a Non-Lewis Causal (43%) Mechanistic to a Lewis Causal Mechanistic (58%) response was significant both for Cohort A (43–58%) (p = 0.043) and for Cohort B + C (13–40%) (p < 0.001). Students in Cohort A were also more likely to explicitly reason about electron movement at the start of OC1 meaning they gave a Lewis Mechanistic (19%) or a Lewis Causal Mechanistic response (43%). Less than 30% of Cohort B+C reasoned about electron movement at all at the start of organic chemistry. This difference is significant (2(1) = 20.713, p < 0.001, Cramer’s V = 0.327, medium effect size45-46) but is fades by the end of OC2 (2(1) = 1.411, p = 0.235). While it is encouraging that Cohort B+C began to incorporate more mechanistic 75 thinking into their reasoning, their lack of causal reasoning seems to be a defining difference between these two groups by the end of OC2. This is particularly interesting as all students in Cohorts A, B, and C had the same organic chemistry course and instructor. Figure 4.6. The classification of student explanations for the reaction of H2O + HCl. These students had different GC2 experiences but were given the assessment item at the start of OC1 and the end of OC2. NH3 + BF3 Again, there is a larger percentage of Cohort A students who also invoke a Causal Mechanistic response involving electrostatic interactions relative to students enrolled Cohort B + C who do likewise (Figure 4.7). A Chi-square analysis of responses from the two cohorts (A versus B + C) shows a significant difference between the two groups at the start of OC1, (2 (1) = 9.193, p = 0.002, Cramer’s V = 0.218, small effect size45,46). By the end of OC2, in contrast to the HCl and H2O prompt, there is no significant difference between the cohorts of students (2 (1) = 0.588, p = 0.433). As one might expect, Cohort B + 76 C’s improvement from the start of OC1 to the end of OC2 was significant (p < 0.001) as it was for the HCl and H2O reaction. As discussed in Finding 1a above, almost all students from both cohorts invoke mechanistic reasoning (meaning Descriptive Mechanistic or Causal Mechanistic) at the start of OC1 and end of OC2 for this Lewis-only acid–base reaction. Comparing the proportion of mechanistic responses (DM and CM) to all non-mechanistic codes, we found no difference between the Cohort A and Cohort B+C at the start of OC1 (2(1) = 0.062, p = 0.804) nor at the end of OC2 (2(1) = 0.241, p = 0.623). Figure 4.7. The classification of student explanations for the reaction of NH3 + BF3. These students had different GC2 experiences but were given the assessment item at the start of OC1 and the end of OC2. Finding 2c: Cohort A Students were better at drawing mechanistic arrows. As recorded in Table 4.6, all students improved in their ability to draw mechanistic arrows over time, but Cohort A students were better at this task than Cohort B + C both at the beginning (59–15%; 2(1) = 40.845, p < 0.001, Cramer’s V = 0.459, a medium effect size46) and at the end of organic chemistry (88–75%; 2(1) = 5.044, p = 0.025, Cramer’s V = 0.161, a small effect size46). The gap between 77 the cohorts narrowed over time but did not diminish. The difference between the two groups does seem to parallel the type of explanations that each provided. As in our earlier study, students who provide causal mechanistic explanations are more likely to be able to draw appropriate mechanistic arrows. Time Start OC1 End of OC2 Cohorta Cohort A (N = 92) Cohort B + C (N = 102) Cohort A (N = 92) Cohort B + C (N = 102) Answers, % Correct Incorrect 59 15 88 75 41 85 12 25 aThese students had different GC2 experiences but were given the assessment item at the start of OC1 and the end of OC2. Table 4.6. Comparison of Percentage of Correct Mechanistic Arrow Drawings for the Reaction of H2O + HCl. NH3 + BF3 A comparison of the mechanistic arrow drawings for NH3 + BF3 shows that there is little difference between students’ drawings of mechanistic arrows (Table 4.7). Even at the start of OC1, the majority of all students are able to draw the one arrow that would indicate the formation of the Lewis acid–base complex as shown in Figure 4.8. It is clearly a much easier task for most students than the Brønsted acid–base reaction. Figure 4.8. An example of a correct arrow drawing for the reaction of NH3 + BF3. 78 Time Cohorta Answers, % Correct Incorrect Start of OC1 End of OC2 Cohort A (N = 92) Cohort B + C (N = 102) Cohort A (N = 92) Cohort B + C (N = 102) 87 72 93 88 13 28 7 12 aThese students had different GC2 experiences but were given the assessment item at the start of OC1 and the end of OC2. Table 4.7. Comparison of Percentage of Correct Mechanistic Arrow Drawings for the Reaction of NH3 + BF3. Discussion This study contributes to our overarching research goals that involve determining how student reasoning develops, how that reasoning can be elicited by appropriately designed prompts, and ultimately how a focus on causal mechanistic reasoning can support student learning in chemistry. In this study we were able to replicate our original finding (that students at the end of a CLUE general chemistry course tend to provide Lewis causal mechanistic explanations)14 with a different group of students (Cohort A–OC) in the next course of the sequence. It is encouraging that many of these students were able to provide a sophisticated causal mechanistic explanation, even after a summer break. Often faculty complain that students do not seem to remember material that they have learned in earlier courses, but in this case we see little difference between the data collected at end of GC2 in spring 2015 and the start of OC1 in fall 2015. We see a similar pattern the same time period for the reaction of NH3 and BF3, which is clearly more recognizable as a Lewis acid–base reaction. Since we did not have access to students from Cohorts B and C, during their general chemistry experience, we must confine our remarks to what they know at the beginning of OC1. While Cohorts B and C have had very different experiences in GC2, they look remarkably similar in their responses to the 79 questions of how and why HCl and H2O react. That is, both students from highly selective courses and students who have had no GC2 experience tend to provide descriptions of the acid–base reaction using the Brønsted model, rather than a causal mechanistic explanation that invokes a Lewis acid–base model. We believe that the difference between the CLUE cohort (Cohort A) and the others is a function of their general chemistry experiences. Traditional general chemistry courses typically do not include such scientific practices as construction of models, arguments, and explanations, whereas the CLUE curriculum is built around such use of knowledge. While it is unlikely students would learn to use scientific practices without significant support and practice, it is encouraging that all the cohorts improve as they move through the two semesters. That is, exposure to ideas about reaction mechanisms in an organic chemistry context does improve students’ ability to reason about simple acid– base reactions. However, there is still a significant difference between Cohort A and Cohort B + C at the end of OC2. Similarly, student ability to draw mechanistic arrows for this simple reaction improves over the course of two semesters, although again Cohort A–OC performs better both at the beginning and at the end of OC2 than the others. Just as in our prior studies,14,26 the ability to draw mechanistic arrows seems to be correlated with the use of causal mechanistic reasoning to explain how the reaction occurs. There are numerous studies highlighting the difficulties that students have with drawing appropriate mechanistic arrows, and these findings seem to support the idea that causal mechanistic reasoning should be an explicit component of chemistry courses. It should be noted, however, that the ability to understand and draw mechanisms for simple acid–base reactions is not reflected in an increase in overall grades for Cohort A relative to Cohort B. One might imagine that the ability to explain and draw mechanisms would result in an improvement in organic course grades. However, just as with many organic course examinations (including the ACS examination and many “elite” chemistry departments) the examinations for this course did not explicitly address such mechanistic reasoning.27 By the end of 80 the second semester there was no difference in course grades for Cohorts A and B, though Cohort C was slightly lower. Whatever the course examinations are measuring, most students end up with equal facility regardless of their background. However, if an organic course were transformed such that students were explicitly required to engage in scientific practices such as constructing models, arguments, and explanations, and to incorporate mechanistic reasoning, we might find that students who have already developed those habits of mind and approaches would be better prepared to engage with them. We are currently developing and testing such a transformed organic chemistry curriculum and will report on our findings as we move forward. The results from the reaction that is more recognizable as a Lewis acid–base reaction (NH3 + BF3) also show that students tend to move toward a causal mechanistic explanation over the two semesters of organic chemistry. It is interesting that a greater proportion of students in Cohort B + C–OC tend to discuss the involvement of electrons from the beginning than they did for the Brønsted acid–base reaction. Because the reaction of NH3 with BF3 is typically introduced in the context of Lewis acids and bases, it is likely that this prompt activates resources aligned with the Lewis acid–base model, which requires that students discuss the involvement of electrons. Clearly most of the students have grasped the idea of a Lewis acid–base reaction and are also able to draw an appropriate arrow to denote the mechanism. Implications for Teaching and Further Research The results of this study show that students who have taken a curriculum that emphasizes causal mechanistic reasoning are indeed more likely to be able to employ such reasoning when prompted to do so and are also more likely to draw appropriate mechanistic arrows. While an emphasis on causal mechanistic reasoning about phenomena may not be an explicit goal of most chemistry courses, it is certainly implicit for organic chemistry. Most organic faculty emphasize that the construction of an electron pushing mechanism begins at a source, and ends at a sink, but perhaps what 81 is missing (or that students are unable to incorporate into their thinking) is an explicit emphasis on the cause of this electron movement. (Indeed, perhaps the arrow formalism should be renamed “electron pulling” rather than “electron pushing”, to emphasize the attraction between electrophile and nucleophile.) Numerous studies describe the difficulties that students have with drawing such mechanisms, but there are few that show improvements. Analysis of many organic examinations shows an emphasis on drawing a correct mechanism, but not on reasoning about how and why that mechanism is drawn that way.27 We believe that an emphasis on asking students to articulate how and why a chemical phenomenon occurs provides them with the cognitive tools to use as they construct causal mechanistic explanations. These practices are emphasized in the CLUE curriculum which has been extensively discussed elsewhere.44 As the curriculum builds from interactions of atoms to networked biological reactions, students are asked to think about how and why these chemical phenomena occur. For example, questions such as “why do neutral atoms attract each other?” and “why do neutral molecules attract each other?” are used to drive instruction. Students are asked after almost every class to construct models and explanations using the online homework system beSocratic (as well as answer more traditional items such as calculations and skill development). Students become used to answering such questions, and we believe such repetition may help them to develop a set of cognitive tools that can be brought to bear on other problems. However, it may be that CLUE students simply are able to reproduce responses to familiar questions, rather than developing resources that might help them answer different questions. Clearly there is much work to be done in this area to determine whether students develop knowledge that is useful in other contexts, but there is some evidence from another study to support the idea that students are at least thinking about mechanisms more broadly.50 Students who were concurrently enrolled in both a CLUE general chemistry course and a molecular biology course that was also undergoing transformation to focus on core ideas51 were asked whether there were 82 recurring ideas or “themes” in each course. In chemistry one of the ideas that students discussed was “structure–property relationships”, and for biology a similar theme was “structure–function”. Of the 14 students interviewed, 9 students also spontaneously described a causal relationship in which molecular structure determines properties that determine function, despite the fact that this connection between the courses had not been made by the instructors. They saw that in biology there was often no explicit mechanism to link the structure to the function. One student said “I think [the courses] worked together because I took what I learned in chemistry from structure determining properties, and was really able to apply that when I was thinking of structure going from properties and then that really changing the function, in biology.” This finding has prompted us to further explore the impact of causal mechanistic reasoning across disciplines. We are also conducting more extensive studies on how causal mechanistic reasoning about more complex organic chemistry reactions develops, and how this affects student construction of electron pushing (pulling) mechanisms. We believe that there is merit in helping students construct these kinds of explanations, however, many faculty are unable or unwilling to undergo a complete transformation such as would be required to adopt CLUE, and we have previously offered suggestions for those who would like to help students develop the “habit” of providing causal mechanistic reasoning. These include the idea that Lewis acid–base theory should be introduced in general chemistry and situated in a wider range of reactions so that students understand that this model can also be applied to Brønsted acid–base reactions. Students should be routinely asked both how and why chemical phenomena occur and asked to construct models (drawings) and explanations to accompany their answers. The construction of these questions can be quite difficult, because a prompt that is too vague may not activate the appropriate resources to answer the question, and a prompt that is too scaffolded may result in an overestimate of what students actually know. 83 Even if the general chemistry course that precedes OC1 does not emphasize mechanistic reasoning and an understanding of how and when to invoke different acid–base theories, organic chemistry instructors could expand the typically rather routine overview of acid–base reactions that most OC1 courses begin with to include the ideas described above and in our earlier publication. Additionally, an emphasis on why electrons move the way they do during a reaction, followed with activities in which students draw mechanisms and explain why electrons move from source to sink, may start to develop these kinds of reasoning skills. It is true that students do improve over the course of a year of organic chemistry, but it should be noted that Cohort A begins organic chemistry with a level of mechanistic reasoning and appropriate use of mechanistic arrows that the other two cohorts only achieve after a year of organic chemistry. There are several limitations to this study. Limitations First, we do not know what knowledge Cohorts B and C had before they enrolled in organic chemistry, therefore we do not make claims about what ideas they bring with them or how much they forgot over the summer. However, they do not seem to begin organic chemistry with the same level of facility that Cohort A has. Second, the reactions studied are quite simple, and we do not know how students might fare with more complex tasks. However, it is our experience that we must begin studies by investigating simple systems to understand how students will respond in the best-case scenario. It can be very difficult to disentangle student reasoning if they do not understand the nature of the reaction. It bears noting that our goal is not to determine what students do not know, but rather to understand how students are able to construct and use explanations arguments and models. If we had begun these studies with the complex reactions typically taught in organic chemistry it is unlikely that we would have 84 been able to disentangle student reasoning about what was happening from other problematic ideas about structure, properties, and reactivity.52 That being said, without studies on more complex systems it is entirely possible that students from the transformed curriculum are simply repeating explanations and arrow pushing mechanisms that they have memorized. That is, we may have exchanged one set of memorization tasks for another. Future Work Now that we have more understanding of how students from a range of backgrounds address simple acid–base reactions, our plan is to expand the methodology to more complex reactions: for example, nucleophilic substitutions and electrophilic additions. As reactions become more complex, we will investigate whether students are able to construct the same kinds of causal mechanistic explanations. We also plan to explore the correlation (or lack thereof) between sophistication of explanation and ability to draw mechanistic arrows as the system gets more complex. There is a great deal of evidence to support the idea that many learners leave an organic Final Thoughts chemistry course without an understanding of the central concepts and skills that would make the course meaningful. The idea that organic chemistry is a course that can be mastered by memorization and pattern recognition is anathema to instructors, but yet we see many students using these strategies—and being successful. It is our hope that by changing the emphasis of organic chemistry (and all chemistry courses) to emphasize the use of knowledge, rather than the knowledge itself, that organic chemistry will become a more useful and meaningful course to students. These studies on causal mechanistic reasoning provide us with some evidence about how to support students as they think through problems. 85 APPENDIX 86 Figure 4.9. Permissions to reproduce manuscript in its entirety. 87 Characterization Scheme for the Reaction of HCl with Examples H2O No Response No answer or their explanations were unreadable or incomprehensible Non-normative Students provide non-normative or unrelated explanations. In addition, students do not recognize it is an acid-base reaction and instead attribute the mechanism to other types of reactions or other macroscopic observations General Descriptive (what) Students provide scientifically simplistic description and may discuss bond breaking or forming Bronsted Descriptive (what) Students provide Bronsted acid-base explanation including identification of acid and/or base and discussion of proton transfer Bronsted Causal (what and why) Students provide Bronsted acid-base causal reasoning that includes discussion of polarity of one or both of the reactants Lewis Mechanistic (what and how) Students provide Lewis acid-base explanation, including role of lone pair (may also encompass the Bronsted explanation) Lewis Causal Mechanistic (what, how, and why) Students provide Lewis acid-base causal reasoning that includes discussion of polarity of one or both of the reactants (may also encompass the Bronsted explanation). Viktor: “I do not really have a reasoning” Raymond: “The hydrogen on the HCl is donating its electrons to the oxygen on the water.” Catherine: “The acid is reacting with the base and the acid is a proton donor while the base is a proton acceptor” Heather: “The HCl is the acid meaning it is a proton donor and the water is the base meaning it is a proton acceptor. At the molecular level the hydrogen from the HCl is breaking off and the water is gaining it forming H3O+” Remy: “The oxygen is extremely electronegative and attracts the proton of the hydrogen. The hydrogen donates its electron to the chlorine so that its proton can go to the oxygen.” Claire: “The oxygen atom in water bonds to the hydrogen atom in hydrochloric acid as the hydrogen and chlorine atom break apart. The partial negative oxygen in water is attracted to the partial positive hydrogen in hydrochloric acid. When the oxygen and hydrogen form a bond the hydrogen and chlorine break their bond creating the products H3O+ and Cl-“ Jackie: “The O in the H2O gives its electrons to the H in the HCl bond, and simultaneously the HCl bond breaks, placing those electrons onto the Cl. This reaction happens because it is more favorable.” Doug: “The HCl acts as a proton donor and donates a proton to water which is the proton acceptor. H2O and HCl are attracted to each other because of their partial charges. When the H on HCl interacts with the lone pair on O, the HCl bond breaks and the Cl is left with the bonding electrons” Francis: “The lone pair on the water molecule attracts the Hydrogen from HCl. The H-Cl bond is broken and forms a new bond with oxygen. The reaction occurs because the partial negative charge on oxygen attracts the partial positive charge on the hydrogen. The bond between the Hydrogen and Cl is less strong than the bond that forms between hydrogen and oxygen.” Table 4.8. Published Characterization Scheme for Student Reasoning about the Reaction of HCl with H2O. 88 Student Responses by Type (N) Answer Category NA Incorrect Mechanism Correct Mechanism Ratio of correct/incorrect 2 0 0 NN 2 1 0.5 GD 5 5 1.0 BD 7 10 1.4 BC 3 8 2.7 LM 4 16 4.0 LC 8 36 4.5 Total 31 76 2.5 Table 4.9. Published Distribution of Students' Incorrect and Correct Mechanism Drawings and the Ratio of Correct to Incorrect Drawings by Each Type of Student Response for the reaction of HCl and H2O. 89 Characterization Scheme for the Reaction of NH3 with BF3 No Response Student does not provide an answer or explanations are unreadable or incomprehensible Non-normative Student attributes the mechanism to other types of reactions or other macroscopic observations Descriptive General Student provides a scientifically simplistic description of bond formation • • • Explanation discusses only bond formation between the boron and the nitrogen atom Explanation DOES NOT include the discussion of lone pairs and their activities Explanation DOES NOT include discussion of an attraction between the two species Descriptive Causal Student provides an explanation that discusses the electrostatic attraction of the species • • Evidence that the student understands that the nitrogen is attracted to the boron The student may also add in that the nitrogen is partial negative and the boron is partial positive. This is not necessary to be causal but is something that may be observed. Descriptive Mechanistic Student provides a Lewis acid-base explanation • • • • Evidence that student understands that the lone pair of electrons on nitrogen goes to the empty p- orbital on boron Evidence that the student understands that the bond is formed because of the electrons and their movement Just mentioning that the nitrogen has electrons is NOT sufficient. The response must correctly discuss what the lone pair is doing. Student DOES NOT mention an attraction between the two species Causal Mechanistic Student provides a causal and a mechanistic explanation for the reaction • • Evidence that the student understands that the lone pair of electrons on nitrogen are attracted to the empty p-orbital on the boron. The student may also add in that the nitrogen is partial negative and the boron is partial positive. This is not necessary to be causal mechanistic but is something that may be observed. Examples Daniel: “The B in the 2nd reactant is more electronegative than N in the 1st reactant, they react to give a new product.” Kate: “The acid, the NH3 is accepting electron pair from BF3 then they come together due to ionic bond.” Aaron: “Two compounds come together to make one new compound.” Rachel: “The nitrogen bonds to the boron to make the new complex.” Casey: “The boron is electron deficient and is attraction to the nitrogen.” Andrew: “The partially negative nitrogen is pulled to the boron.” Tony: “I believe that is because the boron has an available space/orbital around it that will allow the lone pair from the nitrogen to bond. Because N and B are both neutral, the bonding causes the nitrogen to have a positive charge, and the boron negative charge.” Michelle: “Boron has a vacant orbital in which the lone electrons on the N can form a bond.” Devon: “The electrons on N go to the B.” (This is less desirable answer but is still considered a mechanistic response.) Mary: “The lone pair in the NH3 is able to give its electrons to the B in BF3. It acts as a nucleophile and is partially negative while the B is partially positive.” Walt: “F is withdrawing electrons more than the B so the B is open to attack from the N’s electrons forming a bond with the BF3.” Timothy: “The lone pair of electrons on the nitrogen attacks the partial positive boron which creates a new shared bond between them.” Table 4.10. Characterization Scheme for the Reaction of NH3 with BF3. 90 Measure Cohort N Mean Median Mann-Whitney z p-value Effect Size Mann-Whitney Comparisons of Demographic Measures GC2 grade CLUE – GCi 107 Cohort A – OCii Cohort B - OCiii Cohort A - OC 92 48 92 ACT CLUE – GC 107 Cohort A – OC Cohort B - OC Cohort A - OC Cohort C - OCiv Cohort A - OC Cohort C - OC Cohort B - OC GPA prior Cohort B - OC to OC1 OC1 Grade OC2 Grade Cohort A - OC Cohort C - OC Cohort A - OC Cohort C - OC Cohort B - OC Cohort B - OC Cohort A - OC Cohort C - OC Cohort A - OC Cohort C – OC Cohort B - OC Cohort B - OC Cohort A - OC Cohort C – OC Cohort A - OC Cohort C - OC Cohort B - OC 92 48 92 54 92 54 48 48 92 54 92 54 48 48 92 54 92 54 48 48 92 54 92 54 48 2.9 3.3 3.3 3.3 25.7 26.6 28.3 26.6 26.7 26.6 26.7 28.3 3.5 3.5 2.2 3.5 2.2 3.5 3.9 3.7 3.6 3.7 3.6 3.9 3.4 3.3 3.0 3.3 3.0 3.4 3.0 3.5 3.5 3.5 25.0 26.0 28.5 26.0 26.0 26.0 26.0 3.6 3.5 3.0 3.5 3.0 3.6 4.0 4.0 4.0 4.0 4.0 4.0 4.0 3.5 3.0 3.5 3.0 4.0 U 3617.0 -3.202 0.001 2086.0 -0.556 0.579 3758.0 -1.736 0.083 (r) 0.23 1555.5 -2.548 0.011 0.22 2155.5 -0.198 0.843 2033.0 -0.769 0.442 1419.5 -4.187 < 0.001 0.35 1703.0 -2.893 0.004 0.24 2297.0 -0.889 0.374 2071.0 -0.647 0.517 2078.0 -1.734 0.083 iCLUE – GC: Students had CLUE for GC2 and were given the assessment items at the End of GC2. iiCohort A – OC: Students had CLUE for GC2 and were given the assessment items at the Start of OC1 and End of OC2. iiiCohort B – OC: Students had a selective course for GC2 and were given the assessment items at the Start of OC1 and End of OC2. ivCohort C – OC: Students transferred GC2 credit or did not take GC2 and were given the assessment items at the Start of OC1 and End of OC2. Table 4.11. Statistical analysis of academic measures for the four cohorts. 91 Chi-Square Analysis of Demographic Measures Gender Cohort N Male Female Pearson Chi- 107 33% 92 35% 67% 65% Square 0.162 CLUE - GC2 i Cohort A – ii OC Deg of Freedom p-value 1 0.687 Cohort 48 27% 73% 0.857 1 0.355 B – iii OC Cohort A – OC Cohort C – iv OC Cohort A – OC 92 54 35% 22% 65% 78% 92 35% 65% 2.550 1 0.110 Chi – Square across all 3 cohorts Major Cohort N Pre- professional CLUE - GC2 Cohort A – OC Cohort B – OC Cohort A – OC Cohort C – OC Cohort A – OC 107 92 48 92 54 92 63% 61% 69% 61% 45% 61% Animal and Plant Science 12% 7% 15% 7% 33% 7% Physical Science and Engineering Other Pearson Deg of p-value Cramer' Chi- Square Freedom s V 4% 4% 6% 4% 2% 4% 21% 2.631 28% 10% 7.280 28% 20% 18.011 28% 3 3 3 0.452 0.630 < 0.001 0.351 i CLUE – GC: Students had CLUE for GC2 and were given the assessment items at the End of GC2. ii Cohort A – OC: Students had CLUE for GC2 and were given the assessment items at the Start of OC1 and End of OC2. iii Cohort B – OC: Students had Lyman Briggs for GC2, Honor’s GC2, or Major’s GC2. Given the assessment items at the Start of OC1 and End of OC2. iv Cohort C – OC: Students transferred GC2 credit or did not take GC2 and were given the assessment items at the Start of OC1 and End of OC2. Table 4.12. Chi-Square Analyses. 92 Comparison of Cohorts A, B, and C at the Start of OC1 for HCl + H2O Start OC1: Cohort A - OC (N = 92) Start OC1: Cohort B - OC (N = 48) Start OC1: Cohort C - OC (N = 54) 60% 50% 40% 30% 20% 10% 0% s e s n o p s e R t o % n e d u S t f No Non-Normative Response General Descriptive Bronsted Descriptive Bronsted Causal Lewis Mechanistic Lewis Causal Mechanistic Figure 4.10. Comparison of the Cohorts A, B, and C for the reaction of HCl with H2O at the start of OC1. This figure shows the similar trends of Cohort B and Cohort C. Cohorts B and C were combined to simplify data visualization in the chapter. 93 Comparison of Cohorts A, B, and C at the End of OC2 for HCl + H2O End OC2: Cohort A - OC (N = 92) End OC2: Cohort B - OC (N = 48) End OC2: Cohort C - OC (N = 54) 60% 50% 40% 30% 20% 10% 0% s e s n o p s e R t o % n e d u S t f No Non-Normative Response General Descriptive Bronsted Descriptive Bronsted Causal Lewis Mechanistic Lewis Causal Mechanistic Figure 4.11. Comparison of the Cohorts A, B, and C for the reaction of HCl with H2O at the start of OC1. This figure shows the similar trends of Cohort B and Cohort C. Cohorts B and C were combined to simplify data visualization in the manuscript. Chi-Square Analysis of Student Reasoning for HCl + H2O Comparing Non-Lewis Causal codes to Lewis Causal codes Time Cohort Start OC1 Cohort A – OC End OC2 Cohort A – OC N 92 92 Pearson Chi- Deg of Freedom p- Cramer's V Square value 1 Table 4.13. Chi-square analysis of student reasoning for HCl and H2O. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for HCl + H2O Time Start OC1 End OC2 Cohort Cohort A – OC Cohort A – OC Non-Lewis Causal Lewis Causal 57% 42% 43% 58% Table 4.14. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for HCl + H2O. 94 Chi-Square Analysis of Student Reasoning for HCl + H2O Comparing Non-Lewis Causal codes to Lewis Causal codes Time Cohort Start OC1 Cohort A – OC Cohort B + C – OC End OC2 Cohort A – OC Cohort B + C – OC N 92 102 92 102 Pearson Chi-Square Deg of Freedom p-value Cramer's V 23.010 5.872 1 1 < 0.001 0.344 0.015 0.174 Table 4.15. Chi-Square Analysis of Student Reasoning for HCl + H2O Comparing Non-Lewis Causal codes to Lewis Causal codes. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for HCl + H2O Time Start OC1 End OC2 Cohort Cohort A – OC Cohort B + C – OC Cohort A – OC Cohort B + C - OC Non-Lewis Causal Lewis Causal 57% 87% 42% 60% 43% 13% 58% 40% Table 4.16. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for HCl + H2O. Student Response Percentages for HCl and H2O Time Cohort N No Non- Response Normative General Descriptive Bronsted Descriptive Bronsted Lewis Causal Mechanistic Lewis Causal Total End GC2 Start OC1 End OC2 CLUE - 107 GC Cohort A - OC Cohort B - OC Cohort C - OC Cohort B + C - OC Cohort A - OC Cohort B - OC Cohort C - OC Cohort B + C - OC 92 48 54 102 92 48 54 102 2% 0% 2% 4% 3% 0% 0% 0% 0% 3% 0% 4% 9% 7% 0% 0% 5% 3% 9% 1% 8% 6% 7% 2% 4% 2% 3% 16% 20% 29% 35% 32% 11% 15% 17% 16% 10% 17% 23% 22% 22% 10% 10% 5% 8% 19% 19% 19% 13% 16% 19% 31% 30% 30% Mechanistic 41% 100% 43% 100% 15% 100% 11% 100% 13% 100% 58% 100% 40% 100% 41% 100% 40% 100% Table 4.17. Student Response Percentages for HCl and H2O. 95 Comparison of Cohorts A, B, and C at the Start of OC1 for NH3 + BF3 Start OC1 - Cohort A - OC (N = 92) Start OC1: Cohort B - OC (N = 48) Start OC1: Cohort C - OC (N = 54) 60% 50% 40% 30% 20% 10% 0% s e s n o p s e R t n e d u S t f o % No Response Non-Normative Descriptive General Descriptive Causal Descriptive Mechanistic Causal Mechanistic Figure 4.12. Comparison of Cohorts A, B, and C at the Start of OC1 for NH3 and BF3. Comparison of Cohorts A, B, and C at the End of OC2 for NH3 + BF3 End OC2: Cohort A - OC (N = 92) End OC2: Cohort B - OC (N = 48) End OC2: Cohort C - OC (N = 54) 60% 50% 40% 30% 20% 10% 0% s e s n o p s e R t n e d u S t f o % No Response Non-Normative Descriptive General Descriptive Causal Descriptive Mechanistic Causal Mechanistic Figure 4.13. Comparison of Cohorts A, B, and C at the End of OC1 for NH3 and BF3. 96 Time Chi-Square Analysis of Student Reasoning for NH3 + BF3 Comparing Non-Causal Mechanistic to Causal Mechanistic Deg of Freedom Pearson Chi- Cohort N Start OC1 End OC2 Cohort A – OC Cohort A - OC 92 92 Square 1 p-value Cramer's V Table 4.18. Chi-Square Analysis of Student Reasoning for NH3 + BF3 Comparing Non-Causal Mechanistic to Causal Mechanistic. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for NH3 + BF3 Time Start OC1 End OC2 Cohort Cohort A – OC Cohort A – OC Non-Causal Mechanistic Causal Mechanistic 63% 54% 37% 46% Table 4.19. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for NH3 + BF3. Chi-Square Analysis of Student Reasoning for NH3 + BF3 Comparing Non-Causal Mechanistic to Causal Mechanistic Time Cohort Start OC1 Cohort A – OC N 92 Pearson Chi- Square 9.193 Cohort B + C – OC 102 End OC2 Cohort A Cohort B + C 0.588 92 102 Deg of Freedom 1 1 p-value Cramer's V 0.218 0.002 0.433 Table 4.20. Chi-Square Analysis of Student Reasoning for NH3 + BF3 Comparing Non-Causal Mechanistic to Causal Mechanistic. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for NH3 + BF3 Time Start OC1 Cohort Cohort A – OC Cohort B + C – OC End OC2 Cohort A – OC Cohort B + C - OC Non-Causal Mechanistic Causal Mechanistic 63% 82% 54% 60% 37% 18% 46% 40% Table 4.21. Percentage of Non-Lewis Causal Codes to Lewis Causal Codes for NH3 + BF3. 97 Student Response Percentages for NH3 + BF3 Time Cohort N No Non- Descriptive Descriptive Response Normative General Causal Descriptive Mechanistic Causal Total Mechanistic End GC2 Start OC1 End OC2 CLUE - 107 GC Cohort A - OC Cohort B - OC Cohort C - OC Cohort B + C - OC Cohort A - OC Cohort B - OC Cohort C - OC Cohort B + C - OC 92 48 54 102 92 48 54 102 2% 0% 2% 7% 5% 1% 0% 0% 0% 4% 3% 2% 0% 1% 1% 2% 4% 3% 6% 9% 8% 11% 10% 7% 0% 5% 3% 9% 4% 2% 6% 4% 5% 4% 7% 6% 46% 47% 63% 63% 64% 40% 56% 41% 48% 33% 100% 37% 23% 100% 100% 13% 100% 18% 100% 46% 100% 38% 43% 100% 100% 40% 100% Table 4.22. Student Response Percentages for NH3 + BF3. 98 REFERENCES 99 REFERENCES (1) Hand, B. Student Understandings of Acids and Bases: A Two Year Study. Res. Sci. Educ. 1989, 19 (1), 133–144. (2) Demerouti, M.; Kousathana, M.; Tsaparlis, G. Acid–base Equilibria, Part I. Upper Secondary Students’ Misconceptions and Difficulties. Chem. Educ. 2004, 9 (2), 122–131. (3) Sheppard, K. High School Students’ Understanding of Titrations and Related Acid–base Phenomena. Chem. Educ. Res. Pract. 2006, 7 (1), 32–45. (4) Cros, D.; Maurin, M.; Amouroux, R.; Chastrette, M.; Leber, J.; Fayol, M. Conceptions of First‐year University Students of the Constituents of Matter and the Notions of Acids and Bases. Eur. J. Sci. Educ. 1986, 8 (3), 305–313. (5) Nakhleh, M. B.; Krajcik, J. S. Influence of Levels of Information as Presented by Different Technologies on Students’ Understanding of Acid, Base, and Ph Concepts. J. Res. Sci. Teach. 1994, 31 (10), 1077–1096. (6) Bretz, S. L.; McClary, L. Students’ Understandings of Acid Strength: How Meaningful Is Reliability When Measuring Alternative Conceptions? J. Chem. Educ. 2015, 92 (2), 212–219. (7) Cartrette, D. P.; Mayo, P. M. Students’ Understanding of Acids/Bases in Organic Chemistry Contexts. Chem. Educ. Res. Pract. 2011, 12 (1), 29–39. (8) Stoyanovich, C.; Gandhi, A.; Flynn, A. B. Acid–Base Learning Outcomes for Students in an Introductory Organic Chemistry Course. J. Chem. Educ. 2015, 92 (2), 220–229. (9) Bhattacharyya, G. Practitioner Development in Organic Chemistry: How Graduate Students Conceptualize Organic Acids. Chem. Educ. Res. Pract. 2006, 7 (4), 240–247. (10) McClary, L.; Talanquer, V. College Chemistry Students’ Mental Models of Acids and Acid Strength. J. Res. Sci. Teach. 2011, 48 (4), 396–413. (11) McClary, L.; Talanquer, V. Heuristic Reasoning in Chemistry: Making Decisions about Acid Strength. Int. J. Sci. Educ. 2011, 33 (10), 1433–1454. (12) McMurry, J. Organic Chemistry, 8th ed.; Brooks/Cole: Belmont, CA, 2012. (13) Klein, D. Organic Chemistry, 3rd ed.; John Wiley & Sons, Inc.: Hoboken, NJ, 2017. (14) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid–Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703–1712. (15) Brewer, W. F.; Chinn, C. A.; Samarapungavan, A. Explanation in Scientists and Children. Minds and Machines 1998, 8 (1), 119–136. 100 (16) Abrams, E.; Southerland, S. The How’s and Why’s of Biological Change: How Learners Neglect Physical Mechanisms in Their Search for Meaning. Int. J. Sci. Educ. 2001, 23 (12), 1271–1281. (17) Metz, K. E. Development of Explanation: Incremental and Fundamental Change in Children’s Physics Knowledge. J. Res. Sci. Teach. 1991, 28 (9), 785–797. (18) Russ, R. S.; Scherr, R. E.; Hammer, D.; Mikeska, J. Recognizing Mechanistic Reasoning in Student Scientific Inquiry: A Framework for Discourse Analysis Developed from Philosophy of Science. Sci. Educ. 2008, 92 (3), 499–525. (19) Krist, C.; Schwarz, C.; Reiser, B. Identifying Essential Epistemic Heuristics for Guiding Mechanistic Reasoning in Science Learning. J. Learn. Sci. 2018; DOI: 10.1080/10508406.2018.1510404. (20) Talanquer, V. Importance of Understanding Fundamental Chemical Mechanisms. J. Chem. Educ. 2018, 95, 1905-1911. (21) Becker, N.; Noyes, K.; Cooper, M. Characterizing Students’ Mechanistic Reasoning about London Dispersion Forces. J. Chem. Educ. 2016, 93 (10), 1713–1724. (22) Bhattacharyya, G. From Source to Sink: Mechanistic Reasoning Using the Electron-Pushing Formalism. J. Chem. Educ. 2013, 90 (10), 1282–1289. (23) Bhattacharyya, G.; Bodner, G. M. “It Gets Me to the Product”: How Students Propose Organic Mechanisms. J. Chem. Educ. 2005, 82 (9), 1402. (24) Bhattacharyya, G. Trials and Tribulations: Student Approaches and Difficulties with Proposing Mechanisms Using the Electron-Pushing Formalism. Chem. Educ. Res. Pract. 2014, 15 (4), 594–609. (25) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844–849. (26) Grove, N. P.; Cooper, M. M.; Cox, E. L. Does Mechanistic Thinking Improve Student Success in Organic Chemistry? J. Chem. Educ. 2012, 89 (7), 850–853. (27) Stowe, R. L.; Cooper, M. M. Practicing What We Preach: Assessing “Critical Thinking” in Organic Chemistry. J. Chem Educ. 2017, 94 (12), 1852–1859. (28) Pashler, H.; Bain, P. M.; Bottge, B. A.; Graesser, A.; Koedinger, K.; McDaniel, M.; Metcalfe, J. Organizing Instruction and Study to Improve Student Learning: IES Practice Guide; U.S. Department of Education, National Center for Education Research, Institute of Education Sciences: Washington, DC, 2007. https://files.eric.ed.gov/fulltext/ED498555.pdf (accessed Nov 2018). (29) Cooper, M. M. Why Ask Why? J. Chem. Educ. 2015, 92 (8), 1273–1279. (30) Sevian, H.; Talanquer, V. Rethinking Chemistry: A Learning Progression on Chemical Thinking. Chem. Educ. Res. Pract. 2014, 15 (1), 10–23. 101 (31) Cooper, M. M.; Underwood, S. M.; Hilley, C. Z.; Klymkowsky, M. W. Development and Assessment of a Molecular Structure and Properties Learning Progression. J. Chem. Educ. 2012, 89 (11), 1351– 1357. (32) Cooper, M. M.; Grove, N.; Underwood, S. M.; Klymkowsky, M. W. Lost in Lewis Structures: An Investigation of Student Difficulties in Developing Representational Competence. J. Chem. Educ. 2010, 87 (8), 869–874. (33) National Research Council. Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering; National Academies Press: Washington, DC, 2012. (34) White, R. T.; Arzi, H. J. Longitudinal Studies: Designs, Validity, Practicality, and Value. Res. Sci. Educ. 2005, 35, 137–149. (35) Flynn, A. B.; Featherstone, R. B. Language of Mechanisms: Exam Analysis Reveals Students’ Strengths, Strategies, and Errors When Using the Electron-Pushing Formalism (Curved Arrows) in New Reactions. Chem. Educ. Res. Pract. 2017, 18 (1), 64–77. (36) Nielsen, S. E.; Yezierski, E. Exploring the Structure and Function of the Chemistry Self-Concept Inventory with High School Chemistry Students. J. Chem. Educ. 2015, 92 (11), 1782–1789. (37) Nielsen, S. E.; Yezierski, E. J. Beyond Academic Tracking: Using Cluster Analysis and Self-Organizing Maps to Investigate Secondary Students’ Chemistry Self-Concept. Chem. Educ. Res. Pract. 2016, 17 (4), 711–722. (38) Galloway, K. R.; Bretz, S. L. Measuring Meaningful Learning in the Undergraduate General Chemistry and Organic Chemistry Laboratories: A Longitudinal Study. J. Chem. Educ. 2015, 92 (12), 2019–2030. (39) Mitchell, Y. D.; Ippolito, J.; Lewis, S. E. Evaluating Peer-Led Team Learning across the Two Semester General Chemistry Sequence. Chem. Educ. Res. Pract. 2012, 13 (3), 378–383. (40) Szteinberg, G. A.; Weaver, G. C. Participants’ Reflections Two and Three Years after an Introductory Chemistry Course-Embedded Research Experience. Chem. Educ. Res. Pract. 2013, 14 (1), 23–35. (41) Marra, R. M.; Rodgers, K. A.; Shen, D.; Bogue, B. Women Engineering Students and Self-Efficacy: A Multi-Year, Multi-Institution Study of Women Engineering Student Self-Efficacy. J. Eng. Educ. 2009, 98 (1), 27–38. (42) Underwood, S. M.; Reyes-Gastelum, D.; Cooper, M. M. When Do Students Recognize Relationships between Molecular Structure and Properties? A Longitudinal Comparison of the Impact of Traditional and Transformed Curricula. Chem. Educ. Res. Pract. 2016, 17 (2), 365–380. (43) Williams, L. C.; Underwood, S. M.; Klymkowsky, M. W.; Cooper, M. M. Are Noncovalent Interactions an Achilles Heel in Chemistry Education? A Comparison of Instructional Approaches. J. Chem. Educ. 2015, 92 (12), 1979–1987. 102 (44) Cooper, M.; Klymkowsky, M. Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform. J. Chem. Educ. 2013, 90 (9), 1116–1122. (45) Agresti, A. Categorical Data Analysis, 2nd ed.; John Wiley & Sons, Inc: Hoboken, New Jersey, 2002. (46) Cohen, J. Statistical Power Analysis for the Behavioral Sciences, 2nd ed.; Lawrence Erlbaum Associates: Hillsdale, New Jersey, 1988. (47) Bryfczynski, S. P. BeSocratic: An Intelligent Tutoring System for the Recognition, Evaluation, and Analysis of Free-Form Student Input. Doctoral Dissertation, Clemson University, 2012. (48) Cooper, M. M.; Underwood, S. M.; Bryfczynski, S. P.; Klymkowsky, M. W. A Short History of the Use of Technology to Model and Analyze Student Data for Teaching and Research. In Tools of Chemistry Education Research, Cole, R., Bunce, D., Eds.; ACS Symposium Series; American Chemical Society: Washington, DC, 2014; pp 219–239. (49) Cooper, M. M.; Williams, L. C.; Underwood, S. M. Student Understanding of Intermolecular Forces: A Multimodal Study. J. Chem. Educ. 2015, 92 (8), 1288–1298. (50) Kohn, K. P.; Underwood, S. M.; Cooper, M. M. Connecting Structure–Property and Structure– Function Relationships across the Disciplines of Chemistry and Biology: Exploring Student Perceptions. CBE Life Sci. Educ. 2018, 17 (2). (51) National Research Council. A Framework for K–12 Science Education: Practices, Crosscutting Concepts, and Core Ideas; National Academies Press: Washington, DC, 2012. (52) Underwood, S. M.; Reyes-Gastelum, D.; Cooper, M. M. Answering the Questions of Whether and When Learning Occurs: Using Discrete-Time Survival Analysis to Investigate the Ways in Which College Chemistry Students' Ideas about Structure–Property Relationships Evolve. J. Res. Sci. Teach. 2015, 99 (6), 1055–1072. 103 CHAPTER V: ARROWS ON THE PAGE ARE NOT A GOOD GAUGE: EVIDENCE FOR THE IMPORTANCE OF CAUSAL MECHANISTIC EXPLANATIONS ABOUT NUCLEOPHILIC SUBSTITUTION IN ORGANIC CHEMISTRY Preface This chapter discusses our investigation of organic chemistry students’ reasoning about a simple nucleophilic reaction. These students were enrolled in either a transformed organic chemistry or a traditional organic course. This research has been previously published in the Journal of Chemistry Education and is reprinted with permission from Crandell, O.M.; Lockhart, M.A.; Cooper, M.M. Arrows on the Page Are Not a Good Gauge: Evidence for the Importance of Causal Mechanistic Explanations about Nucleophilic Substitution in Organic Chemistry. J. Chem. Educ. 2020, 97(2), 313-327. Copyright 2020 American Chemical Society. A copy of permissions obtained is included in the Appendix. Supplemental Information for this manuscript is included in the Appendix. Introduction Since the release of Morrison and Boyd’s Organic Chemistry in 19591, the use of curved arrows to denote electron flow (that is to show the electron pushing mechanism) has been emphasized in most organic chemistry courses. In a national survey of organic faculty, organic chemistry experts agreed that mechanistic reasoning using the electron-pushing formalism should “conform to patterns established by known mechanisms and reflect an understanding of partial or formal charges that may exist among the reactants and intermediates.”2 There are numerous studies that have identified undergraduate and graduate student difficulties using the electron-pushing formalism in this expert-like way.3-7 However, students may not be demonstrating an understanding of structure-property relationships or electrostatic attractions when they use the electron-pushing formalism but rather they are drawing arrows to “get them to the product”6 or “decorating with arrows”3 after drawing a memorized product. 104 For example, in a study on how students draw mechanisms, Grove et al. determined that only about 50% of students used mechanistic arrows to predict products, and of the students who did draw mechanistic arrows, 20% of them drew the arrows after predicting a product instead of using mechanistic arrows as a tool to guide their prediction.3 In a study on how graduate students use mechanisms, Bhattacharyya found that some students struggled to explain their mechanistic arrow use in terms of electrostatic attractions and often resorted to memorized patterns to draw the mechanism.6 Additionally, Flynn et al. have shown that some students tend to use surface features to predict patterns of reactivity rather than thinking about mechanistic processes8, and Graulich et al. found that many students can still be successful at matching reagents to a given transformation even when relying on surface features.9 In another study on students’ understanding of alkene mechanisms, Graulich et al. found that many novices group reactions by the surface features of the starting material, reagents, or functional group in the product rather than by the type of mechanism.10 These students investigating pattern recognition and problem-solving strategies, student thinking was elicited via qualitative interviews, leading to rich descriptions about the understandings for a small set of students.8-10 Just as multiple choice assessment items have been shown to overestimate what students know11, reproducing a reaction mechanism may also overestimate student understanding of what an organic reaction mechanism actually denotes.3,6 If we are to make assertions about what students know and are able to do in organic chemistry, we believe it is important to elicit more robust evidence of student thinking beyond asking students to draw arrow-pushing mechanisms or simply to draw a predicted product – in other words, we must elicit student reasoning about how and why reactions happen. Importance of scaffolding to activate resources Research on student reasoning in chemistry is typically conducted via student interviews where the researcher can engage with the student by asking follow-up questions to expand or clarify student thinking. These robust qualitative data sets offer rich insights into student thinking for a small sub-set of 105 participants but lack power to make broad generalizations about the larger population, nor do they allow for comparisons between student populations. Our goal in this study is to collect and analyze evidence of student reasoning from large numbers of students. To accomplish this requires that we develop appropriately scaffolded task prompts that signal to students the type of reasoning we are looking for. There is a fairly extensive literature base of theories and studies on how to better scaffold learning environments.12-15 However, the term “scaffolding” has come to mean many things including but not limited to student-teacher interactions12, written instructional supports14, and interactive technology environments.14,15 Pioneering work by Wood et al. defined scaffolding as “…[a] process that enables a child or novice to solve a problem, carry out a task or achieve a goal which would be beyond his unassisted efforts.”16 Wood et al. identified functions of scaffolding for the interactions between a young child and an adult tutor when the child was tasked with solving a puzzle of wooden blocks. While Wood et al.’s study may seem distant from eliciting chemical explanations from young adults, some of these functions of scaffolding apply to our context, namely 1) Reduction of degrees of freedom and 2) Marking critical features.16 Reducing the degrees of freedom simply means simplifying the task so the learner can recognize the expectations of the task. Wood et al. placed this burden on the tutor but in our case, the written instructions from the task prompt must accomplish this without any additional encouragement from the researcher. Reiser’s work with scaffolding in educational technologies and software leveraged this idea when he concluded “…if reasoning is difficult due to complexity or the open-ended nature of the task, then one way to help learners is to use the tool to reduce complexity and choice by providing additional structure to the task.”13 In our previous work, we developed a scaffolded explanation prompt designed to elicit student reasoning about a simple acid-base reaction and have shown that this approach elicits richer responses 106 than simpler, less targeted prompts.17 In this task we asked students to explain both “what is happening” and then asked them to “explain why” the reaction is happening, and found that this task structure elicited more causal mechanistic responses (defined in the next section) than asking students to explain what and why in the same response box. By separating the prompt into two sections – “what” and “why”, students are cued into the fact that “what” and “why” are different.17 In another study that takes place in the context of structure-property relationships, Underwood et al. found that asking students to construct an explanation for why one substance had a higher boiling point elicited evidence of more sophisticated understanding than asking students to construct an argument for which substance has a higher relative boiling point.18 That is, by reducing the degrees of freedom (telling students which substance has the highest boiling point), students were able to produce more robust reasoning. Marking critical features means that relevant features of the task are emphasized and made clear for the learner.16 The prompt directs students to discuss at the molecular level the role of each reactant and to explain why the reactants form the products shown. We saw in our previous work that each of these pieces appears to cue students appropriately so that they draw on their knowledge of the molecular level, the activity of each reactant (rather than just discussing one reactant), and why the reaction occurs (rather than just restating that products form). Scaffolds such as these can “[provide] structured work spaces to help learners recognize important goals to pursue.”13 It is particularly necessary to guide student thinking in organic chemistry because not only do they have more knowledge at their disposal, but also organic chemistry students have been found to hold incorrect ideas about acid strength as measured by a concept inventory19 and interviews20 and fragmented ideas about structure-property relationships of nucleophiles and electrophiles as elicited from interviews.21 Therefore, we have chosen to experiment with different prompt wording to better understand the possible ways that organic students’ ideas (or intellectual resources) may be sensitive to 107 (or activated by) different prompt wording.22 The final goal being a prompt that elicits (or activates) causal and mechanistic elements when engaging in explanation (defined in the next section) and to offer students multiple opportunities within the prompt to articulate the desired causal mechanistic explanation (Figure 5.1). Kraft et al. found student reasoning to be sensitive to the task used to elicit mechanistic problem-solving.23 Two different tasks were used to elicit students’ modes of reasoning: complete a mechanism that was already started or predict products.23 Students’ success varied on these tasks with more successful students being cued to invoke specific prior knowledge about a similar case (case-based reasoning) rather than being cued to invoke sets of memorized rules (rule-based reasoning).23 In summary, if we want students to construct explanations that contain certain components, we must clearly communicate these expectations in the prompt by providing appropriate scaffolding to elicit students’ reasoning. We have used the abovementioned literature on scaffolding to inform the design of our prompts to elicit reasoning. We have alluded to causal mechanistic explanation as our desired type of explanation. In the next section we define causal mechanistic reasoning as it is utilized in our work as well other investigations of reasoning in organic chemistry more broadly. Causal Mechanistic Reasoning in Organic Chemistry Explanation is a central practice to the understanding of science.24,25 The Framework for K-12 Science Education explicitly states “the goal of science is the construction of theories that can provide explanatory accounts of features of the world.”25 The IES report titled Organizing Instruction and Study to Improve Student Learning: IES Practice Guide cites the construction of deep explanations of phenomena as one if its chief pedagogical recommendations with strong evidence to support the importance of “[asking] questions that elicit explanations, such as those with the following question stems: why, what caused X, how did X occur…”24 There is also evidence to support the importance of asking students to themselves ask deep-level questions and construct explanations.26 Asking students to 108 construct deep explanations about why and how is impetus on which causal mechanistic reasoning is built. To define causal mechanistic reasoning, we draw from Russ et al.’s review of mechanistic reasoning where they conclude “…that mechanistic reasoning involves describing how the particular components of a system give rise to its behavior.”27 These authors modify a framework originally posed by Machamer, Darden, and Craver28 and apply it to analysis of student mechanistic reasoning in the context of classroom discussion. Russ et al.’s framework identifies several components of reasoning where the most basic components are 1) identifying entities and 2) identifying activities of those entities. These two components are foundational for more complicated reasoning to take place (e.g. reasoning about a string of events that occurred to bring a phenomenon about or making a prediction about what will happen next). For our work in the context of a chemical reaction, we have defined causal mechanistic reasoning as 1) a discussion of the electrostatic attraction between electron-rich and electron-deficient regions (the underlying causal factors) and 2) a step-by-step account of the activities of the underlying entities responsible for the mechanism: that is the movement of electrons during bond breaking and formation. We acknowledge that there are other causal factors that we have not specifically prompted for at this time. Although Russ et al. argue that mechanistic reasoning is inherently causal27 and therefore use the term mechanistic reasoning as an encompassing definition for the process of a cause bringing about an effect, we emphasize both of these elements because we have evidence that students can engage in one aspect of causal mechanistic reasoning without engaging in the other.17,29 That is, a response can be mechanistic only or causal only. For example, organic chemistry students tended to provide explanations of the reaction of NH3 with the Lewis acid, BF3, in which they discussed electron movement but did not include a discussion of electrostatic attraction for the reaction – that is, why the electrons move in this way.17 Ideally, we want students to construct explanations for chemical reactions that 109 include both an electrostatic cause and an account of electron movement. We believe that this emphasis on causality is important; for example, it may support students as they draw appropriate electron pushing mechanisms. In our previous work, general chemistry students who engaged in causal mechanistic reasoning about a simple acid-base reaction were more successful at drawing correct mechanistic arrows.17 In the context of organic chemistry, other researchers have used somewhat different approaches to defining mechanistic reasoning. Sevian and Talanquer have posed four modes of reasoning that they have used to characterize various levels of complexity in student responses.30 The lowest level modes being descriptive in instances where there is no mechanism or cause and then relational in instances where no mechanisms are discussed but “properties and behaviors are established but not explained or justified.” The next mode of reasoning being linear causal where “relevant direct interactions between entities are invoked…but phenomena [are] reduced to the result of actions of a single entity.” Finally, multicomponent reasoning being the most sophisticated where the mechanism and cause are included but also “…effects of several variables are considered and weighed.” Caspari et al. have utilized these modes of complexity to analyze student responses.31 Caspari et al. define mechanistic reasoning to mean “comparative reasoning about cause-effect relationships between explicit structural differences and structural and energic changes occurring in a mechanistic step.” Their definition of mechanistic reasoning incorporates a comparison between contrasting cases rather than reasoning about a single phenomenon in isolation.31 Flynn et al. have similarly incorporated these modes of reasoning in their analysis of causal mechanistic arguments.32 They define a descriptive mechanism as “the elementary steps that comprise an overall reaction” and the causal mechanism as “the reasons or cause behind a phenomenon or process.”32 Our goal in this work is to characterize the nature of an explanation in terms of the causal elements (electrostatic interactions) and mechanistic 110 elements (explicit electron movement) invoked to discuss a simple SN2 reaction and relate this reasoning to their mechanistic arrow use. This study is guided by these research questions: Research Questions 1. How does the nature of the prompt affect student responses about a simple nucleophilic substitution reaction? 2. How does the type of organic chemistry course affect student ability to engage in causal mechanistic reasoning? 3. How does the reasoning about a reaction change over the course of two semesters? 4. How do student written explanations of reaction type compare to their mechanistic arrow drawings? Methods Design of Assessment Tasks The design of the assessment tasks evolved over the first year of this three-year study. In the first year of the study (Year 1), we piloted two different prompt structures (called the Original SN2 Prompt and Modified SN2 Prompt). In years two and three, only the Modified SN2 Prompt was administered. Original SN2 Prompt Because a simple SN2 reaction can be considered a type of Lewis acid-base reaction, we began by adopting the same prompt structure as the one used in our acid-base work17,29 where students were asked to: i) Classify the reaction and explain their reasoning, ii) Describe what is happening on the molecular level, iii) Please explain why the reaction occurs using a molecular level explanation, and iv) Draw arrows onto pre-drawn Lewis structures to afford given products (Figure 5.1A). We also added a space for students to v) Explain why they drew their arrows as indicated to give even more opportunities to activate the relevant resources.22 The original acid-base reaction was replaced with the reaction of 111 methyl bromide (CH3Br) with hydroxide (OH-). A “textbook” example of an SN2 reaction – a methyl halide substrate with a good leaving group, and a strong unhindered nucleophile that only undergoes SN2 reactions. Lewis structures of the reactants and products were included as shown because we wanted students to explain a simple, clear-cut reaction where the products are clearly given rather than engage in argumentation about whether the reaction should be an SN2 or an SN1 by giving them a set of ambiguous reaction conditions that could be argued to be an SN2, SN1, E2, or E2. This prompt structure will be referred to as the Original SN2 Prompt and was administered at the end of Spring 2017 to students completing organic chemistry 2 (OC2). (A) Original SN2 Prompt (B) Modified SN2 Prompt i. ii. iii. iv. v. How would you classify this reaction? Please explain why you chose that classification. Can you describe in full detail what you think is happening on the molecular level for this reaction? Specifically, discuss the role of each reactant. Using a molecular level explanation, please explain why this reaction occurs? Specifically, why the reactants form the products shown. For the following reaction, please draw arrows in the BLUE box to indicate how this reaction occurs. Now please explain why you drew your arrows as indicated. i. ii. iii. iv. v. How would you classify this reaction? Please explain why you chose that classification. Please describe the sequence of events that occur at the molecular level during the reaction shown above. Please explain why these reactants interact. For the following reaction, please draw arrows in the BLUE box to indicate how this reaction occurs. Now please explain why you drew your arrows as indicated. Figure 5.1. A: Original SN2 Prompt structure administered using beSocratic. B: Modified SN2 Prompt administered using beSocratic.33 Modified SN2 Prompt We modified the Original SN2 Prompt slightly to investigate how the wording of the prompt might be activating different resources in organic chemistry students. We thought that the original wording of “Specifically, why the reactants form the products shown” might not activate resources related to 112 electrostatic interaction and rather might activate reasons for why the products are more stable or other teleological reasons for why the products “want” to form. Thus, the “describe what” phrasing was modified to “Please describe the sequence of events that occur at the molecular level during the reaction shown above” and “explain why” was changed to “Please explain why these reactants interact” (Figure 5.1B). We recognize that there are other causal factors, however we did not specifically prompt for them at this time. We made this specific choice to prompt for students’ resources concerning the core idea of electrostatic interactions.34 This prompt structure will be referred to as the Modified SN2 Prompt. Both versions (original and modified SN2 prompts) were administered at the end of Spring 2017 to students completing OC2 (Year 1 – Time Point 2) (see Figures 5.1 and 5.2). Based on our analysis (shown in Results and Discussion below), the Modified SN2 Prompt shown in Figure 5.1B seemed to elicit richer responses. Therefore we continued with the modified SN2 Prompt which was administered twice the next year: in the middle of OC1 in Fall 2017 (Year 2 – Time Point 1) just after students learned about nucleophilic substitution and then again at the end of OC2 in Spring 2018 (Year 2 – Time Point 2). It was administered again at the end of OC2 in Spring 2019 (Year 3 – Time Point 2). 113 Figure 5.2. Summary of data collections over the three years of this study. Course Contexts Student Participants Students in this study were selected based on their enrollment in two types of organic chemistry courses: Traditional OC (referred to as Traditional students) and Transformed OC (referred to as OCLUE students). Both courses were taught at a large, research-intensive Midwestern university. We have previously reported on a transformed organic chemistry course Organic Chemistry, Life, the Universe, and Everything (OCLUE).35 The course emphasizes connecting student knowledge of reactions and topics to core ideas of chemistry (structure property relationships, electrostatic forces and bonding interactions, stability and change in chemical systems, and energy) in the context of scientific practices.34,36 In contrast, the traditional organic course curriculum is organized by functional group and topic. OCLUE also requires students to use their knowledge to make predictions about phenomena and OCLUE formative and summative assessments are designed to elicit evidence of student reasoning as 114 well as the more traditional tasks such as mechanism construction or predicting products.35,37 OCLUE students are challenged to construct explanations for phenomena such as relative nucleophile strength, relative proton acidity, and kinetic and thermodynamic control in terms of atomic and molecular structure/properties and other core ideas, whereas the assessments in the traditional course ask students to provide missing reactants, products or reagents or draw a mechanism for a given reaction without any explanation.37 Each instructor wrote their own exams for their course; there were no common exams. Assessments send strong messages about what is most important to know and what students should be able to do, and we know that students tend to value (in terms of studying) what will appear on their exams.38 OCLUE students practice constructing explanations for phenomena on their weekly homework, in weekly TA led recitation group work sessions and in lectures so they are prepared to do so on their exams. Homework and recitation work are included in the OCLUE syllabus as 30% of their course grade. Traditional organic students are not given any credit for completing homework or practice problems. Rather, the traditional course grade is composed of summative assessments in the form of exams and quizzes. In an analysis of three years of OCLUE exams using the three-dimensional learning assessment protocol instrument36, between 25-50% of exam points were dedicated to questions that required students to use their knowledge of core ideas to construct an explanation or argument, reason about a model, or analyze and interpret data. Traditional organic exams did not offer any opportunities for students to engage in such scientific practices. During this study, three lecture sections of OC1 were taught each fall and three lecture sections of OC2 were taught each spring by various instructors and by varying course type (i.e. Traditional OC or OCLUE OC). Each section has a maximum enrollment of 360 students. At the time of course enrollment, students do not know the instructor or type of course because enrollment often occurs a year in advance before teaching assignments have been decided. It is also possible, because of scheduling 115 restraints, for students to take the first semester of Traditional OC1 followed by the second semester of OCLUE OC2 or vice versa. Effects of students switching from one course type to the other are not discussed in this paper but will be reported on later. All the instructors from the different course types agreed on the “topics” (e.g. reactivity of certain functional groups, skills, and content) that would be covered in the first semester so students who switched between course types would have been exposed to the same content and skills albeit with different emphases and organization.35 All student participants were informed of their rights as research participants in accordance with our institutions’ IRB. Data for this study was collected over six academic semesters starting in Fall 2016 and ending in Spring 2019. The Fall 2016 to Spring 2017 semesters will be referred to as Year 1: Pilot Phase. The Fall 2017 to Spring 2018 semesters will be referred to as Year 2: Comparing Course Types. Data was collected in Spring 2019 as a replication study of Year 2. Year 1 – Pilot Phase In Spring 2017, two sections of Traditional OC2 were taught by two organic professors both possessing 10 or more years of teaching experience. One section of OCLUE OC2 was taught in Spring 2017 by the co-author of the OCLUE curriculum (MMC). Students who took either Traditional OC2 or OCLUE OC2 (~960 students) in Spring 2017 were randomly assigned to one of the nucleophilic substitution activity prompts (see Figure 5.1) as discussed earlier. Since the purpose of piloting these versions was to identify which wording and prompt structure elicited the most causal mechanistic explanations for the reaction of CH3Br and OH-, we did not separate students based on their organic course type nor will we make any claims about course enrollment for Year 1. The “Original SN2 Prompt” was administered to 298 students across both course types with an 87% response rate. The Modified SN2 Prompt was administered to 317 students across both course types with a 91% response rate. Once the data were collected, we removed students for whom we could not 116 obtain the following information: a reported general chemistry 1 course grade, a general chemistry 2 course grade, an organic chemistry 1 course grade, an organic chemistry 2 course grade, and an ACT score or SAT score. This left 150 responses for the Original SN2 Prompt and 182 responses for the Modified SN2 Prompt (Figure 5.1). To ensure that we could reasonably compare responses across both versions, we compared the students in each group on various academic and demographic measures. A series of Mann-Whitney U- tests were performed and effect sizes reported39 comparing one cohort to the other on ACT or SAT score, GC1 course grade, GC2 course grade, GPA prior to spring 2017, OC1 grade, and OC2 grade. The only observed difference was in OC2 course grade (mean of 3.51 for OC2 course grade for the Original SN2 Prompt compared to 3.32 for the Modified SN2 Prompt U = 11970.0, z = -2.118, p = 0.034, r = 0.116 small effect size). The cohorts were also compared on gender and major and no differences were found (see full statistical output in the Appendix). Year 2 – Course Comparison Phase In Fall 2017, one section of Traditional OC1 was taught by a professor who had over 10 years of teaching experience. Two sections of OCLUE OC1 were taught in Fall 2017. The second author (MMC) taught one and oversaw a post-doctoral researcher with no teaching experience who taught the other. Both OCLUE sections used the same instructional materials and assessments. In Spring 2018, two sections of Traditional OC2 and one section of OCLUE OC2 were offered. Therefore, there were many students who changed course type as they moved from OC1 to OC2. Considering only those students who did not switch course type, we identified students who had completed both Year 2 data collections (Time Point 1 and Time Point 2), and for whom we had obtained the same academic and demographic measures used in Year 1 comparisons. Students who met these criteria were retained for our Year 2 sample. The Appendix provides a summary of this selection process 117 for Year 2 participants. There were 144 students who took a Traditional course for OC1 and OC2 in Year 2 and will be referred to as Year 2 – Traditional. Similarly, there were 108 students who took OCLUE for OC1 and OC2. These students will be referred to as Year 2 – OCLUE. Using a Mann – Whitney comparison, we found a difference in OC2 course grade between Year 2 – Traditional and Year 2 – OCLUE. The mean OC2 course grade was slightly higher for Year 2 – Traditional than for Year 2 – OCLUE (Traditional = 3.43, OCLUE = 3.23, U = 6587.5, z = -2.205, p = 0.027, r = 0.139 small effect size). No differences were found in the comparison of ACT, general chemistry course grades, OC1 course grade, gender distribution, or major (see Appendix). Year 3 – Replication Phase A replication study was performed by administering the Modified SN2 Prompt at the end of OC2. We identified those students who had the same course type for OC1 and OC2 and completed the Year 3 – Time Point 2 data collection for our Year 3 sample. The Year 3 – Traditional (N = 85) cohort averaged a slightly lower OC1 course grade (small effect size) and averaged a higher OC2 course grade (medium effect size) than the Year 3 – OCLUE cohort (N = 79). Finally, we compared these cohorts across years and found Year 2 – OCLUE to have a higher ACT (small effect size), Year 2 – Traditional to have a higher OC1 course grade (medium effect size), and Year 3 – Traditional to have a higher OC2 course grade (medium effect size). All statistical analyses were performed in SPSS and are provided in the Appendix. Data Collection The data reported in this study were collected on an online homework platform beSocratic33 and take the form of students’ typed explanations and drawn mechanistic arrows. The beSocratic system allows students to type responses to questions and draw mechanistic arrows, chemical structures, and 118 drawings using a mouse, trackpad or pen and touchscreen. This system also allows us to replay student responses so that we can determine the sequence of arrows drawn by students. Year 1 At the end of Spring 2017, all three sections of OC2 participated in the study. Students in Traditional OC2 and OCLUE OC2 were randomly assigned one of the two beSocratic prompts shown in Figure 5.1. Both versions of the activity were administered in the 14th week (that is, near the end) of OC2. This round of data collection is referred to as Year 1 – Time Point 2 (Figure 5.2). For Year 1 – OCLUE students, this beSocratic activity was included as part of their final homework assignment for the course. Students in OCLUE completed an average of two beSocratic homework assignments per week which counted for 15% of their total OCLUE course grade. In other words, the assessment was part of one out of ~25 homework assignments. These homework assignments are typically graded for participation not correctness. Year 1 – Traditional students completed this beSocratic activity for 5% of their total course grade. That is, the overall contribution to the grade was higher for the traditional students. Most students were familiar with the platform because it is used for general chemistry at this institution. Year 2 and Year 3 The Modified SN2 Prompt was administered in Years 2 and 3. In Year 2, at the 10th week of Fall 2017 to both Year 2 – Traditional students and Year 2 – OCLUE students. At this time in the semester, students in both groups had discussed nucleophilic substitution including SN2 and SN1 and eliminations. Students had not seen these prompts before nor were they provided with the “desired” response that could have been memorized. This data collection is referred to as Year 2 – Time Point 1 (Figure 5.2). Completion of this activity was part of the OCLUE student’s regular homework assignments. Traditional OC1 students were offered a small amount of extra credit for completing the activity (approximately 2% 119 of their final course grade). The same Modified SN2 Prompt was given to Traditional OC2 and OCLUE OC students at the end of OC2 in Spring 2018. These data will be referred to as the Year 2 – Time Point 2 (Figure 5.2). The Modified SN2 Prompt was once again given at the end of OC2 in Spring 2019 to both Traditional and OCLUE students. This data collection is referred to as Year 3 – Time Point 2 to attempt to replicate findings from Year 2. Data Analysis Characterization of Causal Mechanistic Reasoning The coding scheme reported below (Table 5.1) is modified from the published acid-base coding schemes previously applied to the reaction of HCl + H2O and NH3 + BF3.29 Explanations that describe bond formation are characterized as Descriptive General (DG). Phyllis’s response is a description of the reaction given to her in the prompt: “First the OH attacks the carbon center and the Br leaves (carbon- bromine bond breaks) this happens in one step.” Some organic students use more advanced vocabulary to discuss a nucleophilic substitution reaction than we found with responses to simple acid-base reactions.17,29 For example, students might use the terms nucleophile and electrophile but simply using appropriate vocabulary does not serve as evidence of understanding without further explanation. A response that identifies the nucleophile and says that it attacks the electrophile is still characterized as a description and does not demonstrate evidence of understanding causality or the mechanism, and as such was also classified as DG. For example, Calvin’s response “The nucleophile attacks the electrophile which makes the leaving group leave” was coded as DG. Responses that demonstrate evidence of understanding that the reaction occurs because of an electrostatic attraction between a negatively charged species and a positively charged species are classified as Descriptive Causal (DC). Ryan demonstrates this type of reasoning when he said “The carbon is slightly positive because the bromine is pulling the electrons away from the carbon. The negative oxygen attracts the partially positive carbon and the bromine is pushed off and a new bond is made between the carbon and the oxygen.” Descriptive Causal responses do not discuss the role of electrons 120 in bond formation and bond breaking, and so Ryan’s response does not meet our criteria for causal mechanistic reasoning. On the other hand, Descriptive Mechanistic (DM) responses do discuss bond breaking and formation in terms of electrons and their activities but do not discuss the causal factors that brought about the reaction. Wanda’s response shows how it is possible to reason about the mechanism without evidence of understanding electrostatic interactions. Wanda said “The electrons from the negatively charged OH are going to attack the carbon. This will push off the bromine and the bromine will get the electrons from the bond between the carbon-bromine bond.” A Causal Mechanistic (CM) response involves both the role of electrons and electrostatic interactions in the mechanism. Megan demonstrates causal mechanistic reasoning about an SN2 process when she says “The carbon has a partial positive on it due to the Br and so the negatively charged O attacks positive carbon with its lone pair breaking the bond of C-Br and those electrons go to the Br.” For the purposes of this coding activity, reasoning about an SN1 mechanism in a Causal Mechanistic way is also coded as CM, as demonstrated by Travis “The bromine leaves and takes the C-Br electrons with it. This leaves a carbocation which then attracts the lone pair on the oxygen to make the bond.” A comprehensive codebook is provided in Table 5.1. There were instances in which student responses were non-normative. For example, Emmy’s response “the oxygen accepted a proton and formed O-H bond and Br leaves” was explaining a process other than a nucleophilic substitution. We also observed instances where students explained an SN1 reaction instead of an SN2. We still analyzed these responses using the causal mechanistic codes in Table 5.1. We expand further on this analysis in the next section. We have used these coding schemes because they are based on the ways that most organic texts discuss such reactions. Ideally, we might want students to also incorporate the idea that reactions begin with collisions between the reacting entities, that the collisions must have enough energy to surmount the activation energy barrier, and that they must be in the correct orientation. However, the prompt 121 does not specifically probe for this kind of reasoning, and little evidence of this emerged. We also use the term “attack” because most instructors describe reactions in this way, and we did not want to privilege any particular group of students with the coding scheme. 122 Mutually Exclusive Code with Description Examples of Related Student Responses Student does not provide an answer. Explanations are unreadable or incomprehensible. Student does not even attempt to answer. Student provides a non-normative or unrelated explanation. Student attributes the mechanism to other types of reactions or other types of macroscopic observations. The response is discussing incorrect entities and/or incorrect processes. Jessica: “I don’t know what to say.” Emmy: “While oxygen accepted a proton and formed O–H bond and Br leaves.” Mason: “The Br is attracted to the H.” Student provides a scientifically simplistic description of bond formation and bond breaking. Calvin: “The nucleophile attacks the electrophile which makes the leaving group leave.” No Response (NR) Non- Normative (NN) Descriptive General (DG) Descriptive Causal (DC) Phyllis: “First the OH attacks the carbon center and the Br leaves (carbon–bromine bond breaks) this happens in one step.” Barbara: “These reactants interact because the OH group has a negative charge and is therefore nucleophilic. It wants to attack a carbon center (or something with a positive charge even if its partial) the Br is a good leaving group (better than OH) so OH is able to come in and take its place.” Ryan: “The carbon is slightly positive because the bromine is pulling the electrons away from the carbon. The negative oxygen attracts the partially positive carbon and the bromine is pushed off and a new bond is made between the carbon and the oxygen.” Wanda: “The electrons from the negatively charged OH are going to attack the carbon. This will push off the bromine and the bromine will get the electrons from the bond between the carbon–bromine bond.” Morgan: “The lone pair on OH is forming a bond with carbon.” Megan: “The carbon has a partial positive on it due to the Br and so the negatively charged O attacks positive carbon with its lone pair breaking the bond of C–Br and those electrons go to the Br.” Travis: “The bromine leaves and takes the C–Br electrons with it. This leaves a carbocation which then attracts the lone pair on the oxygen to make the bond.” Student discusses the electrostatic attraction between the species. Student gives evidence that they understand that there is an attraction between the OH– and the partial positive carbon atom. Students do not need to justify why the carbon is partial positive. They just need to demonstrate an understanding of the intermolecular electrostatic attraction. Descriptive Mechanistic (DM) Causal Mechanistic (CM) Student identifies electrons as the entities responsible for the reaction mechanism and explains their activities that lead to bond formation/bond breaking. Student gives evidence that they understand that electron movement from source to sink is how the reaction occurs. Response may only explicitly discuss the movement of the lone pair of electrons on the OH– or the electrons in the C–Br bond. This is still considered mechanistic. Student provides both the causal and the mechanistic account of the reaction. Evidence that the student understands that the lone pair of electrons on OH– is attracted to the carbon on methyl bromide and the electrons in the C–Br bond go to the Br to become Br–. Table 5.1. Causal Mechanistic Characterization Scheme. 123 While the characterizations in Table 5.1 are mutually exclusive, we did also code other response characteristics that could be applied in addition of the reasoning codes. For example, when students described an SN1 process with a carbocation, the response was tagged with an SN1 Tag. Only those students whose explanation clearly identified the formation of a carbocation or clearly implied the leaving group leaving before the oxygen approached were assigned an SN1 tag. Conversely, we also assigned an SN2 Tag for those students who correctly described a simultaneous process or one in which the approach of the nucleophile initiates the reaction. Students would occasionally justify why the carbon is partially positive by discussing the electronegativity of the bromine atom and/or discussing the polarity of the carbon-bromine bond. Justifying why the carbon is partially positive is not required to be characterized Descriptive Causal or Causal Mechanistic but we did denote responses that discussed electronegativity or polarity with a Polarity Tag. Additionally, student use of the terms “nucleophile” and/or “electrophile” were characterized by a Terminology Tag. These tags are in addition to a General Descriptive, Descriptive Causal, Descriptive Mechanistic, or Causal Mechanistic characterization. We used their frequency counts to make decisions about modifying the prompt going into Year 2. Tagsa SN1 Tag SN2 Tag Polarity Tag Tag Description Example from Student Responses The student describes an SN1 process. It explicitly describes the Br– leaving and then the OH– bonding to the carbocation. “The electrons jump from the C–Br bond to the Br and it becomes negative, since then the C will have a positive charge and then the nucleophile OH– comes into bond.” The student describes an SN2 process. It explicitly describes the OH– approaching and the Br– leaving in that order or simultaneously. The student explains why the carbon is partially positive or explains bond polarity at the electronic level. “The oxygen approaches the carbon and the bromine leaves taking the electrons in the C–Br bond with it.” “I think that the C–Br is polar, so the Br is hogging electrons by induction, so that carbon has a partial positive charge. The OH– nucleophile attacks it, and Br is a good leaving group that can support the electron pair, so it leaves and has a –1 charge.” Terminology Tag The student uses the terms “nucleophile” and/or “electrophile” in their reasoning. “The nucleophile attacked the electrophile and formed the C–OH bond.” aThese tags are applied on top of the characterizations in Table 5.1. Table 5.2. Summary of Tags Assigned to a Response When Warranted. 124 The written explanations were exported into a spreadsheet so the four pieces of a student’s explanation (i.e. Classify…, Describe what…, Explain why…, Explain your arrows…) could be analyzed together. We found it important to consider all four pieces of students’ written work together (see Results and Discussion). For example, students would often include part of their explanation in the “Classify” space. In other cases, the student’s response became more sophisticated when we consider their “Explain your arrows” piece of the response (Finding 1b below). Therefore, we decided to analyze “Classify”, “Describe what”, “Explain why” and “Explain your arrows” together and assigned a code from the Causal Mechanistic coding scheme shown in Table 5.1 and any necessary Tags shown in Table 5.2. All explanations and drawings were coded by the first author with the assistance of two undergraduate coders. The undergraduate coders worked with the authors to refine the previously published acid-base Causal Mechanistic reasoning coding scheme17,29 to encompass new complexities in vocabulary that arose with the new reaction. The first author worked with the undergraduate coders to obtain Cohen’s Kappa values of 0.72 – 0.88 using 20% of the explanation data. The mechanistic arrow drawings were coded as correct or incorrect (Figure 5.3) by the first author and the undergraduate coders. Figure 5.3. Example of a correct mechanistic arrow drawing (top) and an incorrect mechanistic arrow drawing (B). 125 Comparing Explanations to Arrow Drawings. As previously discussed, responses from Years 2 and 3 were coded both for Causal Mechanistic reasoning and whether the reaction was discussed as an SN2 or an SN1 reaction, regardless of the type of reasoning they were using. Additionally, because we can also replay student arrow drawing, we can determine the order in which the arrows were drawn, and thus determine whether the arrows portray an SN2 or an SN1 reaction. To draw a mechanism for an SN2 reaction, we would expect that the first arrow would begin at the lone pair on the oxygen atom (electron source) and end at the carbon atom in the methyl bromide species (electron sink). The second arrow would be from the carbon-bromine bond (electron source) to the bromine atom (electron sink). We recognize that, by definition, an SN2 reaction proceeds by simultaneous bond breaking and bond forming. Attempting to model this presents a limitation it is impossible to draw both arrows simultaneously. However, we have chosen to characterize arrows that start from the oxygen lone pair as the necessary first arrow as this is the first “source” of electrons in the chain of electron movement. Students who displayed use of mechanistic arrows in this way were identified as an SN2 Arrow user (see Table 5.3). However, in some instances we observed students drawing the arrow from the carbon-bromine bond to the bromine first followed by the second arrow from the oxygen lone pair to the carbon atom second. Using arrows in this order is identified as SN1 Arrow user because drawing the arrow from the carbon-bromine bond first indicates that the reaction is initiated by breaking of the carbon-bromine bond rather than the approach of the nucleophile which serves as the initial source of electrons. This gives us several possible combinations of explanations and mechanistic arrow use as shown in Table 5.3. 126 Explanation with Student Quote Mechanistic Arrow Use Code SN2 Explanation: “The oxygen attacks the carbon and then the bromine leaves.” SN2 Arrows SN2 Explanation and SN2 Arrows SN1 Explanation: “The bromine leaves and then the oxygen comes in and bonds.” SN2 Explanation: “The oxygen attacks the carbon and then the bromine leaves.” SN1 Explanation and SN1 Arrows SN2 Explanation and SN1 Arrows SN1 Arrows SN1 Arrows SN1 Explanation: “The bromine leaves and then the oxygen comes in and bonds.” SN2 Arrows SN1 Explanation and SN2 Arrows Incorrect Arrows Incorrect Arrows Table 5.3. Classifications Used to Compare Explanations to Arrow Drawings. 127 Results and Discussion RQ 1: How does the nature of the prompt affect organic student responses about a simple nucleophilic substitution reaction? Finding 1a: More causal mechanistic responses were elicited by the Modified SN2 Prompt. Figure 5.4. Comparison of causal mechanistic reasoning between the Original SN2 Prompt (A) and the Modified SN2 Prompt (B) at the end of Year 1 – Time Point 2. These characterizations for each prompt type are further separated by student use of polarity. No Response (NR), Non-Normative (NN), Descriptive General (DG), Descriptive Causal (DC), Descriptive Mechanistic (DM), Causal Mechanistic (CM). The goal in first year of this study was to investigate how to best elicit causal mechanistic responses for this nucleophilic substitution reaction using two different prompts. Comparing student causal mechanistic reasoning across both versions of the prompt (Figures 5.3A and 5.3B), we found the distributions to be similar when comparing the proportion of Causal Mechanistic responses to Non- Causal Mechanistic (2(1) = 0.626, p = 0.429). While there is no difference between the proportion of Causal Mechanistic responses to Non-Causal Mechanistic responses, there is a significant difference between the proportion of students who explicitly discussed electrostatic attraction and were coded as 128 DC or CM compared to all other characterizations. 52% of students gave a causal explanation for the Original SN2 Prompt versus 66% for the Modified SN2 Prompt (2(1) = 6.633, p = 0.010, Cramer’s V = 0.140, small effect size). We also investigated the number of students whose response justified why the carbon in methyl bromide is partially positive by discussing the polarity of the carbon-bromine bond. Even though we did not specifically ask for that level of depth in their explanations,19% of Original SN2 Prompt responses (28 of the 150 responses) and 28% (51 of the 182 responses) of Modified SN2 Prompt responses included reasoning about why the carbon is partially positive (2(1)= 3.969, p = 0.046, Cramer’s V= 0.109, small effect size. We were interested to see if discussing polarity had any relationship to student’s causal mechanistic reasoning and found that half of the causal mechanistic responses in the Modified SN2 Prompt (Figure 5.4A) discussed polarity compared to a third of the causal mechanistic responses in the Original SN2 Prompt (Figure 5.4B). It was for these reasons that we decided to move forward with the Modified SN2 Prompt in Year 2. Finding 1b: Student responses improve after drawing mechanistic arrows. Responses were assigned a code twice: the first code was based on the student responses to the questions before they drew the mechanism, and the second from any additional response students gave after they drew the mechanism. We found that student responses often became mechanistic after they drew mechanistic arrows. For example, a response that was Descriptive Causal based only on their “Classify…”, “Describe what…” and “Explain why…” could have become Casual Mechanistic after taking their “Explain why you drew your arrows as drawn” into account. In the case of the Modified SN2 Prompt, only 34% of responses explicitly discussed electron movement while this percentage jumped to 51% after drawing mechanistic arrows as shown in Table 5.4 (2(1) = 9.211, p = 0.002, Cramer’s V = 0.175, small effect size). Being asked to model electron movement (i.e. draw mechanistic arrows) and then explicitly explain their model influenced students’ explanations to be more mechanistic. This 129 observation aligns with The Framework’s goals for student engagement in modeling specifically that “science often involves the construction and use of a wide variety of models and simulations to help develop explanations about natural phenomena.”25 As such, organic chemistry students should be using their mechanistic arrows to represent their thinking of how reactions occur. We were best able to elicit this understanding by engaging students in explanation and modeling together further suggesting that student understanding in organic chemistry should be carefully elicited by activating appropriate causal and mechanistic resources.22 Response Codesa Students with Mechanistic Explanations, % Before Drawing Mechanistic Arrows After Drawing Mechanistic Arrows NN, DG, DC DM, CM 66 34 49 51 aNN: Non-Normative; DG: Descriptive General; DC: Descriptive Causal; DM: Descriptive Mechanistic; CM: Causal Mechanistic. Table 5.4. Comparative Percentage of Students Whose Response Became Mechanistic after Explaining Their Drawn Mechanistic Arrows. RQ 2: How does the type of organic chemistry course affect student ability to engage in causal mechanistic reasoning? Finding 2a: Students in both types of courses provide similar distributions of responses to the prompt immediately after learning the construct. At Year 2 – Time Point 1, 10 weeks into the semester, all instructors agreed that their students would be prepared to answer questions about SN2 reactions. After data analysis, we observed that students reasoned similarly, regardless of course type (2(1) = 0.021, p = 0.884). As shown in Figure 5.5A, over 50% of students in both courses constructed Causal Mechanistic explanations. This is evidence that students in both courses: 1) were taught about the electron movement and electrostatic attractions in their course and, 2) interpreted the prompt similarly. Combining the proportion of Descriptive Causal responses and Causal Mechanistic responses, we observed that the majority of students in both courses included a discussion of why the reaction occurred. This contrasts with Anderson and Bodner’s finding 130 who found that while organic instructors do teach why reactions occur but not all students pick up on it.40 Anderson et al. asserted that the fast pace of course hindered students from incorporating “the whys” into their understanding.40 It should be noted that the Year 2 – Time Point 1 assessment was given right after students learned about nucleophilic substitution and therefore, discussion of polarity and explicit discussion of electron movement would have been fresh in their minds and those resources readily available.22 Figure 5.5. Distribution of Causal Mechanistic reasoning characterizations for OCLUE and Traditional cohorts for Year 2 – Time Point 1 (A), Year 2 – Time Point 2 (B), and Year 3 – Time Point 2 (C). NR=No Response, NN=Non-Normative, DG=Descriptive General, DC=Descriptive Causal, DM=Descriptive Mechanistic, CM=Causal Mechanistic. Finding 2b: Student use of organic specific terminology was not an indication of the type of response provided. The causal mechanistic coding scheme is not dependent on the use of organic specific vocabulary. That is, the use of organic chemistry terminology such as nucleophile and electrophile is not necessarily accompanied by appropriate causal mechanistic reasoning. Indeed, the frequency with which students used such terminology (as noted by the Terminology Tag) for causal mechanistic and non-causal 131 mechanistic characterizations was not statistically different between the aggregated causal mechanistic responses and the aggregated non-causal mechanistic responses for each cohort at a given time point (Figure 5.6) (full statistical output reported in SI S15). For example, for the Year 2 – OCLUE cohort 72% of non-causal mechanistic responses invoked terminology such as nucleophile and electrophile but did not demonstrate understanding of electrostatics or explicit electron movement. Similarly, 57% of causal mechanistic responses invoked use of this terminology. This emphasizes the importance of analyzing student responses not only by the sophistication of vocabulary but by the sophistication of the ideas invoked in their reasoning. We recognize that careful analyses such as these may be difficult in large enrollment course settings. Lexical analysis models have been invoked as a possible solution for analyzing large volume open-ended responses.41,42 Figure 5.6. Distribution of organic terminology use for Non-Causal Mechanistic and Causal Mechanistic responses. 132 RQ 3: How does the reasoning about a reaction change over the course of two semesters? Finding 3a: OCLUE students improve over two semesters while Traditional students regress. By the end of OC2, differences in the pattern of responses between the two cohorts emerged, (2 (1) = 14.047, p < 0.001, Cramer’s V = 0.236, medium effect size) as shown in Figure 5.5B. At this time point, the percent of OCLUE students engaging in Causal Mechanistic reasoning had increased from 55% to 62%, while Traditional students had decreased from 56% to 38%, with an accompanying increase in Descriptive Causal responses. By the end of OC2, it appears that Traditional students were less likely to explicitly discuss electron movement (the how), but they were still likely to reason about the why (Figure 5.5B). It is not clear why this decrease occurs, since students were presumably more familiar with electron pushing mechanisms by this time point. There is disparity in the literature surrounding findings of this nature. There are a number of studies that show students do not connect arrow pushing with the interaction of charged species. For example, Bodner and Bhattacharyya have shown students using arrows to “get them to the product”6 and we have found that some students draw mechanistic arrows as an afterthought.3 However, Webber and Flynn found that students did use arrows as a tool in their problem solving process with many mentioning partial charges.43 It may be that the regression in Causal Mechanistic reasoning by Traditional students was the result of different expectations for student use of knowledge and reasoning in the two courses. In OCLUE, students are provided multiple opportunities to explain and reflect on how and why they are constructing mechanisms. Students are required to construct explanations on weekly homework and in weekly recitation sessions. In both of these formative assessment settings, students are given constructive feedback so they can iteratively practice and improve constructing explanations of various phenomena. About half of OCLUE course lecture time is spent reviewing homework explanations from the previous class so students can see what constitutes a thorough explanation and what is merely a description (although it should be noted that no feedback is provided for this particular homework assignment). In contrast, in the Traditional sections, the homework is not reviewed, and the 133 examinations do not require the construction of explanations, models or arguments. The drop in causal mechanistic reasoning for Traditional students suggests that the expectations in the class can affect how students respond to particular prompts. OCLUE Parameters Traditional Parameters Year Time Point Cohort N Year Time Point Cohort N 2 p- Value Cramer’s V 2 2 3 2 2 2 2 1 2 2 2 2 1 1 OCLUE OCLUE OCLUE OCLUE Traditional OCLUE Traditional 108 108 79 108 144 108 144 2 2 3 3 3 2 2 1 2 2 2 2 2 2 Traditional Traditional Traditional OCLUE Traditional OCLUE Traditional 144 144 85 79 85 108 144 0.021 0.884 — 14.047 <0.00 0.236 1 7.782 0.005 0.218 0.068 0.795 0.277 0.599 1.021 0.312 10.473 0.001 — — — — aAll analyses were performed in SPSS. Table 5.5. Chi-Square Comparisons of the Proportions of Non-Causal Mechanistic Responses versus Causal Mechanistic Responses. Finding 3b: Results are replicated from Year 2 to Year 3. Year 2 – Time Point 2 was intended as a delayed post-test measure to measure longitudinal effects of each course experience. After finding such a striking difference between the course types at the end of OC2, we wanted to verify that this phenomenon was replicable. The National Research Council’s report on Discipline-Based Education Research44 indicated that replicated studies provide a moderate level of evidence of a given phenomenon. Indeed, the pattern of data observed in Year 2 were replicated in Year 3 (Figure 5.5C) and no statistical differences were found between Year 2 and 3 (Table 5.5). RQ 4: How do student written explanations of reaction type compare to their mechanistic arrow drawings? Finding 4a: All students tend to be consistent in their explanations and their mechanistic drawings. At Year 2 – Time Point 1, the majority of OCLUE students (71%) discussed an SN2 process as defined by the SN2 Tag in Table 5.2 and also drew their arrows in the order of oxygen lone pair to carbon and 134 then carbon-bromine bond to bromine atom. At that same time point, 41% of Traditional students gave an SN2 explanation and drew arrows consistent with the explanation while 30% discussed an SN1 process and drew SN1-like arrows (i.e. an arrow from the carbon-bromine bond to bromine and then drew a second arrow from the oxygen lone pair to the carbon). The majority of all students drew arrows that were consistent with their written mechanisms and a few students discussed an SN2 mechanistic process and then drew arrows in an SN1-like order and vice versa (Table 5.6). It should be noted that we were only able to determine whether students were intending to portray an SN1 mechanism because we could watch the replay of the arrow drawing. Students who described an SN1 process by discussing a carbocation formation also typically drew their first arrow denoting the cleavage of the C-Br bond. Had this been a more traditional pencil and paper task we would not have known the order of arrows drawn and could not have made this connection. This finding supports the idea that the order in which students draw mechanistic arrows is important. It is clear that the only way to know if a student has a coherent understanding about this reaction mechanism is to watch the drawing replay and read the accompanying reasoning. Traditional organic assessments where students are asked to draw mechanistic arrows or merely predict products without demanding a mechanism rarely elicit such student reasoning37 nor can they provide evidence of the order of their arrow use. We believe that the evidence provided here indicates that the traditional organic task of drawing mechanistic arrows does not necessarily provide strong evidence that a student understands how and why reactions occur. By the end of OC2, 88% of OCLUE students and 65% of Traditional students correctly identified the reaction as an SN2 process and drew canonical arrows. The differences between Traditional and OCLUE, both at the Year 2 – Time Point 1 and Year 2 – Time Point 2, have small to medium effect sizes (see Table 5.7) as determined by Chi-Square test comparing the proportions of students who had an SN2 Explanation & Canonical Arrow versus all the other incorrect explanation/arrow categories. 135 Year Time Point Cohort (N) Students Deploying These Explanations and Mechanistic Arrows, % SN2 Explanation and SN2 Drawing SN1 Explanation and SN1 Drawing SN2 SN1 Explanation Explanation and SN1 Drawing and SN2 Drawing Incorrect Arrows 2 1 OCLUE (108) Traditional (144) 2 2 OCLUE (108) Traditional (144) 2 2 OCLUE (79) Traditional (85) 71 41 88 65 86 75 7 30 5 15 6 6 17 14 2 5 6 9 0 3 1 3 0 3 5 12 4 12 2 7 Table 5.6. Comparison between Reaction Process Explanation and Mechanistic Arrow Use as Characterized in Table 5.3. OCLUE Parametersa Traditional Parametersa 2 b p- Cramer’s Year Time Point Cohort N Year Time Point Cohort N Value V 2 2 3 2 2 1 2 2 2 2 OCLUE OCLUE OCLUE OCLUE Traditional 108 108 79 108 144 2 2 3 3 3 1 2 2 2 2 Traditional Traditional Traditional OCLUE Traditional 144 144 85 79 85 22.844 <0.001 0.301 17.804 <0.001 0.266 3.031 0.082 0.145 0.703 2.845 0.092 — — — aThe proportion of SN2 explanation and SN2 arrows was compared to the proportions of all other characterizations shown in Table 5.6. bTwo cohorts being compared in each given Chi-Square analysis. Table 5.7. Chi-Square Comparison of OCLUE and Traditional Cohorts. Finding 4b: SN2 versus SN1 is difficult for students, even if they do reason in a causal mechanistic way. Figure 5.7 shows the mechanistic reasoning codes coupled with the results from the SN2/SN1 Explanation/Arrows coding scheme. Recall that 55% of OCLUE and Traditional students gave a causal mechanistic response in Year 2 – Time Point 2. Here we see that not all causal mechanistic responses yielded a canonically correct explanation and drawing of an SN2 process. 136 At Time Point 1 (Figure 5.7A) over 40% of OCLUE students and 25% of Traditional students provided a causal mechanistic explanation about SN2 reactions with appropriately drawn arrows. Even for some students who engage in causal mechanistic reasoning, distinguishing between SN2 and SN1 proves to be difficult and this issue persists to a lesser degree at the end of OC2. By the end of OC2, we see that 57% of OCLUE students and 32% of Traditional students provided a causal mechanistic explanation about the canonically correct process (Figure 5.7B). Again, the results are replicated for Year 3 (Figure 5.7C). Figure 5.7. Explanations and arrow drawing comparison compared to Causal Mechanistic reasoning for Year 2 – Time Point 1 (A), Year 2 – Time Point 2 (B), and Year 3 – Time Point 2 (C). NR=No Response, NN=Non-Normative, DG=Descriptive General, DC=Descriptive Causal, DM=Descriptive Mechanistic, CM=Causal Mechanistic. In this three-year study, we have extended our prior studies17,29 on student explanation of acid-base reactions to simple SN2 reactions: we characterized student written responses and their corresponding Summary mechanistic arrows to draw the following conclusions. 1. The task prompts matter - careful construction of the task can elicit more complete and appropriate answers from students. Therefore, we slightly modified the prompt so that students 137 were not “sidelined” by none-productive ideas (in the context of this study). This resulted in a larger proportion of students providing Causal Mechanistic explanations. We also found that building in opportunities to reflect and revise answers improved student responses. 2. Students in organic chemistry may use technical terminology without a concomitant understanding of the meaning of the words. For example: when students use terms such as electrophile and nucleophile this does not necessarily mean that they are able to provide the causal reasoning that underlies those terms. 3. The task prompt elicited comparable types of responses from both OCLUE and traditional students immediately after they learned the reaction. This supports both the construct validity of the prompt – since students in both courses were interpreting the task similarly, and the fact that both courses taught the relevant material from which to construct a causal mechanistic explanation. 4. Students who were in a transformed course tended to provide more causal mechanistic explanations than students in the traditional course by the end of the second semester and these results were replicated in the following academic year. 5. Students who were in the transformed course were more likely to choose the correct sequence of events for this simple SN2 reaction and represent the process with canonically correct arrow use and an appropriate reasoning. Especially in the first semester, even for the simplest reactions, many students have difficulty determining from the reactants whether the mechanism would be SN1 or SN2. However, the majority of all students were able to draw mechanistic arrows that corresponded with the mechanism that they described. Implications for Instruction The structure of the curriculum and the tasks that students are asked to complete matter. Tasks that only require surface understanding or that can be answered by memorization and/or pattern 138 recognition communicate a strong message about what instructors want students to know and be able to do with their knowledge. Exams send a particularly strong message to students about what is most important.38 Ideally, we would like to design learning environments, formative assessments and approaches to feedback that elicit ideas, support reflection and help students make the connections that are the hallmark of deep and useful knowledge. Typical organic chemistry curricula and associated tasks are not designed to support this kind of reasoning and the connections required to support it37 and therefore students cannot be expected to sustain and recall it at a later date. As a broader implication, we uncovered confusion about even the simplest nucleophilic substitution reaction, where many students were not able to determine whether an SN1 or SN2 mechanism was appropriate. Perhaps this should spark conversation about the purpose of emphasizing unimolecular nuclear substitutions in undergraduate organic chemistry. Holliday et al. found that the SN1 reaction mechanism was minimally important in their analyses of the MACiE database of enzyme reaction mechanisms.45 Rather, proton transfers, bimolecular nucleophilic addition (AdN2), unimolecular heterolytic elimination (E1cb), and bimolecular nucleophilic substitutions (SN2) were the most common enzyme reaction mechanisms. At our institution, over 90% of students enrolled in organic chemistry (both Traditional and OCLUE) are pre-professional majors (pre-medical, pre-veterinary, pre-dental, etc.) and so perhaps valuable instruction time could be spent on reactions that are more important for biological processes. Implications for Research This study looks at only the simplest type of organic reaction, and it will be important to extend such studies to other reactions. For example, our future work will build on this study to investigate consistency in reasoning across different reactions overtime, specifically a simple, textbook SN1 reaction and a more complex, intramolecular SN2 reaction. As noted earlier, the present study only discusses data from students who were consistently enrolled in both semesters of OCLUE or Traditional OC. However, 139 because of scheduling issues it will be important to investigate the effects of switching between course types on student reasoning, overall performance, and grades. The coding schemes we have used in all of our work on mechanistic reasoning have centered around the role of interactions (electrostatic forces) as causal elements in reaction mechanisms. In future work we will also capture the role of energy and entropy. Finally, as noted these prompts are highly scaffolded in order to elicit what students know and can do. This brings up the question of what happens when these scaffolds are removed? Do students revert to a simpler explanation, or will “habits of mind” developed over the course of time prevail? And if so, what “dosage” is effective to provide students with the tools to construct causal mechanistic explanations without prompting. There is little research in this area14, but in our work on mechanistic reasoning and London Dispersion Forces we found that while the sophistication of response did drop when scaffolding was removed, student responses were still higher than if the scaffolding had never been used in the first place.46 Limitations Data for this study were collected from low-stakes homework assignments. We have not attempted to elicit reasoning to these specific prompts in a summative assessment environment and therefore, it is possible that these responses may not represent students’ best efforts. However, we attempted to mediate this limitation by replicating our study and found that reasoning trends were similar from Year 2 to Year 3. We have also found in other studies, that there is little difference between student responses for homework and on summative assessments.46 Second, the reaction used in this study is a simple nucleophilic substitution with the structures fully expanded and products provided. This design was intentional so as to eliminate confusion about the representations and guide students toward explaining the given phenomenon, however it may not be representative of the various representations students encounter throughout the course (e.g. line structures, wedge-dash representations, Newman 140 projections, etc). This study did not provide any evidence for how student reasoning would change with more complex reactions or more complex representations. Finally, our causal mechanistic coding scheme characterized students’ productive ideas about how and why the reaction occurred. Students responses did include incorrect ideas and incorrect use of vocabulary such as conflating the terms electronegative and formal negative charge. Rather than cataloging these occurrences, we have chosen to focus on making sense of students’ productive ideas and triangulating their responses with their mechanistic arrow use and study their knowledge over-time via delayed post-test. 141 APPENDIX 142 Figure 5.8. Permissions to reproduce manuscript in its entirety. 143 Summary of Organic Chemistry Instruction Over Three Years Data Collection Academic Semester Year 1 Fall 2016 Course OC1 Spring 2017 Year 2 Fall 2017 Spring 2018 Year 3 Fall 2018 Spring 2019 OC2 OC1 OC2 OC1 OC2 Course Type Traditional OC1 – 2 sections OCLUE OC1 – 1 section Traditional OC2 – 2 sections OCLUE OC2 – 1 section Traditional OC1 – 1 section OCLUE OC1 – 2 sections Traditional OC1 – 2 sections OCLUE OC2 – 1 section Traditional OC1 – 2 sections OCLUE OC1 – 1 section Traditional OC2 – 2 sections OCLUE OC2 – 1 section Table 5.8. Summary of course types during the three years of this study. Each section has ~300-360 students. OC1 Course Type OC2 Course Type Original SN2 Prompt Modified SN2 Prompt (N = 150) (N = 182) Traditional OC1 – Traditional OC2 – Fall 2016 OCLUE OC1 – Fall 2016 Traditional OC1 – Fall 2016 OCLUE OC1 – Fall 2016 Traditional OC1 (some other semester) Traditional OC1 (some other semester) Totals Spring 2017 OCLUE OC2 – Spring 2017 OCLUE OC2 – Spring 2017 Traditional OC2 – Spring 2017 Traditional OC2 – Spring 2017 OCLUE OC2 – Spring 2017 Table 5.9. Year 1 Participants. 71 28 18 29 4 0 71 41 30 30 6 4 150 182 144 Year 1 - Original SN2 Prompt Year 1 - Modified SN2 Prompt Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to Spring 17 OC1 Course Grade OC2 Course Grade Gender Major Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to Spring 17 OC1 Course Grade OC2 Course Grade Gender Major Year 1 Participants Descriptive Statistics N 150 150 150 150 150 150 150 Mean 26.78 3.47 3.48 3.61 3.53 3.51 Median 27 3.5 3.5 3.7 4.0 4.0 Male = 40% Female = 60% Preprofessional and Health Science = 85% N 182 182 182 182 182 182 182 Plant and Animal Physical Science and Other = Science = 6% Engineering = 2% Mean 27.01 3.40 3.39 3.55 3.43 3.32 Median 27 3.5 3.5 3.7 3.5 3.5 Male = 64 Female = 118 7% Preprofessional and Health Science = 88% Plant and Animal Physical Science and Other = Science = 4% Engineering = 3% 5% Table 5.10. Year 1 Participants Descriptive Statistics. 145 Mann – Whitney Comparison of Year 1 – Original SN2 Prompt to Year 1 – Modified SN2 Prompt Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to Spring 17 OC1 Course Grade OC2 Course Grade Year 1 Prompt Version Original Modified Original Modified Original Modified Original Modified Original Modified Original Modified N Mean Median Mann-Whitney Z p-value Effect Size U r 150 182 150 182 150 182 150 182 150 182 150 182 26.78 27.01 3.47 3.40 3.48 3.39 3.61 3.55 3.53 3.43 3.51 3.32 27 27 3.5 3.5 3.5 3.5 3.7 3.7 4.0 3.5 4.0 3.5 13211.0 -0.422 0.673 13186.0 -0.562 0.574 12473.5 -1.424 0.154 12586.0 -1.223 0.221 12571.5 -1.328 0.184 11970.0 -2.118 0.034 0.116 Table 5.11. Mann-Whitney Comparison of Year 1 – Original SN2 Prompt to Year 1 – Modified SN2 Prompt. Chi – Square Analysis of Gender for Year 1 Participants Version Original Modified N 150 182 Male 40% 35% Female Pearson Chi-Square Deg of Freedom 60% 65% 0.822 1 p-value 0.365 Table 5.12. Chi – Square Analysis of Gender for Year 1 Participants. Year 2 Participants Organic Chemistry Enrollment OC1 Course Type OC2 Course Type Cohort Name Total Responded to Responses Y2 –Time Point 1 and Y2 –Time Point 2 Traditional OC1 Traditional OC2 Year 2 – Traditional Fall 2017 OCLUE OC1 Fall 2017 Traditional OC1 Fall 2017 OCLUE OC1 Fall 2017 Spring 2018 OCLUE OC2 Spring 2018 OCLUE OC2 Spring 2018 Year 2 – OCLUE Not reported on here Traditional OC2 Not reported on here 388 Spring 2018 Totals 952 Had GC1, GC2, OC1, OC2, ACT or SAT 144 108 Selected for analysis for RQ 2 and RQ 3 144 108 17 Not reported on 218 487 here Not reported on here 252 278 211 75 210 179 33 343 765 Table 5.13. Students who had ACT (or SAT equivalent), GC1 course, GC2 course, OC1 course that was either Year 2 - OCLUE or Year 2 – Traditional, OC2 course that was either Year 2 – OCLUE or Year 2 – Traditional, took the Year 2 – Time Point 1 and the Year 2 – Time Point 2. 146 Cohort Year 2 – Traditional Year 2 – OCLUE Year 2 Participants Descriptive Statistics Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to Fall 17 OC1 Course Grade OC2 Course Grade Gender Major ACT GC1 Course Grade GC2 Course Grade GPA Prior to Fall 17 OC1 Course Grade OC2 Course Grade Gender Major N 144 144 144 144 144 144 Mean 26.88 3.43 3.33 3.63 3.67 3.43 Male = 28% Female = 72% Preprofessional and Plant and Animal Health Science = Science = 6% 81% 108 108 108 108 108 108 27.19 3.53 3.46 3.63 3.73 3.23 Male = 31% Female = 69% Median 27 3.5 3.5 3.69 4.0 4.0 Physical Science and Engineering Other = 11% = 2% 27 4.0 3.5 3.79 4.0 3.5 Preprofessional and Plant and Animal Health Science = Science = 4% Physical Science and Engineering Other = 4% 92% = 0% Table 5.14. Year 2 Participants Descriptive Statistics. 147 Mann – Whitney Comparison of Year 2 – Traditional and Year 2 – OCLUE Mean Median Mann-Whitney U 26.88 27.19 3.43 3.53 27 27 3.5 4.0 7428.5 6963.5 z -0.610 p-value 0.542 -1.512 0.131 Effect Size r Measure Cohort ACT Y2 – Traditional Y2 – OCLUE Y2 – Traditional Y2 – OCLUE Y2 – Traditional Y2 – OCLUE Y2 – Traditional Y2 – OCLUE Y2 – Traditional Y2 – OCLUE Y2 – Traditional Y2 – OCLUE GC1 Course Grade GC2 Course Grade GPA Prior to Fall 17 OC1 Course Grade OC2 Course Grade N 144 108 144 108 144 108 144 108 144 108 144 108 3.33 3.46 3.63 3.63 3.67 3.73 3.43 3.23 3.5 3.5 3.69 3.79 4.0 4.0 4.0 3.5 7146.5 -1.151 0.250 7406.0 -0.647 0.518 7228.5 -1.166 0.244 6587.5 -2.205 0.027 0.139 Table 5.15. Mann – Whitney Comparison of Year 2 – Traditional and Year 2 – OCLUE. Chi – Square Analysis of Gender for Year 2 Participants Cohort Y2 – Traditional Y2 – OCLUE N 144 108 Male 28% 31% Female Pearson Chi-Square Deg of Freedom p-value 72% 69% 0.129 1 0.719 Table 5.16. Chi – Square Analysis of Gender for Year 2 Participants. OC1 Course Type OC2 Course Cohort Name Total Responded to Type Responses Y3 –Time Point 2 Year 3 Participants Organic Chemistry Enrollment Traditional OC1 Traditional OC2 Year 3 – Traditional Fall 2018 OCLUE OC1 Fall 2018 Traditional OC1 Fall 2018 OCLUE OC1 Fall 2018 Spring 2019 OCLUE OC2 Spring 2019 OCLUE OC2 Spring 2019 Year 3 – OCLUE Not reported on here Traditional OC2 Not reported on here Spring 2019 Totals 416 140 147 173 876 382 119 126 151 778 Table 5.17. Year 3 Participants Organic Chemistry Enrollment. Had GC1, GC2, OC1, OC2, ACT or Selected for analysis for RQ 2 and RQ 3 SAT 301 79 80 92 552 85 79 Not reported on here Not reported on here 164 148 Cohort Year 3 – Traditional Year 3 – OCLUE Year 3 Participants Descriptive Statistics Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to Fall 18 OC1 Course Grade OC2 Course Grade Gender Major ACT GC1 Course Grade GC2 Course Grade GPA Prior to Fall 18 OC1 Course Grade OC2 Course Grade Gender Major N 85 85 85 85 85 85 Mean 26.69 3.39 3.35 3.63 3.29 3.71 Male = 28% Female = 72% Preprofessional and Plant and Animal Health Science = Science = 2% 91% 79 79 79 79 79 79 25.66 3.47 3.46 3.60 3.53 2.97 Male = 33% Female = 67% Median 27 3.5 3.5 3.74 3.5 4.0 Physical Science and Engineering Other = 7% = 0% 25 3.5 4.0 3.74 4.0 3.0 Preprofessional and Plant and Animal Health Science = Science = 2% Physical Science and Engineering Other = 0% 97% = 1% Table 5.18. Year 3 Participants Descriptive Statistics. 149 Mann – Whitney Comparison of Year 3 – Traditional and Year 3 – OCLUE N Mean Median Mann-Whitney U 85 2795.5 26.69 27 z -1.857 p-value 0.063 3035.5 -0.984 0.325 2996.0 -1.255 0.209 3303.5 -0.178 0.859 Effect Size r Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to Fall 2018 OC1 Course Grade OC2 Course Grade Cohort Y3 – Traditional Y3 – OCLUE Y3 – Traditional Y3 – OCLUE Y3 – Traditional Y3 – OCLUE Y3 – Traditional Y3 – OCLUE Y3 – Traditional Y3 – OCLUE Y3 – Traditional Y3 – OCLUE 79 85 79 85 79 85 79 85 79 85 25.66 3.39 3.47 3.35 3.46 3.63 3.60 3.29 3.53 3.71 79 2.97 25 3.5 3.5 3.5 4.0 3.74 3.74 3.5 4.0 4.0 3.0 2598.0 -2.662 0.008 0.208 1655.0 -6.042 < 0.001 0.472 Table 5.19. Mann – Whitney Comparison of Year 3 – Traditional and Year 3 – OCLUE. Chi – Square Analysis of Gender for Year 3 Participants Cohort Y3 – Traditional Y3 – OCLUE N 85 79 Male 28% 33% Female Pearson Chi-Square Deg of Freedom p-value 72% 67% 0.422 1 0.516 Table 5.20. Chi – Square Analysis of Gender for Year 3 Participants. 150 Measure ACT Cohort Y2 – Mann – Whitney Comparison of Year 2 and Year 3 N Mean Median Mann-Whitney U 144 5854.0 26.88 27 z -0.552 p-value 0.581 Effect Size r Traditional Y3 - Traditional Y2 – OCLUE Y3 – OCLUE Y2 – Traditional 85 108 79 144 26.69 27.19 25.66 3.43 Y3 – 85 3.39 Traditional Y2 OCLUE Y3 – OCLUE Y2 – Traditional 108 79 144 3.53 3.47 3.33 Y3 – 85 3.35 27 27 25 3.5 3.5 4.0 3.5 3.5 3.5 GC1 Course Grade GC2 Course Grade 3204.0 -2.915 0.004 0.213 5746.5 -0.657 0.511 3978.0 -0.849 0.396 6066.0 -0.116 0.907 Traditional Y2 – OCLUE Y3 – OCLUE Y2 – Traditional GPA Prior to Fall 2018 108 79 144 3.46 3.46 3.63 3.5 4.0 3.69 4064.0 -0.586 0.558 5890.5 -0.475 0.635 OC1 Course Grade OC2 Course Grade Y3 – 85 3.63 3.74 Traditional Y2 – OCLUE Y3 – OCLUE Y2 – Traditional 108 79 144 3.63 3.60 3.67 3.79 3.74 4.0 Y3 – 85 3.29 Traditional Y2 – OCLUE Y3 – OCLUE Y2 – Traditional 108 79 144 3.73 3.53 3.43 Y3 – 85 3.71 Traditional Y2 – OCLUE Y3 – OCLUE 108 79 3.23 2.97 3.5 4.0 4.0 4.0 4.0 3.5 3.0 Table 5.21. Chi – Square Analysis for Year 3 Participants. 4137.5 -0.352 0.725 4125.5 -4.544 < 0.001 0.300 3603.0 -2.161 0.031 4851.0 -3.009 0.003 0.199 3571.5 -1.950 0.051 Chi – Square Analysis of Gender for Year 2 and Year 3 Participants Cohort Y2 – Traditional Y3 – Traditional Y2 – OCLUE Y3 – OCLUE N 144 85 108 79 Male 28% 28% 31% 33% Female Pearson Chi-Square Deg of Freedom 72% 72% 69% 67% 0.001 0.117 1 1 p-value 0.969 0.732 Table 5.22. Chi – Square Analysis of Gender for Year 2 and Year 3 Participants. 151 OCLUE Year 2 – Time Point 1 (N = 108) Traditional Year 2 – Time Point 1 (N = 144) s e s n o p s e R t n e d u t S f o % 70% 60% 50% 40% 30% 20% 10% 0% NR NN DG DC DM CM NR NN DG DC DM CM Without polarity With polarity Figure 5.9. Analysis of causal mechanistic reasoning with polarity tags at Y2 Time Point – 1. OCLUE Year 2 – Time Point 2 (N = 108) Traditional Year 2 – Time Point 2 (N = 144) s e s n o p s e R t n e d u t S f o % 70% 60% 50% 40% 30% 20% 10% 0% NR NN DG DC DM CM Without polarity NN NR With polarity DG DC DM CM Figure 5.10. Analysis of causal mechanistic reasoning with polarity tags at Y2 Time Point – 2. 152 OCLUE Year 3 – Time Point 2 (N = 79) Traditional Year 3 – Time Point 2 (N = 85) 70% 60% 50% 40% 30% 20% 10% 0% NR NN DG DC DM CM NR NN DG DC DM CM Without polarity With polarity Figure 5.11. Analysis of causal mechanistic reasoning with polarity tags at Y3 Time Point – 2. Time Point Cohort Terminology Causal Mechanistic Frequency 2 p-value Year 2 – Time Point 1 OCLUE Use User Non-User Traditional User Non-User Year 2 – Time Point 2 OCLUE User Non-User Traditional User Non-User Year 3 – Time Point 2 OCLUE User Non-User Traditional User Non-User Characterization Non-CM CM Non-CM CM Non-CM CM Non-CM CM Non-CM CM Non-CM CM Non-CM CM Non-CM CM Non-CM CM Non-CM CM Non-CM CM Non-CM CM 36 14 33 25 35 29 38 42 24 17 35 32 22 67 17 38 21 12 29 17 27 27 15 16 Table 5.23. Analysis of terminology users and non-users. 2.655 0.103 0.735 0.391 0.407 0.523 0.660 0.417 0.003 0.957 0.020 0.886 153 REFERENCES 154 REFERENCES (1) Morrison, R.; Boyd, R. Organic Chemistry, 3rd ed.; Allyn and Bacon: Boston, MA, 1976. (2) Bhattacharyya, G. From Source to Sink: Mechanistic Reasoning Using the Electron-Pushing Formalism. J. Chem. Educ. 2013, 90 (10), 1282–1289. https://doi.org/10.1021/ed300765k. (3) (4) (5) (6) (7) (8) (9) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844–849. https://doi.org/10.1021/ed2003934. Bhattacharyya, G. Trials and Tribulations: Student Approaches and Difficulties with Proposing Mechanisms Using the Electron-Pushing Formalism. Chem. Educ. Res. Pract. 2014, 15, 594–609. Flynn, A. B.; Featherstone, R. B. Language of Mechanisms: Exam Analysis Reveals Students’ Strengths, Strategies, and Errors When Using the Electron-Pushing Formalism (Curved Arrows) in New Reactions. Chem. Educ. Res. Pract. 2017, 18, 64–77. Bhattacharyya, G.; Bodner, G. M. “It Gets Me to the Product”: How Students Propose Organic Mechanisms. J. Chem. Educ. 2005, 82 (9), 1402. https://doi.org/10.1021/ed082p1402. Galloway, K. R.; Stoyanovich, C.; Flynn, A. B. Students’ Interpretations of Mechanistic Language in Organic Chemistry before Learning Reactions. Chem. Educ. Res. Pract. 2017, 18 (2), 353–374. Galloway, K. R.; Leung, M. W.; Flynn, A. B. Patterns of Reactions: A Card Sort Task to Investigate Students’ Organization of Organic Chemistry Reactions. Chem. Educ. Res. Pract. 2019, 20 (1), 30– 52. https://doi.org/10.1039/C8RP00120K. Graulich, N. Intuitive Judgments Govern Students’ Answering Patterns in Multiple-Choice Exercises in Organic Chemistry. J. Chem. Educ. 2015, 92 (2), 205–211. https://doi.org/10.1021/ed500641n. (10) Graulich, N.; Bhattacharyya, G. Investigating Students’ Similarity Judgements in Organic Chemistry. Chem. Educ. Res. Pract. 2017, 18, 774–784. (11) Lee, H.-S.; Liu, O. L.; Linn, M. C. Validating Measurement of Knowledge Integration in Science Using Multiple-Choice and Explanation Items. Appl. Meas. Educ. 2011, 24, 115–136. (12) van de Pol, J.; Volman, M.; Beishuizen, J. Scaffolding in Teacher-Student Interaction: A Decade of Research. Educ Psychol Rev 2010, 22, 271–296. (13) Reiser, B. Scaffolding Complex Learning: The Mechanisms of Structuring and Problematizing Student Work. J. Learn. Sci. 2004, 13 (3), 273–304. (14) McNeill, K. L.; Lizotte, D. J.; Krajcik, J. S. Supporting Students’ of Construction of Scientific Explanations by Fading Scaffolds in Instructional Materials. J. Learn. Sci. 2006, 15 (2), 153–191. 155 (15) Guzdial, M. Software-Realized Scaffolding to Facilitate Programming for Science Learning. Interactive learning environments 1994, 4 (1), 1–44. (16) Wood, D.; Bruner, J. S.; Ross, G. The Role of Tutoring in Problem Solving. J. Child Psychol. Psychiatry 1976, 17 (2), 89–100. (17) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid–Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703–1712. https://doi.org/10.1021/acs.jchemed.6b00417. (18) Kararo, A. T.; Colvin, R. A.; Cooper, M. M.; Underwood, S. M. Predictions and Constructing Explanations: An Investigation into Introductory Chemistry Students’ Understanding of Structure- Property Relationships. Chem. Educ. Res. Pract. 2019, 20, 316–328. (19) McClary, L.; Bretz, S. L. Development and Assessment of A Diagnostic Tool to Identify Organic Chemistry Students’ Alternative Conceptions Related to Acid Strength. Int. J. Sci. Educ. 2012, 34 (15), 2317–2341. (20) Bhattacharyya, G. Practitioner Development in Organic Chemistry: How Graduate Students Conceptualize Organic Acids. Chem. Educ. Res. Pract. 2006, 7 (4), 240–247. https://doi.org/10.1039/B5RP90024G. (21) Anzovino, M. E.; Bretz, S. L. Organic Chemistry Students’ Fragmented Ideas about the Structure and Function of Nucleophiles and Electrophiles: A Concept Map Analysis. Chem. Educ. Res. Pract. 2016, 17, 1019–1029. (22) Hammer, D. Student Resources for Learning Introductory Physics. Am. J. Phys. 2000, 68 (S1), S52– S59. (23) Kraft, A.; Strickland, A. M.; Bhattacharyya, G. Reasonable Reasoning: Multi-Variate Problem- Solving in Organic Chemistry. Chem. Educ. Res. Pract. 2010, 11, 281–292. (24) Pashler, H.; Bain, P. M.; Bottge, B. A.; Graesser, A.; Koedinger, K.; McDaniel, M.; Metcalfe, J. Organizing Instruction and Study to Improve Student Learning; National Center for Education Research, Insitute of Education Sciences, U.S. Department of Education: Washington, D.C., 2007. (25) A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas; National Academies Press: Washington, D.C., 2012. (26) Rosenshine, B.; Meister, C.; Chapman, S. Teaching Students to Generate Questions: A Review of the Intervetion Studies. Review of Educational Research 1996, 66 (2), 181–221. (27) Russ, R. S.; Scherr, R. E.; Hammer, D.; Mikeska, J. Recognizing Mechanistic Reasoning in Student Scientific Inquiry: A Framework for Discourse Analysis Developed from Philosophy of Science. Sci. Educ. 2008, 92 (3), 499–525. https://doi.org/10.1002/sce.20264. 156 (28) Machamer, P.; Darden, L.; Craver, C. F. Thinking about Mechanisms. Philosophy of Science 2000, 67 (1), 1–25. (29) Crandell, O. M.; Kouyoumdjian, H.; Underwood, S. M.; Cooper, M. M. Reasoning about Reactions in Organic Chemistry: Starting It in General Chemistry. J. Chem. Educ. 2018, 96 (2), 213–226. (30) Sevian, H.; Talanquer, V. Rethinking Chemistry: A Learning Progression on Chemical Thinking. Chem. Educ. Res. Pract. 2014, 15 (1), 10–23. (31) Caspari, I.; Kranz, D.; Graulich, N. Resolving the Complexity of Organic Chemistry Students’ Reasoning through the Lens of a Mechanistic Framework. Chem. Educ. Res. Pract. 2018, 19, 1117. (32) Nicholas E Bodé; Jacky M. Deng; Alison B. Flynn. Getting Past the Rules and to the WHY: Cusal Mechanistic Arguments When Judging the Plausibility of Organic Reaction Mechanisms. J. Chem. Educ. 2019, 96 (6), 1068–1082. (33) Bryfczynski, S. BeSocratic: An Intelligent Tutoring System for the Recognition, Evaluation, and Analysis of Free-Form Student Input; Clemson University, Clemson, SC, 2012; Vol. Ph.D. Dissertation. (34) Cooper, M. M.; Posey, L. A.; Underwood, S. M. Core Ideas and Topics: Building Up or Drilling Down? J. Chem. Educ. 2017, 94 (5), 541–548. https://doi.org/10.1021/acs.jchemed.6b00900. (35) Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96, 1858–1872. (36) Laverty, J. T.; Underwood, S. M.; Matz, R. L.; Posey, L. A.; Carmel, J. H.; Caballero, M. D.; Fata- Hartley, C. L.; Ebert-May, D.; Jardeleza, S. E.; Cooper, M. M. Characterizing College Science Assessments: The Three-Dimensional Learning Assessment Protocol. PLOS ONE 2016, 11 (9), e0162333. https://doi.org/10.1371/journal.pone.0162333. (37) Stowe, R. L.; Cooper, M. M. Practicing What We Preach: Assessing “Critical Thinking” in Organic Chemistry. J. Chem Educ. 2017, 94 (12), 1852–1859. https://doi.org/10.1021/acs.jchemed.7b00335. (38) Momsen, J.; Offerdahl, E.; Kryjevskaia, M.; Montplaisir, L.; Anderson, E.; Grosz, N. Using Assessments to Investigate and Compare the Nature of Learning in Undergraduate Science Courses. CBE Life Sci Educ. 2013, 12 (2), 239–249. (39) Cohen, J. A Power Primer. Psychological Bulletin 1992, 112 (1), 155–159. (40) Anderson, T. L.; Bodner, G. M. What Can We Do about “Parker”? A Case Study of a Good Student Who Didn’t “get” Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9, 93–101. (41) Dood, A. J.; Fields, K. B.; Raker, J. R. Using Lexical Analysis To Predict Lewis Acid-Base Model Use in Responses to an Acid-Base Proton-Transfer Reaction. J. Chem. Educ. 2018, 95 (8), 1267–1275. 157 (42) Dood, A. J.; Fields, K. B.; Cruz-Ramirez de Arellano, D.; Raker, J. R. Development and Evaluation of a Lewis Acid-Base Tutorial for Use in Postsecondary Organic Chemistry Courses. Can. J. Chem. 2019, 97 (10), 711–721. (43) Webber, D. M.; Flynn, A. B. How Are Students Solving Familar and Unfamiliar Organic Chemistry Mechanism Questions in a New Curriculum? J. Chem. Educ. 2018, 95 (9), 1451–1467. (44) National Research Council. Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering; Singer, S. R., Nielsen, N. R., Schweingruber, H. A., Eds.; National Academies Press: Washington, D.C., 2012. (45) Holliday, G. L.; Almonacid, D. E.; Mitchell, J. B. O.; Thornton, J. M. The Chemistry of Protein Catalysis. J. Mol. Biol. 2007, 372, 1261–1277. (46) Noyes, K.; Cooper, M. M. Investigating Student Understanding of London Dispersion Forces: A Longitudinal Study. J. Chem. Educ. 2019, 96 (9), 1821–1832. 158 CHAPTER VI: “WHAT ABOUT THE STUDENTS WHO SWITCHED COURSE TYPE?”: AN INVESTIGATION OF INCONSISTANT ORGANIC COURSE EXPERIENCE Introduction It has been 20 years since the National Academies released their 1999 report Transforming Undergraduate Education in Science, Mathematics, Engineering, and Technology. This report laid out recommendations to undergraduate institutions, academic departments, and faculty to fundamentally transform SME&T for all students. The overarching goal being: “Institutions of higher education should provide diverse opportunities for all undergraduates to study science, mathematics, engineering, and technology as practiced by scientists and engineers, and as early in their academic careers as possible.” The Chemistry Education Research community has responded to the call for transformation with examples of transformed general chemistry curricula1,2, transformed organic chemistry curricula3,4, four- year transformation5, and studies of evidence-based instructional practices.6 Even still, transformation is slow. Despite a few transformation efforts, organic chemistry, for example, remains relatively unchanged7 with today’s commercial textbooks resembling Morrison and Boyd’s textbook structure from 1959 and exam questions that tend to focus on memorization and pattern recognition.8 There is no evidence that this approach to instruction supports student learning nor evidence that the traditional organic content is relevant for the majority of students enrolled in organic chemistry. Numerous studies have characterized student difficulties in organic chemistry, many of which are reviewed in Chapter III of this dissertation. Student difficulties with visualization9, acid strength10-14, and spectroscopy15-17 have also been reported. The Chemistry Education Research community has responded with a number of efforts to improve learning in undergraduate organic chemistry. Flynn and Ogilvie4 have proposed a transformed organic curriculum in which topics are organized by mechanism rather than functional group. Others have designed sequences in which students are introduced to organic chemistry topics in their first year of college.18,19 Others have reported using a flipped-classroom 159 curriculum where students watch lecture videos out of class to allow for problem-solving during lecture.20-23 To our knowledge, there are no other studies discussing student experiences moving in and out of transformed organic chemistry courses. However, McPadden and Brewe have reported on student performance in a transformed introductory physics course called University Modeling Instruction.24 This two-semester curriculum invokes multiple representations to engage students in modeling of physical phenomena. Ideally, student complete the first-semester course focusing on mechanics and then advance into the second semester focusing on electricity and magnetism (called returning students), but some students do not take the first course before entering the second course (referred to as new students). To study possible differences between these students with different course experiences, these researchers developed a survey to probe student use of various representations for different problems administered at the start and end of the electricity and magnetism semester. They found that new students used fewer representations than the returning students when approaching mechanics questions consistently at the start and end of the semester, meaning the new students never caught up on this metric as the new students did not take the transformed mechanics course. New students also trailed returning students on representation use for electricity and magnetism problems. In fact, new students ended the semester using the same number of representations that returning students used at the start suggesting that taking only the second semester of Modeling Instruction does not have the same benefits for students’ representation use as two semesters.24 What is the impact of changing between a transformed organic chemistry course and a Research Question traditional organic chemistry course on students’ causal mechanistic reasoning? 160 Methods Design of Assessment Task The assessment task is the same as that discussed in prior chapters. It is designed to elicit causal mechanistic explanations and students’ mechanistic arrow drawings and is shown in Figure 5.1B. The prompt asks students to: i) Classify the reaction and explain their reasoning, ii) Describe what is happening on the molecular level, iii) Please explain why the reaction occurs using a molecular-level explanation, and iv) Draw arrows onto pre-drawn Lewis structures to afford given products, and v) Explain why they drew their arrows as indicated after being asked to draw their arrows. Student participants included in this study were selected from those who were enrolled in Student participants organic chemistry in the 2017-2018 and 2018-2019 academic years. As previously discussed, students were enrolled in either a transformed organic chemistry course (OCLUE) or a traditional organic course. Due to scheduling constraints and the fact that students enroll months before instructor assignments are listed, it is possible that students do not take the same organic course type for their first and second semester (i.e. they may take OCLUE for OC1 and then take Traditional OC2 or vice versa). This led to four different course experience “pathways”. These four cohorts will be referred to using a two-part name indicating the course type taken for OC1 listed first and then OC2 listed second. For example, students who took OCLUE for OC1 and OC2 will be referred to as the OCLUE-OCLUE cohort. Once we identified students by their two-semester course sequence, OCLUE-OCLUE, Traditional-Traditional, OCLUE- Traditional, or Traditional-OCLUE, we removed any participants for whom we did not have the following information: general chemistry 1 and 2 course grades, organic chemistry 1 and 2 course grades, and an ACT or SAT score. Finally, we refined our sample by identifying students who completed both data collection activities – one at the start of OC2 and one at the end of OC2. 161 In fall 2017, two sections of OCLUE OC1 were offered (~720 students) and one section of 2017-2018 Traditional OC1 (~360 students) was offered. One section of OCLUE (~360 students) was taught by the second author (MMC) and the other by a post-doctoral researcher with no prior teaching experience. The same notes, homework, recitation, and exams were used in both OCLUE sections with the primary author overseeing the novice instructor. The instructor of the traditional course had over 10 years of teaching experience. In spring 2018, only one section of OCLUE OC2 was taught (~360 students) by the second author (MMC) and two sections of Traditional OC2 were taught (~720 students). From these classes the following cohorts emerged: OCLUE-OCLUE-S18 (N = 103), Traditional-Traditional-S18 (N = 128), OCLUE-Traditional-S18 (N = 195), Traditional-OCLUE-S18 (N = 17). The Traditional-OCLUE-S18 cohort is understandably small since there was only one section of Traditional OC1 taught in the fall and then one section of OCLUE OC2 taught in the spring and this sample was further refined by the above- mentioned criteria. For these reasons, we will not report any findings for the Traditional-OCLUE-S18 cohort. Many but not all of these students had also participated in a prior data collection that took place in the middle of OC1 in fall 2017. These cohorts are summarized in Table 6.1. 2018-2019 In fall 2018, only one section of OCLUE OC1 was taught with two sections of Traditional OC2. The same was true in spring 2019, meaning only one section of OCLUE OC2 and two sections of Traditional OC2 were offered. Just as before, we refined the cohorts using the criteria listed above. However, data were only collected once in S19 at the end of OC2 which was intended to be a replication of the prior year. As there were two sections of Traditional OC1 and then two sections of Traditional O2 offered, and many of these students had general chemistry, organic chemistry, and standardized test scores in addition to completing the activity (~270 students), a random selection of approximately one- third of the participants were retained for the Traditional-Traditional-S19 cohort (N = 85) to create a 162 Traditional-Traditional cohort similar in size to the other groups. The other three cohorts are OCLUE- OCLUE-S19 (N = 64), OCLUE-Traditional-S19 (N = 93), and Traditional-OCLUE-S19 (N = 67). Mann- Whitney U tests were used to compare each of these cohorts on course grades, GPA, and ACT. These analyses were run in SPSS, and results are provided in the Appendix. We have highlighted the significant differences in Table 6.2. The majority of differences were between OC1 and OC2 course grades for the S19 cohorts. Mid OC1 – F17 Start OC2 and End OC2 – S18 End OC2 – S19 OCLUE – OCLUE (N = 108) (Previously published cohort) OCLUE – OCLUE – S18 (N = 103) OCLUE – OCLUE – S19 (N = 64) Traditional – Traditional (N = 144) Traditional – Traditional – S18 Traditional – Traditional – S19 (Previously published cohort) (N = 128) (N = 85) OCLUE – Traditional (N = 190) OCLUE – Traditional – S18 OCLUE – Traditional – S19 (N = 195) (N = 93) Table 6.1. Summary of cohorts. Traditional – OCLUE – S19 (N = 67) 163 Measure Mann-Whitney U Z p-value Effect Size r OCLUE-OCLUE-S19 (N = 64) compared to Trad-Trad-S19 (N = 85) OCLUE-OCLUE-S19 (N = 64) OC1 Course Grade OC2 Course Grade 1984.5 1448.0 -3.012 0.003 -5.360 < 0.001 0.247 0.439 compared to OC2 Course Grade 2089.0 -3.341 0.001 0.267 OCLUE-Trad-S19 (N = 93) OCLUE-OCLUE-S19 (N = 64) compared to OC1 Course Grade 1486.0 -3.226 0.001 0.282 Trad-OCLUE-S19 (N = 67) Trad-Trad-S19 (N = 85) compared to OC2 Course Grade 1593.0 -5.156 < 0.001 0.418 Trad-OCLUE-S19 (N = 67) OCLUE-Trad-S19 (N = 93) compared to OC2 Course Grade 2233.5 -3.230 0.001 0.255 Trad-OCLUE-S19 (N = 67) OCLUE-OCLUE-S18 (N = 103) compared to OCLUE-OCLUE-S19 (N = 64) Trad-Trad-S18 (N = 128) compared to Trad-Trad-S19 (N = 85) OCLUE-Trad-S18 (N = 195) ACT OC2 Course Grade OC1 Course Grade OC2 Course Grade 2360.0 2349.0 3643.5 3568.0 -3.092 -3.247 0.002 0.001 -4.477 < 0.001 -4.624 < 0.001 0.239 0.251 0.307 0.317 compared to OC1 Course Grade 6690.0 -4.033 < 0.001 0.238 OCLUE-Trad-S19 (N = 93) Table 6.2. Summary of comparisons of academic measures. For brevity, only differences that were found to be significantly different are reported here. The full statistical outputs are reported in the Appendix. Data Collection As previously described, data reported in this study were collected using the online homework system called beSocratic.25 In the 2017-2018 academic year, data were collected at the start of OC2 in both OCLUE and Traditional courses. For the OCLUE students, this activity was part of their first homework assignment in the first week of the course. For Traditional students, the assignment was given at the end of the second week of the course. We will refer to this time point as Start OC2 S18. Data were collected once again at in the last week of the semester from both courses referred to as End OC2 SS18. In the 2018-2019 academic year, data were only collected at the end of OC2 referred to as End OC2 S19. These activities were administered as part of regular homework assignments for OCLUE students. OCLUE students complete approximately 20-22 required homework assignments throughout that semester. OCLUE homework assignments are not graded for accuracy and count for 15% of their 164 course grade with data collection counting for >1% of their total course grade. For Traditional students, the assignment was offered as extra credit (approximately 2% of their course grade). Student explanations were coded using the same scheme from the prior study (Table 5.1). Data Analysis Students’ mechanistic drawings were similarly coded as in prior studies (Table 5.3). Our prior work explored student reasoning right after they learned about nucleophilic Results substitution reactions in the middle of OC1 and then again at the end of OC2. For this analysis, we also explored student reasoning at the start of OC2 to better understand how student reasoning changes overtime. Finding 1a: OCLUE-OCLUE students’ reasoning improved overtime. In the prior chapter, evidence showed that OCLUE-OCLUE and Traditional-Traditional students reasoned similarly about this reaction directly following their instruction about nucleophilic substitution in the middle of OC1 as shown in Figure 6.1A (55% of both groups giving a causal mechanistic response). This prior work established validity for this prompt by showing: 1. Students in both courses interpreted the prompt similarly and 2. Students in both courses were equally capable of engaging in causal mechanistic reasoning right after instruction. By the start of OC2, OCLUE-OCLUE students have maintained this level of reasoning with nearly 60% of OCLUE-OCLUE-S18 students still engaged in causal mechanistic reasoning. By the end of OC2, 65% of OCLUE-OCLUE-S18 students engaged in causal mechanistic reasoning (Figure 6.1). Finding 1b: Traditional-Traditional students regress in their engagement in causal mechanistic reasoning by the start of OC2. Right after learning the material in OC1, 55% of Traditional-Traditional-S18 students constructed causal mechanistic responses. By the start of OC2, this percentage dropped to 41%. A Chi-Square test of 165 homogeneity was used to compare this proportion to that of OCLUE-OCLUE students at the start of OC2. The difference in these proportions was found to be significant (2(1) = 7.248, p = 0.007, Cramer’s V = 0.177, small effect size). This decrease in reasoning at the start of OC2 for Traditional-Traditional students is noteworthy since there is a relatively short time between the two data collections: the data in Figure 6.1A were collected in the middle of OC1 and data in Figure 6.1B were collected at the start of the next semester. At the end of OC2, there was still a significant difference between the OCLUE-OCLUE- S18 (65% CM) and Traditional-Traditional-S18 (40% CM) (2(1) = 13.630, p < 0.001, Cramer’s V = 0.243, medium effect size). Finding 2a: OCLUE-Traditional students reasoned similarly to the other students right after they learned the material much like OCLUE-OCLUE and Traditional-Traditional students. Our analysis of OCLUE-Traditional students’ reasoning began by verifying that these students reasoned similarly to the other cohorts right after learning the material. We would expect that they would reason similarly to OCLUE-OCLUE students as OCLUE-Traditional and OCLUE-OCLUE students were co-enrolled in the same OC1 course. We indeed found that OCLUE-Traditional students reason similarly to the other two cohorts right after they learn the material as shown in Figure 6.1. Specifically, 55% of OCLUE-OCLUE, 56% of Traditional-Traditional, and 52% of OCLUE-Traditional students constructed causal mechanistic responses right after learning the material. Finding 2b: OCLUE-Traditional student regressed when they changed course types in OC2. Once OCLUE-Traditional-S18 students transitioned in the Traditional course in OC2, we found that OCLUE-Traditional-S18 students’ causal mechanistic reasoning fell to 45% at the start of OC2. Figure 6.1B shows OCLUE-Traditional students to reason somewhat in the middle of OCLUE-OCLUE and Traditional-Traditional students. OCLUE-Traditional reasoning remains somewhat in the middle at the end of OC2. 166 To understand these findings, it is useful to consider the nature of core idea centered knowledge and the use of scientific practices to reason about cause and effect. Engaging in causal mechanistic reasoning about this reaction requires students to identify the sub-atomic entities and consider their properties along with other chemical principles such as electrostatic interactions. The key to constructing a causal mechanistic explanation is connecting these core ideas together (i.e. structure/property relationships and forces/interactions). This evidence suggests that OCLUE students are more likely to connect these core ideas together weeks after they have learned about how to invoke these core ideas in this context. However, if this were the only factor, we would expect similar trends from all students who had OCLUE for OC1. Students in the OCLUE-Traditional cohort were co-enrolled in the same OCLUE OC1 course as the OCLUE-OCLUE students and therefore, we would expect similar reasoning trends for these two groups. Indeed, OCLUE-OCLUE, Traditional-Traditional, and OCLUE-Traditional students all reasoned similarly right after they learn the material in the middle of OC1 (Figure 6.1A). However, by the start of OC2, we observe that OCLUE-Traditional students’ reasoning is somewhere in the middle relative to OCLUE-OCLUE and Traditional-Traditional students. To understand this regression in OCLUE-Traditional and Traditional-Traditional students’ mechanistic reasoning, we must also consider when these data were collected. As discussed above, OCLUE students were sampled in the first week of OC2. However, because of logistical barriers, data were not collected from Traditional students until the end of the second week of the course. Because of this, it seems likely that Traditional-Traditional and OCLUE-Traditional students had already been immersed engaged in the norms and expectations of the course for nearly two weeks by the time they were sampled for this study. This finding suggests that students’ tendency to engage in causal mechanistic reasoning is influenced by the different ways in which students are expected to use their 167 knowledge in a traditional organic course in comparison to OCLUE, even from the very beginning. We will expand on the implications of this finding in the Discussion section below. Figure 6.1. Characterization of Causal Mechanistic Reasoning for the reaction of CH3Br with OH-. For simplicity of representation, the No Response and Non-Normative bins are removed from this representation. The proportions for No Response and Non-Normative can be found in the Supplemental Information. DG-Descriptive General, DC-Descriptive Causal, DM-Descriptive Mechanistic, CM-Causal Mechanistic. Finding 3: Causal mechanistic reasoning trends observed in S18 are replicated in S19. Just as we replicated our findings for the OCLUE-OCLUE and Traditional-Traditional cohorts (discussed in Chapter 5), here we have also replicated the trends observed for Traditional-OCLUE students in both S18 and S19 (2(1) = 3.677, p = 0.055) (Figure 6.1D). All Chi-Square analyses were 168 performed to test the difference in the proportion of causal mechanistic responses to non-causal mechanistic responses. We have used a Bonferroni adjusted α = 0.017 to limit the likelihood of Type 1 error with making multiple comparisons between these three cohorts across multiple years.26 These replication studies lend significant merit to these findings about the impact of course experience on student casual mechanistic reasoning. Finding 4: Traditional-OCLUE students also exhibit reasoning that is in the “middle.” We now shift focus to the Traditional-OCLUE-S19 cohort. As discussed above, we were only able to capture students with this course experience in the 2018-2019 academic year because of limitations in student enrollment in the first year of this study. However, we observed that these students also show reasoning patterns that appear to be intermediary between those of the OCLUE-OCLUE and Traditional-Traditional students. That is, they show similar reasoning patterns to those students who were enrolled in OCLUE first and then Traditional OC second (Table 6.3). Using Chi-Square test, we tested the differences in proportion of causal mechanistic responses to non-causal mechanistic responses. A Bonferroni adjusted α = 0.008 was used to limit the likelihood of Type 1 error when making multiple comparisons between these four cohorts in S19.26 We observed no significant differences in the percentage of causal mechanistic reasoning between the Traditional-OCLUE-S19 and any of the other three cohorts (Table 6.4). 169 OCLUE-OCLUE (N = 108) (Previously published cohort) Traditional-Traditional (N = 144) (Previously published cohort) OCLUE-Trad-S18 (N = 190) Mid OC1 - F17 NR NN 1% 3% DG 5% DC DM CM 18% 18% 55% 0% 1% 10% 16% 17% 56% 1% 0% 14% 26% 7% 52% Start OC2 - S18 NR NN 0% 0% 0% 0% 5% 2% DG 7% DC DM CM 25% 9% 59% 14% 27% 13% 41% 15% 30% 8% 45% NR NN DG DC DM CM 0% 0% 0% 2% 7% 18% 8% 65% 3% 11% 34% 12% 40% 1% 10% 31% 7% 51% OCLUE-OCLUE-S18 (N = 103) Traditional-Traditional-S18 (N = 128) OCLUE-Traditional-S18 (N = 195) End OC2 - S18 OCLUE-OCLUE-S18 (N = 103) Traditional-Traditional-S18 (N = 128) OCLUE-Traditional-S18 (N = 195) End OC2 - S19 NR NN DG DC DM CM OCLUE-OCLUE-S19 (N = 64) 0% 0% 6% 19% 16% 59% Traditional-Traditional-S19 (N = 85) 0% 11% 7% 29% 16% 37% OCLUE-Traditional-S19 (N = 93) Traditional-OCLUE-S19 (N = 67) 1% 0% 3% 8% 37% 12% 39% 0% 10% 29% 13% 48% Table 6.3. Distribution of reasoning characterizations for all cohorts at each relevant time point. NR-No Response, NN-Non-normative, DG-Descriptive General, DC-Descriptive Causal, DM-Descriptive Mechanistic, CM-Causal Mechanistic. 170 Time Point S18 Start OC2 – S18 Start OC2 – S18 Start OC2 – S18 End OC2 – S18 End OC2 – S18 End OC2 – S18 S19 End OC2 – S19 End OC2 – S19 End OC2 – S19 End OC2 – S19 End OC2 – S19 End OC2 – S19 Comparing S18 to S19 End OC2 End OC2 End OC2 Cohorts 2 p-value Cramer’s V OCLUE-OCLUE-S18 Traditional-Traditional-S18 OCLUE-OCLUE-S18 OCLUE-Traditional-S18 Traditional-Traditional-S18 OCLUE-Traditional-S18 OCLUE-OCLUE-S18 Traditional-Traditional-S18 OCLUE-OCLUE-S18 OCLUE-Traditional-S18 Traditional-Traditional-S18 OCLUE-Traditional-S18 OCLUE-OCLUE-S19 Traditional-Traditional-S19 OCLUE-OCLUE-S19 OCLUE-Traditional-S19 Traditional-Traditional-S19 OCLUE-Traditional-S19 OCLUE-OCLUE-S19 Traditional-OCLUE-S19 Traditional-Traditional-S19 Traditional-OCLUE-S19 OCLUE-Traditional-S19 Traditional-OCLUE-S19 OCLUE-OCLUE-S18 OCLUE-OCLUE-S19 Traditional-Traditional-S18 Traditional-Traditional-S19 OCLUE-Traditional-S18 OCLUE-Traditional-S19 7.248 0.007a 0.177 5.356 0.021 0.435 0.509 13.630 < 0.001a 0.243 5.570 0.018 3.194 0.074 7.703 0.006b 0.227 6.497 0.011 0.095 0.758 1.774 0.183 1.968 0.161 1.306 0.253 0.544 0.461 0.371 0.543 3.677 0.055 Table 6.4. Chi-Square tests of Homogeneity of the proportions between cohorts. A Bonferroni correction was used to determine significance to help reduce the chance of Type 1 error. a An α = 0.017 was used for the series of 3 pair-wise comparisons. b An α = 0.008 was used for the series of 6 pair-wise comparisons. Discussion and Conclusions This evidence suggests that course enrollment influences students’ engagement in causal mechanistic reasoning overtime. When students first learn about simple nucleophilic substitutions, Traditional students and OCLUE students alike are able bring to bear the intellectual resources to construct a causal mechanistic explanation. Ideally, students will continue to construct explanations invoking explicit electron discussion in conjunction with electrostatic interactions even after they have learned more complex material. These data suggest that students’ knowledge and their use of that 171 knowledge must be supported overtime for students to continue to engage in this scientific practice. Indeed, OCLUE-OCLUE students were consistent with 65% of students constructing a causal mechanistic response by the end of the course. However, students who were enrolled in OCLUE for the first semester and then Traditional for the second semester quickly regressed from their prior level of reasoning in a matter of weeks when enrolled in a different course, where such reasoning is not expected. Students enrolled in the traditional course for both semesters were least likely to construct causal mechanistic explanations both at the start and end of OC2. Students who transition from one course type to another, either from Traditional to OCLUE or vice versa show similar patterns of reasoning, intermediate between OCLUE-OCLUE and Traditional-Traditional. This phenomenon certainly suggests that consistency overtime is key for developing this knowledge in use for many students. However, we do not know how much “dosage” would be required to move students from a traditional course up to the level of students who have had two semesters of OCLUE. The design of this study allowed us to systematically probe student reasoning over time Implications for Instruction detecting changes in reasoning due to course type. In contrast to the traditional course studied here, OCLUE students are expected to construct explanations on homework and in recitation sessions throughout the semester. Students complete homework independently and are asked to construct written explanations for familiar phenomena but also for phenomena that might be unfamiliar. Students receive feedback on homework in lecture, although not individual feedback; the instructor discusses examples of authentic anonymized student work to highlight key features of successful and less successful explanations. Homework is graded for participation not correctness, but prior work27 has shown that students take these assignments seriously putting forth similar answers to those on their exams. In recitation, students work in groups to complete a single worksheet that emphasizes constructing explanations and receive feedback from a TA. The approach followed in OCLUE is 172 consistent with the following recommendation from The Institute of Educational Sciences: “selectively ask[ing] students to try to answer ‘deep’ questions that focus on underlying causal and explanatory principles” because of the wealth of strong evidence supporting this instructional practice. Additionally, the Framework for K-12 Science Education promotes a vision in which knowledge is used via the scientific practices such as constructing explanation. While this study has shown the effects of switching course types mid-stream, we do not have Implications for Research specific evidence for the cause(s) of the drop-off for OCLUE-Traditional, and the increase for Traditional- OCLUE. The reasons for the changes may stem from different causes. It may be possible to investigate these differences by interviewing students and/or asking targeted survey questions about the students’ perceptions of what is important in the course. The effects of changing course types were only explored in the context of causal mechanistic explanations. While explanation is key to deep understanding, organic chemists communicate primarily with electron-pushing mechanism representations. Future research is needed to understand how the “switchers” use mechanistic arrows as a predictive tool to predict reaction mechanisms and products. Limitations The data in this study were collected on an online homework and research tool that experienced some technical difficulties in fall 2018 and spring 2019 making it difficult to collect data throughout the academic year. As a result, we were not able to completely replicate our study of student reasoning in the middle of OC1 and at the start of OC2. Similarly, we were only able to study students with a Traditional-OCLUE course experience in the second year of this study due to the low number of students who moved from a Traditional course to an OCLUE course in fall 2017 to spring 2018. These data were collected in low-stakes homework assignments and we have not attempted to use these prompts in a summative assessment setting. However, we have attempted to mediate this 173 limitation by replicating the study when possible. Additionally, prior work has shown that students work on these low-stakes homework assignments reflect effort similar to that given on summative assessments. Finally, students’ explanations did contain incorrect ideas such as misuse of vocabulary and identification of this reaction as an SN1 reaction. These inaccuracies were not accounted for in the characterization scheme. Instead, our analysis focused on characterizing the productive resources students invoked to reason about this reaction instead of characterizing student misconceptions. 174 APPENDIX 175 Cohort OCLUE-OCLUE-S18 (N = 103) Traditional-Traditional-S18 (N = 128) OCLUE-Traditional-S18 (N = 195) OCLUE-OCLUE-S19 (N = 64) Traditional-Traditional-S19 (N = 85) OCLUE-Traditional-S19 (N = 93) Traditional-OCLUE-S19 (N = 67) Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Mean 27.3 3.6 3.5 3.7 3.8 3.5 Median 27 4.0 3.5 3.8 4.0 3.5 70% Female Major: 91% Preprofessional/Health Science; 0% Plant and Animal Science; 4% Physical Science and Engineering; 5% Other 30% Male Gender ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Gender 26.7 3.5 3.4 3.6 3.7 3.4 27 3.5 3.5 3.7 4.0 3.5 30% Male 70% Female Major: 78% - Preprofessional/Health Science; 8% Plant and Animal Science; 2% Physical Science and Engineering; 12% Other ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Gender 27.5 3.5 3.4 3.6 3.7 3.4 28 3.5 3.5 3.8 4.0 3.5 34% Male 66% Female Major: 90% Preprofessional/Health Science; 7% Plant and Animal Science; 1% Physical Science and Engineering; 3% Other ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade 25.6 3.5 3.5 3.6 3.6 3.1 25 3.5 3.8 3.7 4.0 3.0 62% Female Major: 98% Preprofessional/Health Science; 0% Plant and Animal Science; 2% Physical Science and Engineering; 0% Other 38% Male Gender ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Gender 26.7 3.4 3.4 3.6 3.3 3.7 28% 27 3.5 3.5 3.7 3.5 4.0 72% Major: 88% Preprofessional/Health Science; 4% Plant and Animal Science; 0% Physical Science and Engineering; 8% Other ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade 26.7 3.4 3.4 3.6 3.5 3.5 27 3.5 3.5 3.7 3.5 4.0 68% Female Major: 86% Preprofessional/Health Science; 5% Plant and Animal Science; 5% Physical Science and Engineering; 4% Other 32% Male Gender ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Gender 26.8 3.3 3.2 3.6 3.2 3.0 27 3.5 3.5 3.7 3.5 3.0 39% Male 61% Female Major: 85% Preprofessional/Health; 5% Plant and Animal Science; 4% Physical Science and Engineering; 6% Other Table 6.5. Descriptive statistics for the OCLUE-OCLUE, Traditional-Traditional, OCLUE-Traditional, and Traditional-OCLUE cohorts. 176 Table A6.5 (cont’d) Traditional-Traditional-S19 (N = 85) OCLUE-Traditional-S19 (N = 93) Traditional-OCLUE-S19 (N = 67) ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Gender 26.7 3.4 3.4 3.6 3.3 3.7 28% 27 3.5 3.5 3.7 3.5 4.0 72% Major: 88% Preprofessional/Health Science; 4% Plant and Animal Science; 0% Physical Science and Engineering; 8% Other ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Gender 26.7 3.4 3.4 3.6 3.5 3.5 27 3.5 3.5 3.7 3.5 4.0 32% Male 68% Female Major: 86% Preprofessional/Health Science; 5% Plant and Animal Science; 5% Physical Science and Engineering; 4% Other ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Gender 26.8 3.3 3.2 3.6 3.2 3.0 27 3.5 3.5 3.7 3.5 3.0 39% Male 61% Female Major: 85% Preprofessional/Health; 5% Plant and Animal Science; 4% Physical Science and Engineering; 6% Other OCLUE-OCLUE-S18 (N = 103) compared to Trad-Trad-S18 (N = 128) OCLUE-OCLUE-S18 (N = 103) compared to OCLUE-Trad-S18 (N = 195) Trad-Trad-S18 (N = 128) compared to OCLUE-Trad-S18 (N = 195) OCLUE-OCLUE-S19 (N = 64) compared to Trad-Trad-S19 (N = 85) Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Mann-Whitney U 5977.0 5859.0 5892.0 6179.0 5921.0 5892.0 Mann-Whitney U 9805.5 9143.0 9788.5 965.5 9335.5 9788.5 Mann-Whitney U 10886.0 12227.0 11601.5 12222.5 11979.0 11601.5 Mann-Whitney U 2228.5 2443.5 2382.5 2684.5 1984.5 1448.0 Z -1.224 -1.560 -1.455 -0.819 -1.639 -1.455 Z -0.336 -1.370 -0.380 -0.534 -1.161 -0.380 Z -1.950 -0327 -1.126 -0.314 -0.644 -1.126 Z -1.893 -0.992 -1.364 -0.136 -3.012 -5.360 p-value 0.221 0.119 0.146 0.413 0.101 0.146 p-value 0.737 0.171 0.704 0.593 0.246 0.704 p-value 0.051 0.743 0.260 0.753 0.520 0.260 p-value 0.058 0.321 0.173 0.892 0.003 < 0.001 Effect Size r Effect Size r Effect Size r Effect Size r 0.247 0.439 Table 6.6. Non-parametric comparisons between OCLUE-OCLUE, Traditional-Traditional, OCLUE- Traditional, and Traditional-OCLUE cohorts. 177 Table A6.6 (cont’d) OCLUE-OCLUE-S19 (N = 64) compared to OCLUE-Trad-S19 (N = 93) OCLUE-OCLUE-S19 (N = 64) compared to Trad-OCLUE-S19 (N = 67) Trad-Trad-S19 (N = 85) compared to Trad-OCLUE-S19 (N = 67) OCLUE-Trad-S19 (N = 93) compared to Trad-OCLUE-S19 (N = 67) Trad-Trad-S19 (N = 85) compared to OCLUE-Trad-S18 (N = 93) OCLUE-OCLUE-S18 (N = 103) compared to Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Measure ACT GC1 Course Grade OCLUE-OCLUE-S19 GC2 Course Grade (N = 64) Trad-Trad-S18 (N = 128) compared to Trad-Trad-S19 (N = 85) OCLUE-Trad-S18 (N = 195) compared to OCLUE-Trad-S19 (N = 93) GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Measure ACT GC1 Course Grade GC2 Course Grade GPA Prior to OC1 OC1 Course Grade OC2 Course Grade Mann-Whitney U 2375.5 2906.5 2690.5 2861.5 2467.0 2089.0 Mann-Whitney U 1711.5 1857.0 1684.0 1951.5 1486.0 2118.0 Mann-Whitney U 2788.0 2685.5 2572.5 2566.5 2795.0 1593.0 Mann-Whitney U 3086.0 2796.5 2714.5 2966.5 2607.0 2233.5 Mann-Whitney U 3873.5 3662.5 3849.5 3744.5 3415.0 3192.5 Mann-Whitney U 2360.0 2957.5 3160.0 3089.5 2837.5 2349.0 Mann-Whitney U 5275.5 4916.5 5408.0 5358.5 3643.5 3568.0 Mann-Whitney U 7850.5 8752.0 8712.0 8569.0 6690.0 8424.5 178 Z -2.155 -0.264 -1.078 -0.409 -1.968 -3.341 Z -2.001 -1.382 -2.216 -0.887 -3.226 -0.123 Z -0.222 -0.500 -1.058 -1.044 -0.202 -5.156 Z -0.103 -1.156 -1.440 -0.516 -1.835 -3.230 Z -0.231 -0.750 -0.313 -0.607 -1.637 -2.584 Z -3.092 -1.209 -0.477 -0.680 -1.868 -3.247 Z -0.375 -1.105 -0.076 -0.185 -4.477 -4.624 Z -1.849 -0.508 -0.568 -0.755 -4.033 -1.043 p-value 0.031 0.792 0.281 0.682 0.049 0.001 p-value 0.045 0.167 0.027 0.375 0.001 0.902 p-value 0.825 0.617 0.290 0.296 0.840 < 0.001 p-value 0.918 0.248 0.150 0.606 0.066 0.001 p-value 0.817 0.453 0.754 0.544 0.102 0.010 p-value 0.002 0.227 0.634 0.496 0.062 0.001 p-value 0.707 0.269 0.940 0.853 < 0.001 < 0.001 p-value 0.064 0.611 0.570 0.450 < 0.001 0.297 Effect Size r 0.267 Effect Size r 0.282 Effect Size r 0.418 Effect Size r 0.255 Effect Size r Effect Size r 0.239 0.251 Effect Size r 0.307 0.317 Effect Size r 0.238 REFERENCES 179 REFERENCES (1) (2) (3) (4) Cooper, M.; Klymkowsky, M. Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform. Journal of Chemical Education 2013, 90 (9), 1116–1122. https://doi.org/10.1021/ed300456y. Sevian, H.; Talanquer, V. Rethinking Chemistry: A Learning Progression on Chemical Thinking. Chem. Educ. Res. Pract. 2014, 15, 10–23. Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96, 1858–1872. Flynn, A. B.; Ogilvie, W. W. Mechanisms before Reactions: A Mechanistic Approach to the Organic Chemistry Curriculum Based on Patterns of Electron Flow. J. Chem. Educ. 2015, 92 (5), 803–810. (5) McGill, T. L.; Williams, L. C.; Mulford, D. R.; Blakey, S. B.; Harris, R. J.; Kindt, J. T.; Lynn, D. G.; Marsteller, P. A.; McDonald, F. E.; Powell, N. L. Chemistry Unbound: Designing a New Four-Year Undergraduate Curriculum. J. Chem. Educ. 2019, 96, 35–46. (6) (7) (8) (9) Stains, M.; Harshman, J.; Barker, M. K.; Chasteen, S. V.; Cole, R.; DeChenne-Peters, S. E.; Eagan, M.; Esson, J. M.; Knight, J. K.; Laski, F. A.; Levis-Fitzgerald, M.; Lee, C. J.; Lo, S. M.; McDonnell, L. M.; McKay, T. A.; Michelotti, N.; Musgrove, A.; Palmer, M.; Plank, K.; Rodela, T.; Sanders, E.; Schimpf, N.; Schulte, P.; Smith, M.; Stetzer, M.; Van Valkenburgh, B.; Vinson, E.; Weir, L. K.; Wendel, P. J.; Wheeler, L. B.; Young, A. M. Anatomy of STEM Teaching in North American Universities. Science 359 (6383), 1468–1470. Borman, S. A Tale of Two Textbooks - History and Influence of Sienko & Plane and Morrison and Boyd Detailed at Symposium. Chem. Eng. News 83, 48–51. Stowe, R. L.; Cooper, M. M. Practicing What We Preach: Assessing “Critical Thinking” in Organic Chemistry. J. Chem Educ. 2017, 94 (12), 1852–1859. https://doi.org/10.1021/acs.jchemed.7b00335. Head, J.; Bucat, R.; Mocerino, M.; Treagust, D. F. Exploring Students’ Abilities to Use Two Different Styles of Structural Representation in Organic Chemistry. Can. J. Math Sci. and Tech. Educ. 5 (1), 133–152. (10) Bhattacharyya, G. Practitioner Development in Organic Chemistry: How Graduate Students Conceptualize Organic Acids. Chem. Educ. Res. Pract. 2006, 7 (4), 240–247. https://doi.org/10.1039/B5RP90024G. (11) Cartrette, D. P.; Mayo, P. M. Students’ Understanding of Acids/Bases in Organic Chemistry Contexts. Chem. Educ. Res. Pract. 2011, 12 (1), 29–39. https://doi.org/10.1039/C1RP90005F. 180 (12) Bretz, S. L.; McClary, L. Students’ Understandings of Acid Strength: How Meaningful Is Reliability When Measuring Alternative Conceptions? J. Chem. Educ. 2015, 92 (2), 212–219. https://doi.org/10.1021/ed5005195. (13) McClary, L.; Bretz, S. L. Development and Assessment of A Diagnostic Tool to Identify Organic Chemistry Students’ Alternative Conceptions Related to Acid Strength. Int. J. Sci. Educ. 2012, 34 (15), 2317–2341. (14) Rushton, G. T.; Hardy, R. C.; Gwaltney, K. P.; Lewis, S. E. Alternative Conceptions of Organic Chemistry Topics among Fourth Year Chemistry Students. Chem. Educ. Res. Pract. 9, 122–130. (15) Cartrette, D. P.; Bodner, G. M. Non-Mathematical Problem Solving in Organic Chemistry. J. Res. Sci. Teach. 47 (6), 643–660. (16) Topczewski, J. J.; Topczewski, A. M.; Tang, H.; Kendhammer, L. K.; Pienta, N. J. NMR Spectra through the Eyes of a Student: Eye Tracking Applied to NMR Items. J. Chem. Educ. 2017, 94 (1), 29–37. (17) Connor, M. C.; Finkenstaedt-Quinn, S. A.; Shultz, G. V. Constraints on Organic Chemistry Students’ Reasoning during IR and 1H NMR Spectral Interpretation. Chem. Educ. Res. Pract. 20, 522–541. (18) Ege, S. N.; Coppola, B. P.; Lawton, R. G. The University of Michigan Undergraduate Chemistry Curriculum 1. Philosophy, Curriculum, and the Nature of Change. J. Chem. Educ. 74 (1), 74. (19) Malinak, S. M.; Bayline, J. L.; Brletic, P.; Harris, M. F.; luliucci, R. J.; Leonard, M. S.; Matsuno, N.; Pallack, L. A.; Stringfield, T. W.; Sunderland, D. P. The Impacts of an “Organic First” Chemistry Curriculum at a Liberal Arts College. J. Chem. Educ. 91 (7), 994–1000. (20) Christiansen, M. A. Inverted Teaching: Applying a New Pedagogy to a University Organic Chemistry Class. J. Chem. Educ. 91 (11), 1845–1850. (21) Flynn, A. B. Structure and Evaluation of Flipped Chemistry Courses: Organic & Spectroscopy, Large and Small, First and Third Year, English and French. Chem. Educ. Res. Pract. 16, 198–211. (22) Fautch, J. M. The Flipped Classroom for Teaching Organic Chemistry in Small Classes: Is It Effective? Chem. Educ. Res. Pract. 16, 179–186. (23) Mooring, S. R.; Mitchell, C. E.; Burrows, N. L. Evaluation of a Flipped, Large-Enrollment Organic Chemistry Course on Student Attitude and Achievement. J. Chem. Educ. 93 (12), 1972–1983. (24) McPadden, D.; Brewe, E. Impact of the Second Semester University Modeling Instruction Course on Students’ Representation Choices. Phys. Rev. Physics Educ. Res. 13 (2), 1–15. (25) Bryfczynski, S. BeSocratic: An Intelligent Tutoring System for the Recognition, Evaluation, and Analysis of Free-Form Student Input; Clemson University, Clemson, SC, 2012; Vol. Ph.D. Dissertation. 181 (26) MacDonald, P. L.; Gardner, R. C. Type 1 Error Rate Comparisons of Post Hoc Procedures for I j Chi- Square Tables. J. Chem. Educ. 60 (5), 735–754. (27) Noyes, K.; Cooper, M. M. Investigating Student Understanding of London Dispersion Forces: A Longitudinal Study. J. Chem. Educ. 2019, 96 (9), 1821–1832. 182 CHAPTER VII: THE EFFECT OF SCAFFOLDING ON CAUSAL MECHANISTIC REASONING Introduction In 1976, Wood, Bruner, and Ross published pioneering work on the various modes of tutoring to support young learners in a novel task.1 Their work coined the term “scaffolding” as a metaphor for helping a learner complete a task that would otherwise be too difficult for them if left unsupported. Wood et al. suggested various functions of scaffolding such as reducing the degrees of freedom, marking critical features, direction maintenance and frustration control to focus student thinking. The scaffolds in Wood et al.’s study were adult tutors supporting young children in their quest to solve a block puzzle. It has been suggested that these functions of scaffolding were successful because they provided support for what Vygotsky called the students’ Zone of Proximal Development.2,3 The Zone of Proximal Development is “the distance between the actual development level as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance with more capable peers”(p. 86).4 In other words, the Zone of Proximal Development describes what students could be capable of with assistance. Bruner suggested that the tutor provides scaffolding to: “[do] what the child could not do. For the rest, she made things such that the child could do with her what he plainly could not do without her. And as tutoring proceeded, the child took over parts from her part of the task that he was not able to do at first, but with mastery, became consciously able to do under his own control” (p. 76).3 The term scaffolding itself is an analogy for a temporary support used to accomplish some task that is then removed when it is no longer needed.5 The above passage by Bruner refers to key characteristics of scaffolding, mainly the gradual fading of the scaffold and shifting responsibility to the learner without the scaffold.5 McNeill et al. investigated the effect of fading instructional scaffolds by 183 assigning students to two different learning conditions: constructing explanations with continuous scaffolding and constructing explanations with scaffolding that is gradually faded overtime.6 Over the course of this 8-week study, seventh-grade students received either continuous scaffolding to support them in their construction of scientific explanations or they began with scaffolding that was slowly faded away while still being engaged in explanation. Post-test measures found that students who were exposed to faded scaffolding conditions were better able to construct explanations for reduced scaffolding prompts about structure/property relationships compared to students who were supported continuously with scaffolding. This finding suggests that fading instructional scaffolds overtime is beneficial for students to construct explanations later on with or without scaffolding.6 Noyes and Cooper found that careful scaffolding improved students’ mechanistic drawings of London Dispersion Forces.7 The drawing scaffolds in this study provided students with three consecutive boxes to elicit drawings containing causes, effects and underlying mechanistic processes. In two post- test measures (one at the start of the next semester and one at the end of the two-semester course) students gave fewer causal mechanistic drawing responses compared to those elicited using the three- box scaffold. Students in this study7 did not receive instruction using faded scaffolds like that observed in the 8-week McNeill et al. study6, thus, the sudden drop in causal mechanistic drawings suggests the scaffolding itself was a key factor in student success constructing causal mechanistic drawings. The study presented below builds on this prior work by investigating student engagement in causal mechanistic reasoning across various scaffolding designs and over multiple time points. 184 This study is guided by these research questions: Research Questions 1. How does reduced scaffolding affect student engagement in causal mechanistic reasoning and mechanistic arrow drawings for students enrolled in transformed and traditional organic chemistry courses? 2. How does expanded scaffolding affect student engagement in causal mechanistic reasoning and mechanistic arrow drawings for students enrolled in transformed and traditional organic chemistry courses? 3. How consistent are students in engaging in causal mechanistic reasoning within each prompt from the start to the end of OC2? 4. How consistent are students in engaging in causal mechanistic reasoning across multiple prompt structures? Methods Design of Assessment Tasks Prior work has shown that students’ engagement in causal mechanistic reasoning was influenced by the nature of the prompt, specifically, asking students to “describe what is happening” and then separately asking “explain why” elicited more causal mechanistic responses than when these tasks were combined into one prompt.8 However, this prior work was conducted with general chemistry students with a simple acid-base reaction. We were interested in understanding how changes in scaffolding might influence reasoning for organic chemistry students in the context of simple nucleophilic substitution reactions. 185 This study investigates student reasoning about two prompt types: an SN2 intramolecular Reduced Scaffolding Prompt reaction with reduced scaffolding and an SN1 reaction with increased scaffolding. The reduced scaffolding prompt presented students with Lewis structures for the reactants and the products for the intramolecular reaction of 6-bromohexan-2-olate. This intramolecular reaction was chosen because it proceeds via a simple SN2 process with a strong nucleophile and leaving group. The goal was to direct students toward how and why this process occurs rather than distracting students’ thinking about other reaction mechanisms. Students were prompted to “Please explain how and why this reaction occurs” and provided a single box to enter their response. Students were then presented with the Lewis structures and asked “Now, please draw mechanistic arrows in the BLUE box to represent how this reaction occurs.” This prompt is shown in Figure 7.1. Please explain how and why this reaction occurs. Please draw mechanistic arrows in the BLUE box to represent how this reaction occurs. Figure 7.1. Reduced scaffolding prompt administered via beSocratic9. Expanded Scaffolding Prompt The second prompt structure investigated in this study asked students about an SN1 reaction with expanded scaffolding. The expanded prompt structure was modified from the prompt used in the prior studies discussed in Chapters V and VI of this dissertation. In this expanded prompt, students were 186 presented with Lewis structures for the reaction of t-butyl bromide with an iodide ion. Products of the reaction are provided and methanol as solvent over the reaction arrow as shown in Figure 7.2. Students were asked to explain “How would you classify this reaction? Please explain why you chose that classification” (Figure 7.2A). Next, students were presented with just the starting materials and were prompted to draw a mechanism for the first step of this reaction and the products of that first step. Next, they were asked to describe the events of this first step, explain why it occurred, and explain how the first step occurred. The same was repeated for a second step (Figure 7.2C). Finally, students were asked if they have any final steps to add to their mechanism (Figure 7.2D). This expanded scaffolded prompt was designed to hint to students to expand their explanation into steps to give more fine-grained reasoning. Data from pilot studies with this SN1 reaction showed that students often tried to consolidate their reasoning for both steps. This expanded scaffolding might have also prompted students to consider that this as a two-step SN1 reaction, however, as discussed in the Results section, many students from both OCLUE and Traditional courses still identified and discussed this as an SN2 reaction. As we were most interested in students’ reasoning rather than identifying misconceptions in their knowledge, the prompt was specifically designed to engage students in explanation and not argumentation about whether this is an SN2 or SN1 process. The expanded scaffolding prompt was also designed to elicit student mechanistic arrow use in the drawing portion of the prompt. beSocratic9 drawing modules allow students to build on their prior work by allowing drawings from a prior slide to be presented back to the student on a later slide where they can add and subtract from their prior work. We used this feature to help students build up their reaction mechanism drawing step by step as they proceeded through the activity. 187 A How would you classify this reaction? Please explain why you chose that classification. B We can think about this mechanism in steps. Draw a mechanism for the first step of this reaction and draw the products of this first step in the BLUE box. Describe sequences of events at the molecular level for the first step of this reaction. Why does the first step occur? What is the cause of this step? How exactly does this first step occur at the molecular level? C Here is the mechanism you drew for step 1. Draw a mechanism for the second step of this reaction and draw the products of this second step in the BLUE box. Describe the sequence of events at the molecular level for the second step of this reaction. Why does this second step occur? What is the cause of this step? How exactly does this second step occur at the molecular level? D Here is the mechanism you drew for steps 1 and 2. Are there any other steps you would like to add? If so, add them in the BLUE box. Explain any additions that you made. Figure 7.2. Structure of the Expanded Scaffolded Prompt administered across four slides via beSocratic.9 188 Student participants included in this study were selected based on their enrollment in Student Participants transformed organic chemistry, OCLUE, or a traditional organic course for two consecutive semesters using the criteria described in the methods sections in Chapters V and VI. Participants reported on in this study are OCLUE-OCLUE-S18 cohort (N = 103), Traditional-Traditional-S18 cohort (N = 128) for the 2017- 2018 academic year and OCLUE-OCLUE-S19 cohort (N = 64) and Traditional-Traditional-S19 cohort (N = 85) for the 2018-2019 academic year. These are the same students reported on in Chapter IV of this dissertation. Data Collection Data were collected via an online homework system called beSocratic9 introduced in the prior studies. In the 2017-2018 academic year, data were collected at the start and end of OC2 in both OCLUE and Traditional courses referred to as Start OC2-S18 and End OC2-S18, respectively. In the 2018-2019 academic year, data were only collected at the end of OC2 referred to as End OC2-S19. These activities described in Figures 7.1 and 7.2 were administered as part of the same regular homework assignments for OCLUE students and as an extra credit assignment for Traditional students as described in prior studies. Data Analysis The coding schemes reported below are modified from the previously published coding scheme used to characterize causal mechanistic reasoning for the reaction of HCl with H2O.8 The definition of causal mechanistic reasoning does not change in each scheme, but examples of responses for each characterization are provided for each prompt type. Characterization schemes for mechanistic arrow drawings are also explained for each prompt (Table 7.1). Responses for both prompt types were analyzed by the author with the assistance of a trained undergraduate coder, and we discuss each data analysis procedure and the characterization schemes below. 189 Reduced Scaffolding Prompt Characterization Schemes In the Reduced Scaffolding Prompt, students were asked to reason about an intramolecular reaction proceeding via an SN2 reaction mechanism. Regardless, some students explained an SN1 process by describing the Bromide leaving the Carbon before the Oxygen atom approached. Responses were tagged with either an SN2 or an SN1 tag depending on which process they described as shown in Table 7.2. These codes were mutually exclusive and assigned in addition to the causal mechanistic characterizations described in Table 7.1. Very few students chose not to engage with the prompt but the few who did were assigned No Response (NR). Additionally, students who provided response such as Kara’s “It is a heat reaction with methyl shifts” were characterized as Non-Normative. Students who only described simple bond breaking and forming were characterized as Descriptive General as exemplified by Brittany’s reasoning “The O acts as a nucleophile and the bromine leaves to create a ring.” Brittany’s use of the term nucleophile is still considered descriptive as there is no explicit evidence that the student understands the meaning of the term nucleophile in terms of electrostatic interactions and causality of this reaction. Explicit evidence of engagement in causal reasoning is demonstrated by Adam who explains that “The O is attracted to the partial positive on the C.” Responses of this nature were characterized as Descriptive Causal. However, some students did not invoke ideas about electrostatic attractions but did discuss explicit activities of electrons as demonstrated in Miranda’s response “The electron pair between C and Br is completely transferred to the Br while the O forms a bond.” Ideally, students would combine these two elements to give a causal mechanistic response. Marie demonstrates this reasoning when she says “The positive charge on the carbon attracts the oxygen, which is negatively charged, so they begin to form a bond. At the same time, the Br takes the electrons in its bond with C and leaves the molecule…” Just as with the reaction of CH3Br, there are other causal factors such as collisions between reacting entities and activation energies that are not specifically prompted for here and little evidence of 190 reasoning with these causal factors were observed. These characterizations are summarized in Table 7.1 and are mutually exclusive. Code and Code Description No Response (NR) Student does not provide an answer or does not even attempt to answer. Non-Normative (NN) Student provides a non-normative or unrelated explanation. Descriptive General (DG) Student provides a scientifically simplistic description of bond breaking or bond formation. Descriptive Causal (DC) Student discusses electrostatic attraction between species. For the Reduced Scaffolding, student gives evidence that they understand that there is an attraction between the alkoxide and the partial positive carbon adjacent the leaving group. For the Expanded Scaffolding prompt, student gives evidence that they understand that the Iodide ion is attracted to carbocation intermediate. Descriptive Mechanistic (DM) Student identifies electrons or lone pairs as the entities responsible for the reaction and explains their activities that lead to bond formation/bond breaking. Causal Mechanistic (CM) Student provides both the causal and mechanistic account of the reaction. Evidence that the student understands that the lone pair of electrons are attracted to the carbon adjacent the leaving group to form a bond. Examples of Student Responses Reduced Scaffolding Prompt Intramolecular reaction of 6- bromohexan-2-olate Jason: “I don’t remember this reaction.” Examples of Student Responses Expanded Scaffolding Prompt Reaction of tert-butyl bromide with iodide Margaret: “I don’t know.” Kara: “It is a heat reaction with methyl shifts.” Ray: “The solvent makes the reaction happen.” Brittany: “The O acts as a nucleophile and the bromine leaves to create a ring.” Benjamin: “The bond between C and Br breaks then the nucleophile can now attack the carbon.” Adam: “The O is attracted to the partial positive on the C attached to the Br and the Br leaves.” Rebecca: “The Bromine is the leaving group. The charge on the oxygen will attack carbon 1 because its attracted to its positive charge. This will kick out the Bromine and form a ring.” Kimberly: “The negative charge on the Iodine associates with the positive on the carbon and forms a bond.” Aaron: “The Bromine leaves and then there is a carbocation that the Iodide is attracted to so you get a new bond.” John: “The lone pairs from the O make the bond with the carbon and Br leaves.” Miranda: “The electron pair between C and Br is completely transferred to the Br while the O forms a bond.” Marie: “The positive charge on the carbon attracts the oxygen, which is negatively charged, so they begin to form a bond. At the same time, the Br takes the electrons in its bond with C and leaves the molecule...” Michael: “The Br will attract the electrons from the carbon bond it has, resulting in a positive charge on the carbon which then bonds with the lone pairs on the oxygen forming a ring.” Paige: “The electrons favor the more electronegative atom (Br) and then leave with the Br causing a carbocation and a bromine anion.” Joseph: “The iodine shares its two electrons with the carbon to form a single bond.” Warren: “The electrons between Br and C leave with the Br. After the Br leaves there is a positive charge on the C, thus the negative charge of the I attacks it.” Tanya: “The lone pair on the OMe attack the +C forming (CH3)3COHMe. The lone pair are attracted to the carbon and a bond is formed” Table 7.1. Causal Mechanistic Reasoning Characterization Scheme for the Reduced Scaffolding Prompt and Expanded Scaffolding Prompt. 191 Tag and Description SN1 Tag The response describes and SN1 mechanism. The student describes the leaving group leaving forming a carbocation and then the nucleophile approaches to form a bond. SN2 Tag The response describes an SN2 mechanism. The student clearly describes the nucleophile approach as initiating the reaction and the leaving group leaving. Example of Student Response Reduced Scaffolding Prompt Intramolecular reaction of 6- bromohexan-2-olate “First the bromine falls off and leaves carbon positive. Then the oxygen can attack with its electrons.” Example of Student Response Expanded Scaffolding Prompt Reaction of tert-butyl bromide with iodide “The bromine leaves because it interacts with MeOH. Then the carbocation reacts with iodide.” “The electrons on oxygen attract to the carbon and then the electrons shift from the carbon-bromine bond to the bromine and it leaves.” “The iodine attracts the carbon, which makes the bromine break off the carbon, and make the iodine attach to the carbon instead.” Table 7.2. Summary of tags assigned to an explanation response when warranted. These tags are assigned in addition to the Causal Mechanistic Characterization codes. After constructing an explanation, students were prompted to drawn mechanistic arrows onto the Lewis structures as shown in Figure 7.3. Students were expected to draw a mechanistic arrow starting from the Oxygen atom to the Carbon located next to the Bromine and then a second arrow that starts at the Carbon-Bromine bond and ends at the Bromine atom as shown in Figure 7.3A. Students who drew anything else such as the examples shown in Figure 7.3B and Figure 7.3C were coded as incorrect mechanistic arrow drawings. In addition to showing a static picture of students’ drawings, beSocratic9 also has the capability to record students’ work stroke by stroke. Using this feature, we were able to replay student responses to determine the order in which students drew their mechanistic arrows. Without this additional analysis, all correct responses like the one shown in Figure 7.3A look the same. Similar to the analysis of the SN2 reaction in Chapter V, we expected students would first draw an arrow from the Oxygen atom to the Carbon atom and then draw a second arrow representing the leaving group leaving. Drawings of this nature were tagged with an SN2 Arrows Tag. Students who drew the leaving group leaving before the approach of the Oxygen were assigned an SN1 Arrows Tag as shown in Table 7.3. For this reduced scaffolding prompt, training sets of 30 random explanation responses were coded separately until a Kappa of 0.85 was obtained between the two coders. Once this level of 192 agreement was reached, the undergraduate coder and author divided the responses evenly and coded the data set for both cohorts at both time points. The same process was repeated to code the drawn mechanistic arrows. Figure 7.3. Examples of correct static mechanistic arrow drawing (A) and incorrect static mechanistic arrow drawings (B and C) for the Reduced Scaffolding Prompt. Mechanistic Arrow Use Explanation Description SN2 Arrows Tag Student drew the arrow from the Oxygen to the Carbon first and then the arrow from the Carbon-Bromine bond to the Bromine second. SN1 Arrows Tag Student drew the arrow from the Carbon-Bromine bond first and the arrow from the Oxygen second. Table 7.3. Classification scheme for non-static coding of mechanistic arrow drawings for the Reduced Scaffolding prompt with the intramolecular reaction of 6-bromohexan-2-olate. Expanded Scaffold Prompt Characterization Schemes In the expanded scaffolding prompt, students were asked to reason about an SN1 reaction – tert- butyl bromide with an iodide ion. Students were provided with methanol as solvent over the reaction 193 arrow to aid in their explanation about how the leaving group might leave to form a carbocation. Though clearly an SN1 process, some students still described an SN2 process. Explanation responses were identified as either SN1 or SN2 with the explanation tags listed in Table 7.2. Student responses were characterized according to the causal mechanistic coding scheme in Table 7.1 regardless of whether they discussed an SN1 or an SN2 process. All explanations and arrow drawings were analyzed by the author a trained undergraduate coder. Training sets of 30 random explanation responses and explanations were coded separately and then compared and discussed until all coding discrepancies were resolved. Once 100% agreement was reached for explanations and drawings on trainings sets, both coders analyzed the expanded scaffolding data set separately and all responses were compared to ensure all responses were coded correctly. Any discrepancies were discussed until agreement was reached. Percent agreement was used for this data set since there were numerous codes and mechanistic pathways that students could have discussed in their explanations and drawing. This prompt structure elicited student explanations in multiple steps as shown in Figure 7.2. We considered all written explanation pieces together and assigned a single causal mechanistic characterization for the whole response. Just as with the reduced scaffolding prompt, some students did not engage with the prompt and were characterized as No Response or posed Non-Normative explanations. Students who merely described the sequence of events but never developed their response further than that were characterized as Descriptive General such as Benjamin’s response “The bond between C and Br breaks and the nucleophile can now attack the carbon.” A Descriptive Causal response for this prompt had to describe the Iodide’s attraction to the positively charge carbon. In instances where students described an SN1, students would identify a carbocation as the positive entity as exemplified in Aaron’s response “The Bromine leaves and then there is a carbocation that the Iodide is attracted to so you get a new bond.” In instances where students described an SN2 process, students 194 would still identify an attraction between the carbon and the iodide even if they had not formed a carbocation. A response characterized as descriptive mechanistic does not explicitly discuss attractions but does explicitly discuss electron movement as exemplified by Joseph when he reasoned that “The iodine shares its two electrons with the carbon to form a single bond.” Our goal is for students to explicitly discuss electrostatic interactions and electron movement together as exemplified by Warren when he stated “The electrons between Br and C leave with the Br. After the Br leaves there is a positive charge on the C, thus the negative charge of the I attacks it.” For the Expanded Scaffolding prompt, students constructed mechanistic arrow drawings alongside their explanations for each step of the reaction. As discussed in the Data Collection section above, students developed their mechanistic arrow drawings step by step. We reviewed all three parts of their drawings simultaneously to understand how the drawings changed as the students progressed through the activity. Several possible pathways emerged and are summarized in Figure 7.4. The goal was for students to construct a canonically correct SN1 mechanism for this reaction where the leaving group is removed in the first step followed by the explicit formation of a carbocation. Next, the student would represent the nucleophile forming a bond at the carbocation and draw the final product. These reaction conditions contained two possible nucleophiles: the Iodide ion and the Oxygen atom in methanol. These pathways are shown in Figure 7.4A and Figure 7.4B and were both considered canonically correct. In some cases, students included full mechanisms but failed to draw final product (Figure 7.4C). We also observed that some students failed to draw any mechanistic arrows and only drew structures for intermediates and products (Figure 7.4D). Additionally, some students represented an SN2 process by drawing two mechanistic arrows in a single step and drew the substitution product but did include a carbocation intermediate (Figure 7.4E). Finally, some drawings were non-normative. For example, some students appeared to be using the carbon in methanol as a nucleophile or 195 represented the product as an alcohol rather than an ether. Some students formed methane as the product. Examples of these non-normative pathways are shown in Figure 7.4F and Figure 7.4G. Figure 7.4. Examples of various mechanistic pathways observed in student drawings. These examples were recreated in ChemDraw for clarity with student work represented in blue. A: Example of canonically correct mechanistic pathway for the formation of t-butyl iodide. B: Example of canonically correct mechanistic pathway for the formation of an ether product. C: Example of response that omitted a final product. D. Example of response that did not include any mechanistic arrows. E: Example of response representing an SN2 process. F and G: Examples of non-normative responses. 196 Results and Discussion RQ 1: How does reduced scaffolding affect student engagement in causal mechanistic reasoning and mechanistic arrow drawings for students enrolled in transformed and traditional organic chemistry courses? Finding 1a: Reduced scaffolding elicited less causal mechanistic reasoning compared to the CH3Br prompt. The Reduced Scaffolding prompt elicited less causal mechanistic reasoning and more descriptive causal reasoning (meaning causal-only) than prompts used in prior studies (Chapters V and VI). At the start of OC2, only 15% of students in OCLUE-OCLUE and Traditional-Traditional cohorts constructed causal mechanistic explanations (Figure 7.5D). More commonly, students constructed descriptive causal explanations meaning they discussed electrostatic interactions but did not invoke explicit discussion of electron movement. This was true at both the start and the end of OC2 for both cohorts. At the start of OC2, 53% of OCLUE students and 45% of Traditional students constructed descriptive causal explanations. Similar trends persisted at the end of OC2 with 51% of OCLUE students and 49% of Traditional students constructing a causal-only response as shown in Figure 7.5E. The overall decreased engagement in causal mechanistic reasoning with this prompt was replicated in S19 (Figure 7.5F). Indeed, for Traditional students in both years, 25% of students constructed descriptive general responses, 50% constructed descriptive causal responses, and less than 10% constructed a causal mechanistic response at the end of OC2 as shown in Figure 7.5F. Interestingly, we observed a slightly different trend in OCLUE students’ reasoning in S19 compared to S18. In S19, 38% OCLUE students constructed descriptive general responses which is equal to number of descriptive causal responses. Next, we compared the proportion of descriptive causal responses to all other characterization categories using a Chi-Square test and found no differences between engagement in descriptive causal reasoning for OCLUE and Traditional at any of the three time points (Table 7.5). For 197 example, at the start of OC2 in S18, the proportion of OCLUE students who constructed a descriptive causal explanation is 53% which leaves 47% of the responses as non-descriptive causal. We compared this proportion to the 45% of Traditional students who responded with a descriptive causal response and the subsequent 65% who did not. It is encouraging to find that the majority of students in both course types invoked knowledge of electrostatic interactions. Explicit discussion of electrostatics was captured by both descriptive causal and causal mechanistic characterizations. Adding these proportions together for each group we find that between 50 and 70% of students invoked electrostatics in their reasoning. Conducting a similar analysis for explicit discussion of electron movement, we added together the proportion of descriptive mechanistic and causal mechanistic characterizations. We found that no more than 25% of students in either cohort invoked explicit discussion of electrons at any time point. Indeed, it appears from these findings that the reduced scaffolding failed to activate student resources about electron movement. Mechanistic reasoning is often implicitly assessed in traditional organic chemistry courses by asking students to construct reaction mechanisms or predict a set of products, thus it is necessary to analyze these students’ mechanistic drawings to better understand student thinking in this context. 198 Figure 7.5: Characterization of Causal Mechanistic Reasoning for the Reduced Scaffolding Prompt. NR- No Response, NN-Non-Normative, DG-Descriptive General, DC-Descriptive Causal, DM-Descriptive Mechanistic, CM-Causal Mechanistic. 199 Table 7.4. Distribution of reasoning characterizations at each time point for three types of prompt. NR- No Response, NN-Non-normative, DG-Descriptive General, DC-Descriptive Causal, DM-Descriptive Mechanistic, CM-Causal Mechanistic. Cohorts OCLUE-OCLUE Traditional-Traditional Chi-Square comparison of these proportions Start of OC2 – S18 End of OC2 – S18 End of OC2 – S19 53 % DC 47% Non-DC 45% DC 55% Non-DC 2 (1) = 1.493, p = 0.222 51% DC 49% Non-DC 49% DC 51% Non-DC 2 (1) = 0.114, p = 0.735 38% DC 64% Non-DC 47% DC 53% Non-DC 2 (1) = 1.361, p = 0.243 Table 7.5. Chi-Square test comparing the proportion of Descriptive Causal responses to other characterizations for the Reduced Scaffolding Prompt. All other characterizations include No Response, Non-Normative, Descriptive General, Descriptive Mechanistic and Causal Mechanistic. Finding 1b: OCLUE students were more successful at drawing correct static mechanistic arrows. In a preliminary analysis, we began by coding the mechanistic arrow drawings as correct or incorrect as shown in Figure 7.3A. For this part of the analysis, we only evaluated the final, static drawing. Drawings where one arrow started at the Oxygen atom and ended at the Carbon atom and then another arrow began at the Carbon-Bromine bond and ended at the Bromine atom were 200 characterized as correct (Figure 7.3A). At the start of OC2, 80% of OCLUE students drew correct mechanistic arrows for this simple intramolecular SN2 process compared to 52% of Traditional students. Using a Chi-Square analysis, we found these proportions to be significantly different with a small effect size (Table 7.6). Both cohorts improved by the end of OC2, but OCLUE students still outperformed Traditional students (90% correct compared to 70% correct, respectively) with a small effect size (Table 7.6). In the following year, OCLUE students still exceled with 88% of students drawing correct arrows, but Traditional students also exceled with 80% of students drawing correct arrows and this difference was not significant. Pulling together students’ reasoning characterizations and their drawings gave a fuller picture of students’ knowledge in use at the start of OC2. OCLUE students were still successful at drawing mechanistic arrows despite their decreased engagement in explicit mechanistic reasoning. Adding together the proportions of descriptive mechanistic and causal mechanistic characterizations, only 21% of OCLUE students invoked explicit mechanistic reasoning, however, 80% of OCLUE students drew correct arrows. The same was not true for Traditional students as only 52% of students constructed correct mechanistic arrows at the start of OC2 with 18% of explanations including explicit discussion of mechanism. This evidence suggests that OCLUE students were better able to use their mechanistic arrows in a canonical way for this reaction that they had never seen before at the start of OC2, and they maintained that ability through the semester. Traditional students struggled to use their arrows in a canonical way for this unfamiliar reaction even after one semester of organic chemistry as these data were collected at the start of OC2. 201 Start of OC2 – S18 End of OC2 – S18 End of OC2 – S19 OCLUE-OCLUE-S18 Traditional-Traditional-S18 Chi-Square comparison of these proportions 80% Correct 20% Incorrect 52% Correct 48% Incorrect 2 (1) = 18.534, p < 0.001, 90% Correct 10% Incorrect 70% Correct 30% Incorrect 2 (1) = 13.839, p < 0.001, 88% Correct 12% Incorrect 80% Correct 20% correct 2 (1) = 1.471, p = 0.225 Cramer’s V = 0.283, small Cramer’s V = 0.245, small effect size effect size Table 7.6. Reduced Scaffolding static arrows results. Finding 1c: OCLUE students were more successful at both discussing and drawing an SN2 process for the Reduced Scaffolding prompt. The prior finding identified students whose arrows were canonically correct from a static perspective. A typical organic chemistry assessment item might assess students in a similar way omitting any reasoning component. beSocratic9 allows the researcher to observe student mechanistic arrow use using a dynamic approach by recording student drawings stroke by stroke. As in our prior study reported in Chapter V of this dissertation, insights can be gained by investigating student mechanistic arrow use in conjunction with their reasoning. As with the reaction of CH3Br, the majority of students are consistent in their reasoning and their drawing. Table 7.7 reports the proportion of students who described an SN2 reaction and also drew SN2-like arrows. However, students also described an SN1 reaction and drew SN1-like arrows. We are most interested in the students who correctly described an SN2 for this reaction and drew arrows reflective of that reasoning. At the start of OC2, the proportion of students who explained and drew an SN2 mechanism is equal to those who drew and explained an SN1 (37% and 36% respectively). This is compared to 14% of Traditional students who discussed and drew an SN2 and 27% who discussed and drew an SN1. This means that, at the start of OC2, Traditional students described and drew an SN1-like process more often than an SN2. However, students in both groups did improve overtime. By the end of OC2, 71% of OCLUE students discussed and drew an SN2 process but still only 44% of Traditional students did the same. 202 These trends were replicated in the following year with 81% of OCLUE students and 55% of Traditional students demonstrating a canonically correct understanding of this reaction. Table 7.7. Consistency of explanations and arrows for all three reaction prompts. RQ 2: How does expanded scaffolding affect student engagement in causal mechanistic reasoning and mechanistic arrow drawings for students enrolled in transformed and traditional organic chemistry courses? Finding 2a: The majority of students from both course types engaged in causal mechanistic reasoning when prompted by Expanded Scaffolding. The expanded scaffolding prompt was designed to provide students with as many opportunities to engage in causal mechanistic reasoning as possible. Consequently, the majority of students in both cohorts constructed causal mechanistic responses with the expanded scaffolding. At the start of OC2, 55% of OCLUE students and 65% of Traditional students constructed causal mechanistic responses (Figure 7.5G). By the end of OC2, 65% of OCLUE students constructed causal mechanistic responses and 56% of Traditional students (Figure 7.5H). Similar trends were observed in the following year in S19 (Figure 7.5I). Chi-square tests between the proportion of causal mechanistic responses and non-causal mechanistic responses showed no statistically significant differences between any of the cohorts at any time points (Table 7.8). 203 Time Point Cohort 1 Cohort 2 Start OC2 – S18 OCLUE-OCLUE-S18 Traditional-Traditional-S18 End OC2 – S18 OCLUE-OCLUE-S18 Traditional-Traditional-S18 End OC2 – S19 OCLUE-OCLUE-S19 Traditional-Traditional-S19 End OC2 End OC2 Cohort OCLUE-OCLUE-S18 OCLUE-OCLUE-S19 Traditional-Traditional-S18 Traditional-Traditional-S19 Time Point 1 Time Point 2 OCLUE-OCLUE-S18 Start OC2 – S18 Traditional- Traditional-S18 Start OC2 – S18 End OC2 – S18 End OC2 – S18 2 2.159 1.844 2.079 0.880 2.200 McNemar Test 2 2.531 2.222 p-value 0.142 0.175 0.149 0.348 0.138 0.112 0.136 Table 7.8. Chi-Square comparisons of causal mechanistic responses verses non-causal mechanistic response for the Expanded Scaffolding prompt. Finding 2b: OCLUE students were more successful at constructing a correct SN1 mechanism. The expanded scaffolding prompt allowed students to draw mechanistic arrows, any intermediate structures, and any final products. Students were prompted to build their drawing in steps allowing the researcher to elicit as much information as possible about students’ ability to represent this SN1 process. At the start of OC2, 53% of OCLUE students and 42% of Traditional students constructed canonical SN1 mechanism drawings complete with all arrows, a carbocation intermediate, and products like those represented as pathways A and B in Figure 7.4. At the end of OC2, 68% of OCLUE students constructed complete SN1 mechanisms compared to only 34% of Traditional students. This trend did not replicate at the end of OC2 in S19. Our analysis found that 42% of OCLUE students constructed a completely correct mechanism and only 13% of Traditional students. Despite explicit prompting, some students did not draw a final product but drew complete and correct mechanistic arrows and explicitly included a carbocation (Figure 7.4C). These students were on the right track with their thinking but left their drawing incomplete. By casting this wider net and including students who drew a complete mechanism or a nearly complete mechanism (Figure 7.4 A-C), 204 we found that the majority of OCLUE and Traditional students were on the right track at the start of OC2 (68% of OCLUE students and 68% of Traditional students). By the end of OC2, 83% of OCLUE students and only 48% of Traditional students constructed a complete mechanism or a nearly complete mechanism. These results were not reproduced in the following year specifically for Traditional students. We found that only 22% of Traditional students constructing reasonable mechanisms compared to 63% of OCLUE students. It is not clear why this trend did not replicate in S19. Additional research is needed to understand students’ thinking about SN1 reactions. As discussed in Finding 2a, trends in students’ explanations in S18 were replicated in S19. However, students’ mechanism drawings did not replicate, particularly for Traditional students. This begs the questions: what is the nature of students’ drawing in S19 and why were they so different than the previous year? We found that 19% of OCLUE students did not include a carbocation intermediate in their S19 drawings (Figure 7.4E). However, 54% of Traditional students in S19 did not include a carbocation intermediate in their mechanism and drew and SN2-like mechanism for the formation of t- butyl iodide. Finding 2c: The majority of students are consistent in their reasoning and mechanistic drawings when engaging with the Expanded Scaffolding prompt. At the start of OC2, three-quarters of students in both cohorts are consistent in the process they discussed and the mechanism they drew (Table 7.7). In both instances, over 60% of the students correctly discussed an SN1 process and drew and SN1 mechanism. By the end of OC2, OCLUE students have improved even more to 80% of students discussing and drawing an SN1. However, Traditional students seem to regress in their knowledge of SN1 reactions as only 49% of students discussed and drew an SN1. Rather, an additional 21% of Traditional students discussed and drew an SN2 reaction at the end of OC2. These results were not replicated in the following year. In S19, fewer students recognized this as an SN1 reaction. Only 36% of OCLUE students and 20% of Traditional students drew 205 and reasoned about an SN1. At this same time, 27% of OCLUE students and 59% of Traditional students discussed and drew an SN2 (Table 7.7). RQ 3: How consistent are students in engaging in causal mechanistic reasoning within each prompt from the start to the end of OC2? Finding 3: Less than half of students constructed a causal mechanistic response at the start and the end of OC2. In assembling cohorts for the 2017-2018 academic year, only students who participated in both the data administrations at the start and end of the semester were included. Doing so provided meaningful insights into how individual students’ reasoning changed overtime. Research questions 1 and 2 address reported reasoning trends at the level of an entire cohort. The aim of research question 3 is to explore the consistency of reasoning at the level of each individual student over time. For this analysis, we identified students who constructed causal mechanistic explanations at both the start and end of OC2 for one prompt. For example, we identified students who constructed a causal mechanistic response for the CH3Br prompt at both time points. Then we repeated this analysis for the reduced scaffolding prompt and then again for the expanded scaffolding prompt as visualized in Figure 7.6. Figure 7.6. Visualization of how data were analyzed to address RQ 3. This analysis revealed that less than half of students consistently construct a causal mechanistic response for a given prompt (Table 7.9). For the CH3Br prompt, 41% of students constructed a causal 206 mechanistic response at both time points compared to 26% of Traditional students. For the Expanded Scaffolding prompt, just over 40% of OCLUE and Traditional students constructed causal mechanistic responses at the start and end of the semester. Since very few students constructed causal mechanistic responses for the Reduced Scaffolding prompt, it follows that very few students constructed a causal mechanistic response at both the start and the end of OC2. Thus, for the Reduced Scaffolding prompt, we also identified those students who consistently constructed descriptive causal responses since that was the most common characterization and found that about a third of both OCLUE and Traditional students consistently constructed causal-only responses (Table 7.9). Prompt Cohort Code at Start OC2 Code at End OC2 Frequency CH3Br Prompt OCLUE-OCLUE-S18 Traditional-Traditional-S18 OCLUE-OCLUE-S18 Traditional-Traditional-S18 OCLUE-OCLUE-S18 Traditional-Traditional-S18 OCLUE-OCLUE-S18 Traditional-Traditional-S18 Reduced Scaffolding Prompt Expanded Scaffolding Prompt CM CM CM CM DC DC CM CM CM CM CM CM DC DC CM CM 41% 26% 9% 4% 33% 30% 45% 43% Table 7.9. Percent of students who constructed consistent reasoning at the start and end of OC2 for each prompt type. RQ 4: How consistent are students in engaging in causal mechanistic reasoning across multiple prompts? Finding 4: Few students constructed causal mechanistic responses across all three reactions. To address this research question, we investigated how consistent students were at engaging in causal mechanistic reasoning for each of the different prompt types at a given time point as visualized by Figure 7.7. All three of these prompts were administered as part of a single activity, and therefore evaluating student reasoning across the three activities at given time point provides evidence of the influence of each prompt on student reasoning. As discussed in the prior findings, very few students constructed causal mechanistic responses to the Reduced Scaffolding prompt. Due to this, few students constructed a causal mechanistic response for all three reactions (Table 7.10). At the start of OC2, 12% of OCLUE students constructed causal mechanistic responses for all three prompts and 16% at the end 207 of OC2. These numbers were even fewer for Traditional students (9% and 8%, respectively). Subsequently, it was very rare that students constructed a causal mechanistic response for all three reactions at both time points. Next, we considered only those students who engaged in causal mechanistic reasoning for the CH3Br prompt and the Extended Scaffolding prompt. This analysis showed that approximately a third of students constructed causal mechanistic responses to both the CH3Br prompt and the Expanded Scaffolding prompt at the start of OC2. This was true for OCLUE students (38%) and Traditional students (33%). At the end of OC2, nearly half of OCLUE students and 41% of Traditional students engaged in causal mechanistic reasoning for both the CH3Br and the Extended Scaffolding prompt. This finding suggests that student causal mechanistic reasoning is sensitive to the nature of the scaffolding of the prompt. Figure 7.7. Visualization of how data were analyzed to address RQ 4. 208 Time Point Cohort % of Student Responses Causal Mechanistic for all three reactions (CH3Br Prompt, Reduced Scaffolding, and Extended Scaffolding) Start OC2 End OC2 Causal Mechanistic for CH3Br Prompt and Start OC2 Extended Scaffolding only End OC2 OCLUE-OCLUE Traditional-Traditional OCLUE-OCLUE Traditional-Traditional OCLUE-OCLUE Traditional-Traditional OCLUE-OCLUE Traditional-Traditional Causal Mechanistic for all three reactions (CH3Br Prompt, Reduced Scaffolding, and Across both time OCLUE-OCLUE points Traditional-Traditional Extended Scaffolding) 12% 9% 16% 8% 38% 33% 47% 31% 5% 2% Table 7.10. Percent of students who constructed consistent reasoning for multiple prompts at a given time point. Summary These preliminary findings suggest that student engagement in causal mechanistic reasoning is sensitive to the level of scaffolding included in the prompt. In the case of the reduced scaffolding, we observed that very few students invoked explicit discussion of electron movement (i.e. mechanism). For the simple, one step SN2 reaction, moderate scaffolding elicited different levels of causal mechanistic reasoning for OCLUE and Traditional students with OCLUE students constructing more causal mechanistic responses than Traditional students as discussed in Chapter VI. Evidence also suggests the majority of OCLUE and Traditional students engage in casual mechanistic reasoning with expanded scaffolding. Evidence suggests that only the CH3Br prompt differentiated between OCLUE and Traditional students’ causal mechanistic reasoning. There were no significant differences between these two cohorts’ reasoning at any time point for the reduced scaffolding prompt and the expanded scaffolding prompt. However, there does appear to be a difference between OCLUE and Traditional students’ abilities to correctly identify the reaction as an SN2 or an SN1 and draw canonical mechanistic arrows. OCLUE students out-performed Traditional students in drawing correct static mechanistic arrows for the reduced scaffolding prompt. OCLUE students were also more likely to discuss and draw an SN2 reaction 209 for the CH3Br prompt and the reduced scaffolding prompt at the start and end of OC2 compared to Traditional students and this trend was replicated the following year. For the SN1 reaction in the expanded scaffolding prompt, OCLUE and Traditional students were equally likely to correctly explain and draw this process at the start of OC2 but OCLUE students exceeded traditional students at the end of OC2. These trends were not replicated in the following year, and additional research is needed to investigate student difficulties with SN1 reactions. The expanded scaffolding prompt perhaps activates the appropriate intellectual resources for students enrolled in both OCLUE and Traditional courses. This study was designed to investigate students’ reasoning at the cohort level but also to investigate trends in reasoning for individual students. This design allowed us to make claims about how many students engage in causal mechanistic reasoning consistently for different prompt designs and at different time points. In doing so, we found that less than half of students in both organic courses construct causal mechanistic responses to the expanded scaffolding prompt at the start and end of OC2. These numbers fall to about a third for the reduced scaffolding prompt but even then, students were only consistent in their ability to construct descriptive causal (meaning causal-only) responses. Finally, these findings suggest that with appropriate support and scaffolding, OCLUE and Traditional students can be equally capable of engaging in causal mechanistic reasoning possibly because of explicit activation of intellectual resources required to engage in causal mechanistic reasoning. However, when the scaffolding is removed, both groups of students omit the explicit mechanistic elements from their explanation. What does this mean for students’ course experiences? More research is needed to understand the elements of curriculum that might be contributing to student engagement in causal mechanistic reasoning in a range of contexts. The use of various prompts elicited different types of reasoning from students although all Implications for Instruction prompt designs discussed in this study elicited some sort of reasoning in conjunction with eliciting a 210 drawn reaction mechanism. The findings presented in this study were only possible because both pieces of evidence were triangulated to investigate student understanding. Indeed, even eliciting evidence of student understanding of SN2 verses SN1 processes for the appropriate reactions required triangulating student explanations with step-by-step replays of student drawings for the reduced scaffolding and CH3Br prompts and step-by-step drawings for the expanded scaffolding. If an instructor places value on students’ abilities to differentiate between SN2 and SN1 processes, this study suggests that a simple multiple-choice question or even students’ static mechanistic arrow drawings will fail to capture the complexity of student thinking and indeed multiple choice items have been found to overestimate student abilities.10 Implications for Research The reactions invoked in this study were very simple examples of SN2 and SN1 reactions. Further research is required to elicit causal mechanistic responses about more complex reactions. However, we found that increasing the complexity from an SN2 with CH3Br to an SN1 reaction with (CH)3CBr required increased prompting to elicit deep and complete explanations. Investigating a different reaction context would require additional pilot testing to elicit the desired reasoning as the prompt structures here were only validated for our populations here for these specific reactions. Limitations As mentioned in the Implications for Research section, these reactions were very simple, and we have not yet attempted to elicit causal mechanistic responses for more complex organic chemistry reactions. It was necessary to understand how students engage with simple phenomenon before investigating more difficult ones. Second, these responses were also elicited in low-stakes formative assessments in an environment where we had little control over the conditions in which students were completing the activities. Students were encouraged not to use their notes or seek help from a TA or 211 another peer, however, we had no way of ensuring this. We attempted to moderate this limitation by measuring students overtime and by replicating our studying in the following year. Third, these prompt structures were designed to probe student reasonings with a variety of prompt features as part of this preliminary investigation. However, there are many factors changing between each prompt structure making it difficult to identify one specific feature of the prompt that might be responsible for changes in reasoning. Future work is needed where only only one of the following factors is varied at a time: prompt scaffolding level (reduced, regular, and expanded) and the reaction type (SN2, SN1, or a cyclization prompt). Finally, our coding scheme did not capture incorrect ideas and statements students included in their responses. Our characterization schemes accounted for the productive ideas used to engage in causal mechanistic responses and we additionally accounted for the process they used. We did not characterize incorrect vocabulary use nor did we accept use of vocabulary such as electrophile and nucleophile as the only evidence of engagement in causal mechanistic reasoning. A different definition of causal mechanistic reasoning might yield a different characterization scheme reflective of different research goals and different results might be obtained. 212 REFERENCES 213 REFERENCES (1) Wood, D.; Bruner, J. S.; Ross, G. The Role of Tutoring in Problem Solving. J. Child Psychol. Psychiatry 1976, 17 (2), 89–100. (2) Cazden, C. Peekaboo as an Instructional Model: Discourse Development at Home and at School. In The Sociogenesis of Language and Human Conduct; Bain, B., Ed.; Springer: Boston, MA, 1983. (3) Bruner, J. S. The Inspiration of Vygotsky. In Actual minds, possible worlds; Harvard University Press: Cambridge, MA, 1986; pp 70–78. (4) Vygotsky, L. S. Mind in Society: The Development of Higher Psychological Processes; Cole, M., John- Steiner, V., Scribner, S., Souberman, E., Eds.; Harvard University Press: Cambridge, MA, 1978. (5) van de Pol, J.; Volman, M.; Beishuizen, J. Scaffolding in Teacher-Student Interaction: A Decade of Research. Educ Psychol Rev 2010, 22, 271–296. (6) McNeill, K. L.; Lizotte, D. J.; Krajcik, J. S. Supporting Students’ of Construction of Scientific Explanations by Fading Scaffolds in Instructional Materials. J. Learn. Sci. 2006, 15 (2), 153–191. (7) Noyes, K.; Cooper, M. M. Investigating Student Understanding of London Dispersion Forces: A Longitudinal Study. J. Chem. Educ. 2019, 96 (9), 1821–1832. (8) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid–Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703–1712. https://doi.org/10.1021/acs.jchemed.6b00417. (9) Bryfczynski, S. BeSocratic: An Intelligent Tutoring System for the Recognition, Evaluation, and Analysis of Free-Form Student Input; Clemson University, Clemson, SC, 2012; Vol. Ph.D. Dissertation. (10) Lee, H.-S.; Liu, O. L.; Linn, M. C. Validating Measurement of Knowledge Integration in Science Using Multiple-Choice and Explanation Items. Appl. Meas. Educ. 2011, 24, 115–136. 214 CHAPTER VIII: CONCLUSIONS, IMPLICATIONS, AND FUTURE WORK Conclusions This dissertation was developed as a series of four studies spanning three academic years. Data were in the form of students’ typed explanations and draw mechanistic arrows both gathered together as part of low-stakes formative assessment activities using the online homework and research tool called beSocratic.1 These data were collected from undergraduate students while they were enrolled in second-year organic chemistry. Students were either enrolled in a transformed organic chemistry course called Organic Chemistry, Life, the Universe and Everything (OCLUE) or a traditional organic chemistry course which served as a control for comparisons to the transformed students. Student responses were characterized as causal mechanistic, mechanistic-only, causal-only, or descriptive-only. Student explanations were also compared to student drawings to characterize consistency across explanation and drawing. 1. Students’ general chemistry experience positively impacted their reasoning abilities in organic chemistry. Chapter IV describes an investigation of students who were enrolled in a traditional organic chemistry course at the time of study. Students were sampled at the start and end of OC2 and were asked to reason about two simple acid-base reactions – the reaction of HCl with H2O and then the reaction of NH3 with BF3. All participants were enrolled in the same organic chemistry course but differed in their general chemistry experience. There were three possible general chemistry course experiences: those enrolled in the transformed general chemistry course (CLUE), those enrolled in a “selective” general chemistry course such as honors or majors courses, and finally those who had not enrolled in the second semester of general chemistry at all as it is not a prerequisite for entering first- semester organic chemistry. 215 These findings show that students enrolled in “selective” general chemistry course performed similarly to those students who did not even take second-semester general chemistry and thus, these cohorts were combined for simplicity. Students in this combined cohort were less likely to construct causal mechanistic explanations at both the start of OC2 and the end for the reaction of HCl and H2O compared to students who had been enrolled in CLUE for general chemistry. Students in all cohorts were less likely to engage in causal mechanistic reasoning for the reaction NH3 and BF3. Rather, most students were characterized as Descriptive Mechanistic, or mechanistic-only for this reaction. 2. Students’ organic chemistry experience positively impacted their engagement in causal mechanistic reasoning in the long term. Chapters V and VI investigated students who were enrolled in either the transformed organic chemistry course, OCLUE, or a traditional organic chemistry course following a traditional textbook. Cohorts were formed based on student enrollment across the two-semester course. Some students were enrolled in OCLUE-OCLUE for both semesters, and we found that these students engaged in casual mechanistic reasoning most often, specifically for the reaction of CH3Br with OH-. Some students had to switch between course types due to scheduling constraints. These students, who switched between OCLUE and Traditional or vice versa, did not engage in causal mechanistic reasoning as often as OCLUE- OCLUE students but did so more often than Traditional-Traditional students. In other words, students who had OCLUE consistently for both semesters engaged in causal mechanistic reasoning the most, students who had some OCLUE course experience in either their first or second semester reasoned “in the middle”, and students who had traditional OC for both semesters were least likely to construct causal mechanistic responses. 3. The chemical reaction and structure of the prompt both impact the nature of student reasoning. The analysis of responses to NH3 with BF3 in Chapter IV showed that students were less likely to construct causal mechanistic responses compared to responses to the reaction of HCl and H2O. Rather, 216 many students constructed Descriptive Mechanistic responses for this Lewis acid-base reaction in which no proton is transferred. Students were less likely to reason about electrostatic interactions in this context. The reverse trend was observed with the reduced scaffolding prompt discussed in Chapter VII. For this prompt structure, OCLUE students and Traditional students alike did not engage in explicit mechanistic reasoning and instead, most students constructed causal-only responses. However, when the scaffolding is expanded to elicit “how” and “why” in a step-by-step prompt, both OCLUE and Traditional students engaged in causal mechanistic reasoning equally. The study presented in Chapter VII was intended as a preliminary study to pilot prompts with different scaffolding and different reactions. Future work is needed to empirically analyze influence of each of these variables on student reasoning. Implications for organic chemistry instruction and assessment These studies characterized engagement in causal mechanistic reasoning for students enrolled in a transformed and traditional organic chemistry courses and suggest that consistent enrollment in the transformed course improved student engagement in causal mechanistic reasoning. However, none of the studies presented here were specifically designed to address the question of why do more OCLUE students engage in causal mechanistic reasoning? Investigating this question would involve a completely different study design involving various observational protocols to quantify various elements of the OCLUE course in comparison to a traditional course. However, the design and implementation of the OCLUE curriculum was informed by evidence about the best practices for teaching and learning science. To that end, the following section discusses key features of the OCLUE curriculum that were previously introduced in Chapters II and III in specific connection to student engagement in causal mechanistic reasoning in organic chemistry. These key features are: 1. Careful consideration about what content is presented so students have more productive, connected resources at their fingertips. 217 2. Construction of explanations leading to deeper understanding. 3. Incentivize “understanding why.” 1. Careful consideration about what content is presented so students have more productive resources at their fingertips. The organic chemistry curriculum is historically chock-full of esoteric reagents and reactions to prepare the next generation of practicing chemists. This ethos affords few opportunities for students to reflect on how their knowledge of structure/property relationships or electrostatic interactions might help them understand the irregular reactivity of these highly specialized reagents. When designing OCLUE, Cooper et al. considered the value of teaching “depth over breadth”2 particularly for pre- professional student who do not need the “breadth” of organic chemistry for their future careers.3 It is possible that a student could develop an expert-like knowledge structure grounded in core-ideas that is connected in ways appropriate for a student enrolled in their second year of college chemistry and they might still be unsuccessful in using their knowledge to understand and make predictions about abstruse named reaction mechanisms (e.g. the Hell-Volhard-Zelinsky or Knoevenagel reactions). Indeed, experts in organic chemistry almost certainly memorize these reactions – although they could surely produce a mechanism if prompted. The question is, why should beginners be asked to memorize such reactions before they have a command of the core ideas and practices of organic chemistry. Cooper et al. made the argument that pre-professional students should not have to waste time and effort trying to incorporate reactions of this nature into their knowledge frameworks. Although OCLUE still includes most of the reactions found in a traditional course (because of external constraints) there is little emphasis in the curriculum and in assessments. In OCLUE, time is better spent focusing on leveraging knowledge of core ideas to understand processes that are most relevant in the chemistry of biological systems. For example, proton transfers and bimolecular nucleophilic substitutions using nitrogen, oxygen or sulfur as nucleophiles adding to carbonyl sites have been identified as the most 218 prevalent mechanisms in biological systems.4 OCLUE students are offered numerous opportunities throughout the course to consider the causal mechanism behind a nucleophilic substitution and are encouraged to draw on their knowledge of structure/property relationships and electrostatic interactions. I suggest that OCLUE students’ intellectual resources surrounding the causal mechanism of nucleophilic substitutions are well connected and grounded in core ideas and are thus more useful in given contexts. When activated by the prompt, these intellectual resources consistently function for the student to construct a causal mechanism at numerous timepoints throughout the semester. It appears that OCLUE’s focus on “depth over breadth” led to more causal mechanistic responses because students, theoretically, have fewer superfluous and disconnected intellectual resources in their knowledge framework compared to traditional students. Additionally, their knowledge is well- connected and beginning to take on the “highly contextualized” characteristic of expert knowledge for nucleophilic substitutions. 2. Construction of explanations leading to deeper understanding. The purpose of science is to explain and predict phenomena.2 Furthermore, the purpose of developing knowledge that connected in meaningful ways is to be able use that knowledge to understand and make predictions in new situations. However, the reverse condition is also true. The act of attempting to construct explanations helps students make meaningful connections in their knowledge structures.5 Simply put, students can use their connected knowledge to explain and by explaining, students can make more connection in their knowledge. This cycle goes around and around. In OCLUE, students are explicitly taught how to construct scientific explanations that include causal and mechanistic components. Specific attention is given to distinguish scientific explanations that connect a claim to scientific evidence from descriptions of phenomena or explanations that are heuristic-based or rule-based (e.g. reactions happen because carbon wants four bonds).6 219 3. Incentivize “understanding why.” There is strong evidence supporting student engagement in explanation as a pedagogical approach to improve learning and understanding.5 However, even this might not convince instructors of its utility. Still, an even more difficult challenge is to obtain buy-in from students about the importance of engaging in explanation. Students will value the knowledge and skills that they will be assessed on to determine their grade in the course. The Framework2 recognized the importance of this by recommending that students engage in scientific practices (such as explanation) while leveraging their knowledge of core ideas in both curriculum and assessment. Cooper et al. have operationalized this in OCLUE by requiring students to engage in explanation throughout the course on a regular basis. OCLUE offers opportunities for students to practice engaging in explanation on formative assessments such as low-stakes homework, in weekly recitations, and in lecture. Students receive feedback on these low- stakes assessments where in the instructor shows anonymized examples of student work highlighting productive and unproductive features of the responses. A key feature of this formative feedback is demonstrating that explanations can be unique and still productive (i.e. writing explanations is not a cookie cutter skill). In this way, students are encouraged to think deeply about the explanations they are constructing while they are constructing them rather than memorizing one single answer. These formative assessment opportunities encourage students to identify relevant information, string information together in a logical sequence in a low-stakes environment. These formative assessments are included in a percentage of students’ final grade to incentive student engagement and communicate the importance of “trying” to use one’s knowledge even if the answer is not completely correct the first time. The goal of these activities is for students to construct explanations to help them learn. There does, of course, come a time when students are expected to know how to do something and do it well – enter summative assessments. However, the expectation on OCLUE summative assessments is that students are using their knowledge to demonstrate their ability to explain a 220 phenomenon instead of just regurgitating memorized facts. Even in a high-stake environment, students are expected to demonstrate that they know how to explain. This expectation is key to OCLUE students’ observed success in engaging in causal mechanistic reasoning. It appears that the prompt not only activates relevant pieces of knowledge but also the relevant habit of constructing a complete causal mechanistic response. However, there is still more work to be done. The reactions studied here were very simple, Future Work textbook examples of nucleophilic substitutions. Additional work should be done to characterize student reasoning about more complex reactions. In this vein, more research is needed to establish the appropriate level of scaffolding required to elicit causal mechanistic reasoning from students as more complex phenomena are introduced. Many students enrolled in organic chemistry are planning to pursue professional careers in the medical sciences and as such, they will progress onto biochemistry where they will be presented with numerous complex biochemical processes – many of which are underpinned by electrostatic interactions, structure-property relationships, energy, and equilibrium. Additional work is needed to explore causal mechanistic reasoning in these contexts both in organic chemistry and beyond organic chemistry in delayed post-tests. Longitudinal studies such as these are very uncommon in the literature but are necessary to investigate student learning throughout a degree sequence.7 221 REFERENCES 222 REFERENCES (1) Bryfczynski, S. BeSocratic: An Intelligent Tutoring System for the Recognition, Evaluation, and Analysis of Free-Form Student Input; Clemson University, Clemson, SC, 2012; Vol. Ph.D. Dissertation. (2) A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas; National Academies Press: Washington, D.C., 2012. (3) Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, 96, 1858–1872. (4) Holliday, G. L.; Almonacid, D. E.; Mitchell, J. B. O.; Thornton, J. M. The Chemistry of Protein Catalysis. J. Mol. Biol. 2007, 372, 1261–1277. (5) Pashler, H.; Bain, P. M.; Bottge, B. A.; Graesser, A.; Koedinger, K.; McDaniel, M.; Metcalfe, J. Organizing Instruction and Study to Improve Student Learning; National Center for Education Research, Insitute of Education Sciences, U.S. Department of Education: Washington, D.C., 2007. (6) Taber, K. S. College Students’ Conceptions of Chemical Stability: The Widespread Adoption of a Heuristic Rule out of Context and beyond Its Range of Application. Int. J. Sci. Educ. 2009, 31 (10), 1333–1358. (7) Review and Synthesis of Research in Chemical Education from 2000-2010: A White Paper for the National Academies National Research Council Board of Science Education Committee on Status, Contributions, and Future Directions of Discipline Based Education Research. January 2011. 223