DEVELOPMENT OF 21ST-CENTURY SKILLS IN PROJECT-BASED GENERAL CHEMISTRY LABORATORY COURSES: A MIXED-METHODS STUDY OF STUDENT AND GRADUATE TEACHING ASSISTANT PERCEPTIONS By Britney Morgan Eggly A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Chemistry – Doctor of Philosophy 2024 i ABSTRACT Employers report that college graduates lack skills necessary to be successful in today’s workforce. Referred to as employer-desired competencies (EDCs) in this dissertation, and soft, transferable, or 21st-century skills throughout the literature, these skills commonly include communication, teamwork, problem-solving, and critical thinking. Lack of proficiency in these skills are reported as a primary cause behind the skills gap - the divide between what students have learned or acquired during their college education and the skills employers desire of new hires. This gap has been noted for its negative effects on employers’ ability to find suitable applicants and graduates’ ability to achieve employment. As a result, educational institutions have been tasked with implementing more career focused opportunities into the curriculum to better prepare students for 21st-century careers. Of particular interest in science, technology, engineering, and mathematics (STEM) education are introductory courses that are generally large enrollment and have gained notoriety due to their influence on retention of STEM majors. Within such courses, project-based learning is a student-centered approach that has been reported to create a learning environment that encourages growth of EDCs, which can kickstart career preparation. The studies presented herein sought to investigate the phenomenon of EDC development in introductory project-based general chemistry laboratory courses from both a student and instructor perspective using a mixed-methods approach. Primarily focusing on qualitative methods of data collection and analysis, interviews were conducted with n = 53 undergraduate students enrolled in the project-based general chemistry laboratory courses and n = 12 graduate student teaching assistants (GTAs) who instructed these courses. Using a transcendental phenomenological approach in the qualitative analysis, we first explored undergraduate student perceptions of what competencies were needed for their planned careers (student-perceived competencies, also referred to as SPCs), how SPCs overlapped with EDCs, development of prevalent competencies, and how course components of the general chemistry laboratory courses supported development. Surveys, administered to undergraduate students as alternatives to participating in interviews, were used to supplement interview findings. Further, GTAs were asked what career-relevant competencies they believed were gained as instructors for these courses and how their teaching experiences contributed to this development. ii Using thematic analysis, six prevalent SPCs were identified by undergraduate students as valuable career competencies – communication, teamwork, problem-solving & critical thinking, prioritization & time management, work ethic, and technical skills. Comparison of these skills to widely recognized EDCs shows that students can identify valuable workplace competencies. Further investigation showed that course components supported development of the first four SPCs. Three main themes were found relating course components to skill development: 1) collaboration with teams, 2) open inquiry learning, and 3) project management. As this study progressed, the COVID-19 pandemic occurred, forcing the project-based general chemistry laboratory courses to operate remotely. This shift to remote learning opened a new area of inquiry not previously planned for – how online learning influenced undergraduate student perceptions of EDC development. Emergent in interview responses was how online learning negatively affected student ability to develop technical skills (e.g., laboratory skills and techniques). Further, being able to adapt to the online environment seemed to affect whether a student believed development of the other EDCs – communication, teamwork, work ethic, problem-solving & critical thinking, and prioritization & time management – were supported, hindered, or unaffected online. Emergent in GTA interviews was that interpersonal skills were the most prevalent skills GTAs believed were developed as instructors for these courses – specifically communication (e.g., verbal) and leadership (e.g., mentoring) skills. Themes surrounding how instructing the general chemistry laboratory courses supported development of these skills included: 1) learning to converse with diverse sets of students, and 2) having to guide students through projects. These findings support project-based learning in the general chemistry laboratory as a pedagogical approach to encourage development of EDCs from the student and instructor perspective. Further, these studies emphasize how technical skills were an important aspect that was missed by students in a remote learning environment. Use of student career goals and perception of what can be gained from a course can be used to drive additional learning goals and curriculum design. By leveraging skills that can contribute to career preparation, introductory courses can continue to be framed as beneficial experiences in students’ undergraduate education. iii It is with sincere gratitude and admiration that I dedicate this dissertation to my mother and father. It is through their love and support that I was able to accomplish this work and I will forever be indebted to them for sitting through my many highs and lows. Your encouragement not only shaped who I am but got me to where I needed to be. iv ACKNOWLEDGEMENTS Countless people have had a positive impact on where I am today but as I only have limited space for acknowledgements, I would like to recognize those who played the biggest roles in this achievement. I would first like to thank Leart Sejdarasi for providing support and giving me ample inspiration to finish my journey. The many laughs that we shared always lifted my spirit and kept me going. Thanks also goes to Gabriella Murashova, Brianna Davis, and Emily Thomas, who are three beautiful souls I had met on my graduate journey that were a guiding light whenever I needed them. To my brother, sister-in-law, and nephews - Cameron, Emily, Aiden, and Aaron Eggly - thank you for always putting a smile on my face and providing me with good times and much needed breaks that I always looked forward to. Grandma Santini, this also wouldn’t be possible without you supporting me in any and every way you could. I would also like to acknowledge Barry DeCicco for showing me how statistics can be fun and rewarding, while also becoming a friend along the way. It was an absolute honor to work with Dr. Brittany Busby. She always lent time and a helping hand in any way she could, providing a much-needed sounding board during times of uncertainty. Her guidance was and continues to be an invaluable resource. Thank you to Dr. Melanie Cooper for providing support and always asking the best questions that got me thinking and made my project the best it can be. Dr. Cooper and the Cooper Group always made me feel so welcome as an honorary group member and to them I am so thankful. The biggest thanks to my advisor, Dr. Lynmarie Posey. If it were not for her welcoming me back to the graduate program with open arms and providing the utmost support as I went through this journey, I would not have even had the chance to get where I am today. It was through her support that I grew immensely as both a researcher and a person. Priya Patterson- Lee, Jordan Brown, and Ladymar Rodrigues Torres were the best undergraduate research assistants I could have asked for and were a great help to advancing my research. Thank you to Dr. Heedok Hong and Dr. John McCracken for being a part of my guidance committee and being an integral part in advancing me through my degree. Thank you to Dr. Gary Blanchard and Dr. Dana Spence for recognizing that needing some time off from my graduate degree was not something to be ashamed of, this act of kindness helped me immensely in my confidence to come back and feel supported within the program. v There are many, many more who had a positive impact on my time as a graduate student and to all who had either put a smile on my face or helped me along the way, I thank you! And finally, to my Dell, I thank you for fighting through your ill battery to continue to support me through this arduous process. Your passing is not in vain, I will forever be grateful. To all of those mentioned above, and the countless others who had a positive impact on my graduate experience, I will forever remember you fondly as I embark on my new journey! Thanks a million! vi TABLE OF CONTENTS CHAPTER I: PURPOSE, QUESTIONS, & AIMS ........................................................................ 1 REFERENCES ............................................................................................................................ 7 CHAPTER II: LITERATURE REVIEW ....................................................................................... 9 REFERENCES .......................................................................................................................... 30 CHAPTER III: THEORETICAL & METHODOLOGICAL FRAMEWORKS .......................... 41 REFERENCES .......................................................................................................................... 56 CHAPTER IV: 21ST-CENTURY SKILL DEVELOPMENT IN PROJECT-BASED GENERAL CHEMISTRY LABORATORY COURSES: A MIXED-METHODS APPROACH .................. 61 REFERENCES ........................................................................................................................ 113 APPENDIX ............................................................................................................................. 121 CHAPTER V: ONLINE & OUT OF THE LAB: EXPLORING THE IMPACT OF VIRTUAL LEARNING ON 21ST-CENTURY SKILL DEVELOPMENT IN PROJECT-BASED GENERAL CHEMISTRY LABORATORY COURSES .............................................................................. 175 REFERENCES ........................................................................................................................ 216 APPENDIX ............................................................................................................................. 225 CHAPTER VI: SURVEY RESULTS & FINDINGS: COMPARISON TO INTERVIEWS & EXPLORING NEW LINES OF INQUIRY ............................................................................... 234 REFERENCES ........................................................................................................................ 300 APPENDIX ............................................................................................................................. 305 CHAPTER VII: BEYOND THE GRADUATE DEGREE: 21ST-CENTURY SKILL DEVELOPMENT AS A GRADUATE TEACHING ASSISTANT FOR PROJECT-BASED GENERAL CHEMISTRY LABORATORY COURSES .......................................................... 421 REFERENCES ........................................................................................................................ 449 APPENDIX ............................................................................................................................. 455 CHAPTER VIII: CONLUSIONS, IMPLICATIONS, & FUTURE DIRECTIONS .................. 470 REFERENCES ........................................................................................................................ 480 vii CHAPTER I: PURPOSE, QUESTIONS, & AIMS Career preparation to support workforce success is a desired outcome of enrolling in and completion of a higher education degree from both the student’s (Broekemier, 2002; Fermin & Pope, 2003; Gallup Inc & Strada Education Network, 2018) and employer’s (Bridgeland et al., 2011; Finley, 2023) perspectives. While some employers are confident in the career preparation higher education can provide (Bridgeland et al., 2011; Finley, 2023), others do not believe college students are adequately prepared by their college education in obtaining the skills needed to enter and flourish in the 21st-century workforce (Carlson, 2022). Employers desire that the recent college graduates they hire come equipped with soft skills, such as communication, teamwork, problem-solving, and critical thinking skills (Finley, 2023; National Association of Colleges and Employers, 2023), in addition to the conceptual knowledge that traditional education focuses on. Although college degrees continue to be valued, many college graduates lack the career competencies needed for the 21st-century workforce leading to what has come to be known as the skills gap. This problem has been recognized in chemistry education both nationally (Kondo & Fair, 2017; Sinex & Chambers, 2013) and internationally (Burnham, 2020; Chadwick et al., 2018; Galloway, 2017; Yasin & Yueying, 2017). Locally, when employers were asked about what they looked for when hiring recent graduates in the natural sciences from this university, they focused on soft skills (Telfor, 2017). Further, many reported that while these students developed strong disciplinary knowledge, their soft skills were often underdeveloped. The research reported in this dissertation examines the role that project-based general chemistry laboratory 1 (GCL1) and 2 (GCL2) courses could play in providing early opportunities for undergraduate STEM students to develop valuable career competencies as an initial step in addressing the skills gap and aiding in career preparation. These project-based laboratory courses are rooted in a pedagogy that incorporates a highly collaborative classroom environment in which students are given projects that include planning experimental investigations to answer a question or solve a problem, engaging in experimentation and data collection, using evidence collected to answer the question or propose a solution to the problem, and preparing reports in several different formats to capture the entirety of the project (Carmel et al., 2017; Cooper, 2012). The project-based method of instruction is believed to support development of desirable 21st-century skills (Bell, 2010; Jollands et al., 2012)(Bell, 2010; Jollands et al., 2012). We set out to not only explore if undergraduate students taking these courses perceived that they were 1 developing valuable employer-desired competencies (EDCs), but also to investigate if the graduate teaching assistants (GTAs) in these courses perceived that they gained valuable EDCs. The sudden transition to remote learning that occurred as a result of the COVID-19 pandemic in March 2020, which was not anticipated when this research began, provided an opportunity to investigate the influence of online learning on perceptions of EDC development. Throughout this study, valuable career competencies will be referred to as soft skills, 21st-century skills, and EDCs interchangeably. Three main areas of inquiry motivated this work: 1) skills undergraduate students perceived as valuable for their intended careers and development of these skills within the GCL1/GCL2 laboratory courses, 2) the effect of online learning on students’ perceived ability to gain valuable career competencies in these courses, and 3) skills that graduate teaching assistants believed they gained from being an instructor for the project-based general chemistry laboratory courses. Chapter II will offer a review of the literature in areas relevant to this research, including the origin of skills gap and the purpose of laboratory learning in education, while Chapter III explores the mixed-methods approach employing both qualitative (e.g., interviews) and quantitative (e.g., surveys) data that was used to answer the research questions. The findings reported herein are aimed at contributing to our understanding of how engagement by undergraduate students and graduate TAs in project-based general chemistry laboratory courses may contribute to building EDCs needed for career success. While this study explores introductory laboratory courses taken within a Chemistry Department as an initial step towards addressing the skills gap, future research investigating development of skills identified as necessary for career success in other disciplines and throughout a student’s college career would greatly benefit this area of research. An overview of the four studies reported in this dissertation, with their aims and research questions, are provided below. Study 1. 21st-Century Skill Development in Project-based General Chemistry Laboratory Courses: A Phenomenological Approach This study, which is reported in Chapter IV, used primarily qualitative methods to explore student perceptions of the skills needed for their planned career goal and how course elements in the project-based general chemistry laboratory courses contributed to development of these skills. From analysis of n = 53 semi-structured interviews, conducted with GCL1 and GCL2 students, following a transcendental phenomenological methodological approach, qualitative 2 themes were generated around 1) participants’ career goals, 2) skills they perceived as necessary for success in their future career, which will be referred to as student-perceived competencies (SPCs), and 3) the ways in which participants thought the GCL1 and GCL2 learning environment supported growth of these skills. The following research questions guided this study: 1. (RQ1) What skills do students believe are needed for their career goal, and how do these student-perceived competencies (SPCs) align with employer-desired competencies (EDCs)? 2. (RQ2) How do students relate course components and activities in a project-based general chemistry laboratory learning environment to skills perceived as necessary for their planned career? Study 2. Online and Out of the Lab: Exploring the Impact of Virtual Learning on 21st- Century Skill Development in Project-based Laboratory Courses The original study to explore student perceptions of competencies required for their planned career (SPCs) and opportunities to develop these skills in project-based general chemistry laboratory courses (Study 1) was planned prior to the COVID-19 pandemic. However, data collection began in Spring 2020 and continued through Fall 2020. As a result, participants in the study experienced GCL1 and GCL2 in several different modalities. Spring 2020 participants had an in-person experience for roughly half the semester followed by emergency remote learning for the remainder of the semester, while Summer 2020 and Fall 2020 participants engaged in the courses entirely online. This provided an unanticipated opportunity to explore student perceptions of the impact of online learning on development of the 21st-century skills desired by employers in project-based GCL1 and GCL2 courses. Transcripts from the interviews conducted for Study 1 were reexamined through the lens of online learning being perceived as a support or hinderance to SPC development. Chapter V presents the results of this study, which aimed to answer the following research question: 1. (RQ1) What impact did the online learning environment have on students’ perceived ability to develop student-perceived competencies (SPCs) in project-based general chemistry laboratories? 3 Study 3. Connecting and Comparing Survey Results to Interview Findings on Student Perceptions of 21st-Century Skill Development During the semesters when interviews were conducted for Study 1 and Study 2, surveys were administered in GCL1 and GCL2 as an alternative extra-credit option for students who did not volunteer or were not selected for interviews. Additionally, surveys were administered in Spring 2021 and Fall 2021 semesters, to collect further evidence on student perceptions of EDC development in the general chemistry laboratory courses. In Chapter VI, we investigate if responses captured from the larger survey sample corroborated or contradicted themes that emerged from interviews in Study 1 and Study 2. The larger sample size also permitted exploration of possible relationships between student perceptions of EDCs needed for their planned career and EDCs developed in GCL1 or GCL2 and other variables, such as planned career, prior exposure to planned career, first-generation status, and class standing. The majority of survey questions paralleled interview questions focusing on 1) participant’s future career goals, 2) skills perceived as valuable to future career, 3) perceived development of valuable career competencies in the GCL1/GCL2 laboratory courses, and 4) the influence online learning was believed to have on skill development (RQ1 - RQ5). During one semester (Spring 2021), a reflective assignment explored how the introduction of EDCs in the course materials, provided at the beginning of each GCL1 project, contributed to building awareness of career-related skills (RQ6). This study was framed by the following research questions: 1. (RQ1) What skills do students believe are needed for their career goal, and how do these skills align with employer-desired competencies (EDCs)? 2. (RQ2) What relationships, if any, existed between student perceptions of valuable career competencies and student characteristics, such as career goal, prior experience & exposure to career goal, first-generation status, and class standing? 3. (RQ3) How do students relate course components and activities in a project-based general chemistry laboratory learning environment to skills perceived as necessary for their planned career? 4. (RQ4) What relationships, if any, existed between student perceptions of development of valuable career competencies within the general chemistry laboratory courses and student characteristics, such as career goal, prior experience & exposure to career goal, first- generation status, and class standing? 4 5. (RQ5): What impact did the online learning environment have on students’ perception of skills considered valuable for their future career and their perceived ability to develop these student-perceived competencies (SPCs) in project-based general chemistry laboratories? 6. (RQ6): How did introduction of EDCs in course materials contribute to building student awareness of EDCs? Study 4. Beyond the Graduate Degree: Development of 21st-Century Skills as a Graduate Teaching Assistant in Project-based General Chemistry Laboratory Courses Graduate teaching assistants (GTAs) serve as the primary instructors for the project-based GCL1 and GCL2 courses. Within this learning environment, GTAs are placed in a mentorship role in which they guide students through course projects and work collaboratively with both students and co-GTAs to ensure that student teams have developed a reasonable plan to answer the question or solve the problem posed, all team members are contributing, and laboratory work is performed safely and efficiently. Teaching assignments, such as these, can potentially provide beneficial opportunities for professional growth and development; however, their potential value can be overlooked in chemistry graduate programs because research is the primary focus with teaching often viewed purely as a means of support. The skills gap is not limited to recent bachelor’s graduates. Concerns also exist about graduate degree holders being inadequately trained and lacking the soft skills necessary to be successful in today’s workforce (National Academies of Sciences, Engineering, and Medicine, 2018; Nature, 2023). It may be possible to leverage the graduate teaching experience to build some of the soft skills desired by employers. Semi-structured interviews were conducted to explore GTA perceptions of their experiences as an instructor for project-based general chemistry laboratory courses in the context of developing valuable workplace skills. Interviews also probed GTA perceptions of the skills developed by their students to examine if there was alignment between GTA perceptions and the skills that students reported developing. Exploring if instructor and student perceptions align could provide useful information for future curriculum design aimed at targeting development of specific EDCs and training of graduate teaching assistants to implement the curriculum as intended. The research questions that guided this study, which is reported in Chapter VII, were as follows: 5 1. (RQ1): What career-relevant skills do graduate teaching assistants believe they are developing as an instructor for the general chemistry laboratory courses? 2. (RQ2): How do graduate teaching assistants perceive that they are developing skills needed for their career while serving as an instructor for the general chemistry laboratory courses? 3. (RQ3): What skills do graduate teaching assistants believe their students are developing? 4. (RQ4) How do the skills that teaching assistants think their students are developing align with skills students perceive they are building in the course? 6 REFERENCES Bell, S. (2010). Project-based Learning for the 21st Century: Skills for the Future. The Clearing House: A Journal of Educational Strategies, Issues, and Ideas, 83(2), 39–43. https://doi.org/10.1080/00098650903505415 Bridgeland, J., Milano, J., & Rosenblum, E. (2011). Across the Great Divide: Perspectives of CEOs and College Presidents on America’s Higher Education and Skills Gap. Civic Enterprises. https://files.eric.ed.gov/fulltext/ED518231.pdf Broekemier, G. M. (2002). A Comparison of Two-year and Four-year Adult Students: Motivations to Attend College and the Importance of Choice Criteria. Journal of Marketing for Higher Education, 12(1), 31–48. https://doi.org/10.1300/J050v12n01_03 Burnham, J. A. J. (2020). Skills for success: student-focused, chemistry-based, skills-developing, open-ended project work. Journal of Chemical Education, 97(2), 344–350. https://doi.org/10.1021/acs.jchemed.9b00513 Carlson, S. (2022). Building Tomorrow’s Work Force. In The Chronicle of Higher Education. The Chronicle of Higher Education. Carmel, J. H., Ward, J. S., & Cooper, M. M. (2017). A Glowing Recommendation: a Project- based Cooperative Laboratory Activity to Promote Use of the Scientific and Engineering Practices. Journal of Chemical Education, 94(5), 626–631. https://doi.org/10.1021/acs.jchemed.6b00628 Chadwick, S., De La Hunty, M., & Baker, A. (2018). Developing Awareness of Professional Behaviors and Skills in the First-Year Chemistry Laboratory. Journal of Chemical Education, 95(6), 947–953. https://doi.org/10.1021/acs.jchemed.8b00025 Cooper, M. M. (2012). Cooperative Chemistry Laboratory Manual (5th ed.). McGraw Hill. Fermin, B., & Pope, M. L. (2003). The Perceptions of College Students Regarding the Factors Most Influential in Their Decision to Attend Postsecondary Education. College and University, 78(4), 19–25. Finley, A. P. (2023). The Career-Ready Graduate: What Employers Say About the Difference College Makes. American Association of Colleges & Universities . https://dgmg81phhvh63.cloudfront.net/content/user-photos/Research/PDFs/AACU-2023- Employer-Report.pdf Galloway, K. W. (2017). Undergraduate Perceptions of Value: Degree Skills and Career Skills. Chemistry Education Research and Practice, 18(3), 435–440. https://doi.org/10.1039/c7rp00011a Gallup Inc, & Strada Education Network. (2018). Why Higher Ed? Top Reasons U.S Consumers Choose Their Educational Pathways. https://news.gallup.com/reports/226457/why-higher- 7 ed.aspx Jollands, M., Jolly, L., & Molyneaux, T. (2012). Project-based Learning as a Contributing Factor to Graduates’ Work Readiness. European Journal of Engineering Education, 37(2), 143– 154. https://doi.org/10.1080/03043797.2012.665848 Kondo, A. E., & Fair, J. D. (2017). Insight into the Chemistry Skills Gap: The Duality Between Expected and Desired Skills. Journal of Chemical Education, 94(3), 304–310. https://doi.org/10.1021/acs.jchemed.6b00566 National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, Board on Higher Education and Workforce, Policy and Global Affairs, Leshner, A. I., & Scherer, L. A. (2018). Graduate STEM Education for the 21st Century. National Academies Press. https://nap.nationalacademies.org/catalog/25038/graduate-stem-education-for-the-21st- century National Association of Colleges and Employers. (2023). Job Outlook 2023. https://www.naceweb.org/store/2022/job-outlook-2023 PhD Training is No Longer Fit for Purpose — It Needs Reform Now. (2023). Nature , 613, 414. Sinex, S. A., & Chambers, T. L. (2013). Developing Online Collaboration Skills in the General Chemistry Laboratory. Journal of Chemical Education, 90(9), 1244–1246. https://doi.org/10.1021/ed300705t Telfor, B. (2017). 16 Essential Health and Science Skill Sets. https://natsci.msu.edu/sites/_natsci/assets/File/Current%20Students/College%20Career%20 Services/Option%201%20Skill%20Sets%20with%20Definitions%20for%20Website- CNS%20Articulation%20Exercise%20Website%20Posting%20BT_ST.pdf Yasin, N. Y. B. M., & Yueying, O. (2017). Evaluating the Relevance of the Chemistry Curriculum to the Workplace: Keeping Tertiary Education Relevant. Journal of Chemical Education, 94(10), 1443–1449. https://doi.org/10.1021/acs.jchemed.7b00296 8 CHAPTER II: LITERATURE REVIEW The driving force for the studies presented in this dissertation was to investigate how the general chemistry laboratory I (GCL1) and II (GCL2) courses were perceived to provide opportunities for development of valuable 21st-century skills by both undergraduate students enrolled in these courses and the graduate student teaching assistants who instructed these courses. As large portions of this study were conducted during the COVID pandemic when undergraduate students were enrolled in remotely operated laboratory courses, the perceived impact that online learning had on students’ ability to develop important career competencies was an additional area of interest. Present in the literature are a plethora of studies, reports, and books exploring each of the factors presented above, leading to four distinct parts explored in this literature review. Part 1 will investigate the motivations for obtaining a higher education degree to showcase why students attend college and what they hope to gain from their experience. Part 2 will define the skills gap and the skills needed to be successful in the 21st-century workforce, while also exploring the connection between higher education and its role in preparing students for modern day careers. Part 3 will provide a brief history of laboratory education and how this learning environment can provide students with opportunities to foster development of valuable 21st- century skills. This literature review will conclude with Part 4 looking into the origins of online learning in university settings, the best pedagogical practices to use when operating remotely, and how learning is affected when students engage at a distance. Part 1. College Education – A Gateway to a Better Life Educational institutions are constantly under pressure to provide evidence for the value of obtaining a college degree and the preparation provided by their programs for future career success. Higher education commercials and internet ads boast the promise of success beyond a college degree through career preparation and job placement programs (Lee, 2019). Prospective students and parents seek out the best tertiary institutions through the hopeful eyes of higher education being an investment for future career success (Broekemier, 2002; Fermin & Pope, 2003; Lee, 2019). However, the value that today’s students attribute to obtaining a college degree does not always align with how higher education was perceived historically. Prior to World War II, pursuing a college degree represented the innate curiosity and drive to expand one’s knowledge reserved for the upper class (Covaleskie, 2014). Economic prosperity and 9 college degree attainment was correlated only with the financial means of the upper social classes and access to higher education institutions. The goals for obtaining a college degree have since evolved. In the years following World War II, it came to be seen as economic currency to be exchanged for a better life because it equated to more fruitful job opportunities. The introduction of the GI Bill allowed broader access to education for veterans from lower social classes, causing a shift in the purpose behind a college degree. Through this, the addition of educational qualifications for job opportunities and admission to higher paying jobs was created, ultimately leading to the perspective that a college degree will open the doors to career success and financial gain. This sentiment is shared by prospective students and parents today when selecting a higher education institution (Broekemier, 2002; Fermin & Pope, 2003; Lee, 2019) and act as a motivator for continuing education through pursuing a graduate degree (Cho-Baker et al., 2022). Along with an increase in accessibility of higher education to broader populations, the Higher Education Act of 1965 was passed to provide financial assistance to those who wanted to pursue higher education but did not have the means to (Fountain, 2018). With this, loan borrowing expanded from federal to private lenders, ultimately leading to today’s student loan debt crisis (Watson, 2019). The benefits of a college degree are now often weighed against the possibility of accumulating a burdensome amount of student loan debt that can accrue during a student’s undergraduate and graduate degree (Schmitt & Boushey, 2012). This raises the question of the worth of a college degree and its return on investment. Nuckols et al. found that college graduates struggled to justify the worth of their college degree and the subsequent acquired student loan debt; although a higher education degree provided students admittance into a chosen career path, the financial gains were not perceived as proportional to degree attainment (Nuckols et al., 2020). Instances of acquaintances without college-level degrees obtaining higher salary jobs in the workforce were cited as an argument against higher education and loan borrowing, although access to desired jobs was perceived as a positive outcome of a college degree. As of 2022, unemployment rates were lower for those who received at least some college education and continued to decline for those who received an associate degree or higher when compared to those with a high school diploma or less (U.S. Bureau of Labor Statistics, 2022). Not only were unemployment rates lower, but median weekly earnings were directly 10 proportional to the level of degree attainment, with increasing pay as the level of education increased. While these look to be promising results for prospective students entering higher education, the workforce is showing some instability with a rise in unemployment rates (Education Dynamics, 2023). A projected increase in jobs available but a lack of educated individuals to fill these spots (Education Dynamics, 2023) makes it clear that there remains a need for higher education institutions to supply graduates with the knowledge and skills needed to be competitive and successful when entering the job market. Part 2. 21st-Century Skills – Defining the Competencies & Qualities Employers Desire from College Graduates The transition to the 21st century was accompanied by a marked change in the job market (National Academies, 2011). Technological advancements, which resulted in an evolving labor force, brought forth careers that required proficiency of skill sets extending beyond conceptual knowledge and technical skills (Binkley et al., 2012; Boyer 2030 Commission, 2022; Burrus et al., 2013). Jobs now required employees to be proficient in skills such as communication, teamwork, problem-solving, and critical thinking, collectively referred to as soft, transferable, or 21st-century skills (National Academies, 2011; Olesen et al., 2020). It is the transferable nature, in which these skills are believed to be applicable across a variety of contexts and careers (National Research Council, Committee on Defining Deeper Learning and 21st Century Skills, et al., 2012; Olesen et al., 2020), that makes them highly valued. However, evidence supporting the transference of skills learned in an educational or training context to the workplace or across domains is scant (Laker & Powell, 2011; National Academies, 2011; Perkins & Salomon, 1992). This highlights how challenging it can be to bridge what was learned in the classroom to on-the- job experiences (Jackson et al., 2019). Due to the lack of evidence supporting skill transfer, accompanied by employers' desire for graduates to enter the workforce with these skills, the moniker of employer-desired competencies (EDCs) will be used to refer to these skills throughout the remainder of this dissertation. EDCs have been of interest for years in higher education due to employers perceiving a lack of preparation and inability of college graduates to display proficiency in these skills (Bridges, 1993; Burrus et al., 2013; Hart Research Associates, 2018; National Association of Colleges and Employers, 2023; The Economist Intelligence Unit Limited, 2014). This problem extends beyond undergraduate students, also affecting graduate students who have been reported 11 to lack valuable 21st-century skills such as teamwork, communication, and leadership (National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018). These findings show a disconnect between what students are learning in colleges and universities and the career preparation needed to be successful in today’s job market. A recent report by American Association of Colleges & Universities (AAC&U) showed that while employers are confident in higher education’s ability to prepare students for the workforce, there continues to be a perceived lack of preparation in EDCs, specifically communication, problem-solving, and critical thinking (Finley, 2023). While some reports focus on employers blaming higher education for not evolving in response to a changing economy and fulfilling the needs of the workforce (Burrus et al., 2013; The Economist Intelligence Unit Limited, 2014), others recognize that both educational institutions and training on-the-job offer different but valuable learning experiences (Hager & Hodkinson, 2009). The recent Boyer 2030 Commission’s report, The Equity/Excellence Imperative, sought to address areas of reform needed in higher education at research universities, focusing on providing equitable means to attaining education that can lead to success in all areas of life (Boyer 2030 Commission, 2022). They emphasized the need to again see the value in liberal education, that provides general education in a variety of areas (e.g., arts, humanities, sciences) throughout an undergraduate’s career, to meet the objective of preparing students in “world readiness”, defined as: Education for world readiness must be coherent, transparent, and explicit in purpose. It must simultaneously prepare undergraduates for life as productive citizens and economic actors where the best way to do both is to prepare students for life itself—life in our times and with an anticipated future in mind, which is to say, for world readiness (Boyer 2030 Commission, 2022). This goes beyond the short-term goal of obtaining a job post-graduation, to recognizing the need to provide an education that allows students to seek fulfilling careers or life goals, in which students feel they are making meaningful contributions within both society and their personal lives. Within this report, pertinent skills such as communication (both oral and written) and critical thinking emerged as necessary for long-term career success. From this came the recommendation to integrate these skills throughout a student’s college education, pushing the 12 focus from discipline-specific knowledge alone to the culmination of broad skill sets. To achieve this outcome, the report suggested being explicit about the importance of general education courses and how they are key agents to better prepare students for life and career success, along with continually introducing students to career opportunities and resources from early in a student’s undergraduate career. Despite the need for better career preparation as a goal of undergraduate education, implementing EDC development in higher education can be problematic due to two key issues – 1) lack of consensus on how to define these skills and 2) inadequate methods of assessment to measure proficiency (National Academies, 2011; Olesen et al., 2020). While clear and measurable definitions of each individual EDC are lacking, which will be explored below, a central issue concerning all EDCs is the umbrella terminology used to describe them. Skills are defined by one’s ability to use and apply knowledge to complete a task (Lamri & Lubart, 2023; National Research Council, Committee on Defining Deeper Learning and 21st Century Skills, et al., 2012), and while EDCs are often described as transferable, Hager & Hodkinson contest the perception that skills are contextually independent and the notion that what is acquired from an educational learning environment can be wholly transferred to the workplace (Hager & Hodkinson, 2009). They argued that instead of viewing formal education as the primary means of preparing workforce-ready graduates equipped with skills and knowledge that transfer to the job setting, educational institutions should acknowledge their place as being one of many contributing to a learner’s experience. With the two environments recognized as disparate contexts, it is posited that transference between an educational setting and workforce environment can be difficult (Jackson, 2016), and that situating learning in close or similar contexts (known as near transfer) is more likely to result in transference than when contexts drastically differ (known as far transfer) (National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, et al., 2018; Olesen et al., 2020; Perkins & Salomon, 1992). Instead of believing knowledge to be transferred as whole, the idea of transfer has been suggested to instead be viewed as a “renovation and expansion of previous knowledge via the experience of dealing with new situations in new settings” (Hager & Hodkinson, 2009). We adopt this notion that the skills students believe they are developing in the general chemistry laboratory classrooms may continue to aid students’ further development in 13 valuable 21st-century skills as they progress throughout their academic and professional careers but will not be the sole experiences in which proficiency in these skills are developed. To better understand the concept of EDCs, we will first explore how four skills that appear throughout the literature are defined, focusing on the skills of communication, teamwork, problem-solving, and critical thinking, followed by discussing strategies proposed to assess these skills. Defining Communication & Teamwork Skills Communication and teamwork skills are often grouped together under the umbrella category of interpersonal skills, defined by social interactions that occur between individuals (National Research Council, Committee on Defining Deeper Learning and 21st Century Skills, et al., 2012; National Research Council et al., 2011). Communication skills are often further divided into the subskills of written (e.g., generating reports/documents, emails) (Moore & Morton, 2017), oral (e.g., speaking, presenting, listening effectively) (Burrus et al., 2013; Hanson & Overton, 2011; National Research Council et al., 2011), and non-verbal (e.g., reading body language) (National Research Council et al., 2011) communication. Teamwork skills are commonly defined as one’s ability to work well with others in diverse teams to reach a common goal (American Management Association, 2012; Partnership for 21st Century Learning, 2019). The modern-day workforce often employs the use of multidisciplinary teams, where expertise and knowledge from diverse disciplines are regarded as valuable contributions towards completion of a goal, as seen across a broad array of industries such as the health sciences (Disis & Slattery, 2010), agriculture (Bullock et al., 2007), and even military operations (Fuell Jr., 2009). Better problem- solving capabilities and solutions can result when fruitful collaboration occurs (Kozlowski & Bell, 2004; National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, et al., 2018) and, beyond this, collaboration has been recognized as a way to support skill transfer on the job (Jackson et al., 2019). Additionally, a positive correlation has been observed between occupations that require strong communication skills and higher pay (Burrus et al., 2013). While interpersonal skills are recognized as valuable to career success, these skills are difficult to measure (National Research Council et al., 2011). In addition to the lack of clarity and breadth of definitions surrounding these skills, social interactions are not easily standardizable and are always in a state of flux as they are influenced by societal culture and 14 norms (National Research Council et al., 2011). When addressing the transferability of EDCs and the need to more clearly define the domains of social context in which these skills are applied, Bridges acknowledged the following: One tempting path is to argue that no two social contexts are never quite alike. [. . .] It is easy to argue from empirical evidence as well as a priori reasoning that any social context is, literally, unique and hence that any reproductions of a skill learned in one social context in another must involve transferability (Bridges, 1993). Although Bridges countered this thought by suggesting the need for identification of broadly applicable skills that are more likely to transfer, it still captures the key issue of context that emerges when trying to define and assess interpersonal skills. More & Morton (2017) added further evidence discounting the transferability of written communication skills across contexts, showcasing the inherent differences in school and work environments. They found that employers, interviewed from a variety of industries (e.g., sciences, education, accounting/finance), recognized a disconnect between the style of communication used in higher education versus what was needed on the job (Moore & Morton, 2017); employers acknowledged a transitional phase, where students had to adapt to a different form of communication, distinct from college norms, and use methods that were specific to each work environment. These findings suggest that written communication skills are not independent from the context in which they are learned or applied, undermining the proposed generalizability of these skills. While there are many challenges to assessing interpersonal skills and their transferability, implementation of programs and curricula that focused on development and assessment of these skills have shown positive short-term (e.g., an increase in self-reported abilities based on survey results administered in a single semester) (Reynders et al., 2019) and long-term (e.g., high school student enrollment in and continuation of higher education) (National Research Council et al., 2011) outcomes. Some of the methods reported in the literature to assess communication and teamwork skills are explored below. 15 Measuring & Assessing Communication & Teamwork Skills AAC&U launched the Valid Assessment of Learning in Undergraduate Education (VALUE) initiative in response to mounting concerns about the value of a college education and the need to better measure student learning beyond standardized testing (Association of American Colleges and Universities, 2009). This initiative resulted in the development of the VALUE rubrics that were designed to assess if student work met the criteria for 16 Essential Learning Outcomes (ELOs). These rubrics consist of a definition for each ELO, criteria for assessment of each ELO, and performance descriptors for different levels of student achievement. These rubrics were intended to be broadly applicable across the undergraduate curriculum and are not discipline- specific. The ELOs include interpersonal skills, such as written communication, oral communication, and teamwork. The ways in which these three skills were defined as ELOs within the VALUE rubrics and corresponding assessment criteria are outlined in Table 2.1. All information provided within Table 2.1 uses the exact wording found within each rubric. Each criterion contained descriptions of four levels of proficiency, from benchmark (minimal) to capstone (exemplary). For example, the assessment criteria for contributes to team meetings, found within teamwork skills, defines benchmark performance as “Shares ideas but does not advance the work of the group.” and captstone performance as “Helps the team move forward by articulating merits of alternative ideas or proposals.”(American Association of Colleges and Universities, 2009b). These rubrics were not designed for grading purposes, but act as a resource to evaluate higher education programs. Table 2.1. VALUE rubric definition, assessment items, and examples of performance for written communication, oral communication, and teamwork skills reproduced from American Association of Colleges and Universities, 2009d, 2009b, 2009e. Skill Written Communication (American Association of Colleges and Universities, 2009c) Definition Written communication is the development and expression of ideas in writing. Written communication involves learning to work in many genres and styles. It can involve working with many different writing technologies, and mixing texts, data, and images. Written communication abilities develop through iterative experiences across the curriculum. Assessment Criteria Context and purpose for writing Content development Genre and disciplinary convention Sources and evaluations Control of syntax and mechanics 16 Table 2.1 (cont’d) Oral Communication (American Association of Colleges and Universities, 2009a) Teamwork (American Association of Colleges and Universities, 2009b) Oral communication is a prepared, purposeful presentation designed to increase knowledge, to foster understanding, or to promote change in the listeners’ attitudes, values, beliefs, or behaviors. Teamwork is behaviors under the control of individual team members (effort they put into team tasks, their manner of interacting with others on team, and the quantity and quality of contributions they make to team discussions). Organization Language Delivery Supporting material Central message Contributes to team meetings Facilitates contributions of team members Individual contributions outside of team meetings Fosters constructive team climate Responds to conflict Other ways in which interpersonal teamwork skills have been assessed include individual and peer reflections (National Research Council et al., 2011). Use of Comprehensive Assessment of Team Member Effectiveness (CATME) Peer Evaluation surveys, in which students reflect on their own performance as well as their teammates and receive feedback from team members, is one such example (Loughry et al., 2007). The CATME assessment criteria include a) contributing to the team, b) interacting with teammates, c) keeping team on track, d) expecting quality, and e) having relevant knowledge, skills, and abilities. Although this assessment tool was developed in 2007, it is still used to assess student performance in teamwork in college classrooms today (Purdue University, 2024). Within chemistry laboratory courses, interpersonal skills have been measured through rubrics that allowed for both self and instructor assessment of performance. Reynders et al. used rubrics developed by the Enhancing Learning by Improving Process Skills in STEM (ELPSS) project to measure student proficiency in process skills (e.g., interpersonal communication, teamwork, critical thinking) (Reynders et al., 2019). These rubrics were created by a multidisciplinary team of researchers to provide STEM instructors with resources for in- classroom assessment of process skills and formative feedback to students to support improvement (Czajka et al., 2021). The rubric for each process skill consisted of multiple assessment criteria. For example, interpersonal communication had a) speaking, b) listening, c) non-verbal, and d) response criteria with scoring on a scale from 0 to 5. When piloting earlier 17 iterations of these rubrics, it was found that after receiving feedback from undergraduate teaching assistants (UTAs), student self-assessment scores regarding skill proficiency became more statistically aligned to those of UTA assessment scores (Czajka et al., 2021). From these findings, the authors suggested that students were able to more accurately assess their performance when integrating feedback from a trained observer. Reynders et al. used these rubrics in self-reflection activities, which resulted in students becoming more aware of their actions regarding communication and teamwork skills and end-of-semester questionnaires returning high numbers for self-reported gains in development of these skills (Reynders et al., 2019). Chadwick et al. designed the Professional Practice Points rubric, to be used by both students and graduate teaching assistants (GTAs) to assess student performance within the three domains of organizational, interpersonal, and work-based skills (Chadwick et al., 2018). Scores were assigned on a four-point scale from not-evident to exemplary based on defined characteristics for competency in each domain. Results of this study yielded similar findings to those in the study reported by Czajka et al. Students became better at assessing their own performance after being given feedback by GTAs. Another positive outcome from GTAs using these rubrics to provide feedback to students was increased confidence for students who had underreported their abilities. The use of rubrics and comparison of self and instructor assessment reveal the benefits of students reflecting upon their performance and being given opportunities to adjust behaviors according to feedback received. However, acknowledged within these studies was the time it takes to train TAs to recognize behaviors indicative of competency and the additional time it takes to provide personalized feedback to students regarding their performance. These potential limitations may act as deterrents to implementing these methods of assessment within the laboratory curriculum, although the positive outcomes seen within the studies above may outweigh those drawbacks. Defining Problem-solving & Critical Thinking Situated in the cognitive domain, problem-solving and critical thinking skills have been a topic of much debate in education for years, due to these terms being broadly used but lacking clarity and clear definition (Cooper, 2016; Halonen, 1995; National Academies 2011). Recent studies in chemical education have explored the perception of students, academic teaching staff, and 18 employers about this poorly defined construct, that is relevant to both education and the workforce (Bowen, 2022, Danczak et al., 2017). When asked how critical thinking is defined, Bowen found that all student participants related this skill to application and use of knowledge, while contrasting it with more passive methods of learning (e.g., rote memorization)(Bowen 2022). Danczak et al. found in surveying students, teaching staff, and employers about critical thinking that all groups identified analysis as a key component, whether of data, evidence, or information (Danczak et al., 2017). In this study, problem-solving was also frequently associated with critical thinking in both student and employer responses; the interconnectedness of these two skills is often referenced in the literature. Partnership for 21st Century Learning, an organization dedicated to advancing student education for workforce success, grouped problem- solving and critical thinking together, defining these skills as the ability to reason through analyzing, evaluating, and interpreting data or information, and seeking solutions to problems through creative or innovative methods (Partnership for 21st Century Learning, 2019). These aspects emerged as defining aspects of problem-solving and critical thinking in both the NRC’s report on the Assessment of 21st Century Skills (National Research Council et al., 2011) and a review of the literature by Binkley et al. (Binkley et al., 2012). Within the Danczak et al. study, teaching staff and employers’ responses contained more themes on average than responses from students, possibly indicating that students have a lack of understanding or experience with this skill. This is consistent the findings of others studies in which it has been found that the ways in which novice learners apply problem solving methods differs from experts (National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, et al., 2018; National Research Council Committee on Developments in the Science of Learning et al., 1999; Persky & Robinson, 2017; Randles & Overton, 2015), suggesting experience shapes how these skills are used. Measuring & Assessing Problem-solving & Critical Thinking Skills Problem-solving and critical thinking skills have been assessed for years (Binkley et al., 2012) using a multitude of methods ranging from standardized tests used in professional school admissions (e.g., Medical College Admissions Test or MCAT) (National Research Council et al., 2011) to intentionally designed course-level assessments (Stowe & Cooper, 2017). Chemistry 19 education has offered key contributions to the literature in determining the best methods for measuring these skills, with some of these examples explored in depth below. In addition to seeking defining factors of critical thinking, Danczak et al. developed the Danczak-Overton-Thompson Chemistry Critical Thinking Test (DOT test) instrument to assess critical thinking skills (Danczak et al., 2020). This research team sought to develop an instrument within the context of chemistry, recognizing that situating an assessment within the context in which it is measured may provide a better representation of students’ critical thinking skills over more generalized testing. This assessment had five multiple-choice sections 1) making assumptions, 2) developing hypotheses, 3) testing hypotheses, 4) drawing conclusions, and 5) analyzing arguments. In each section, a passage outlining relevant context was provided followed by statements that students were asked to assess. For example, students must determine if a statement represents a valid or invalid assumption in the making assumptions section. Stowe & Cooper proposed that ill-defined skills such as problem-solving and critical thinking can be better elucidated and measured through assessment of students’ ability to engage in the scientific practices, which are discussed in Part 3 (Stowe & Cooper, 2017). Using the Three-Dimensional Learning Assessment Protocol (3D-LAP) to measure 118 organic chemistry exam items and their ability to elicit use of the scientific practices, Stowe & Cooper found that 93% of exam prompts in their study did not offer students the opportunity to engage in these practices, frequently because the prompts did not elicit evidence of student reasoning which could be considered a hallmark of critical thinking. In promoting the scientific practices as a path to eliciting evidence of student engagement in intellectual work that could be construed as critical thinking, they offered the 3D-LAP as a guide to designing assessments. Additionally, they emphasized that multiple-choice or selected-response questions commonly found on standardized tests are often not adequate measurements of student reasoning. Using well-crafted activities to elicit student use of critical thinking skills is further supported by findings in Bowen’s study where students were able to identify course activities, developed using a three- dimensional learning framework, and relate these activities to application of critical thinking skills (Bowen, 2022). Stephenson & Sadler-McKnight measured critical thinking skills using the California Critical Thinking Skills Test (CCTST) to compare the critical thinking skills of students enrolled in traditional or inquiry-based general chemistry laboratory courses (Stephenson & Sadler- 20 McKnight, 2016). The inquiry-based courses in the study used the Science Writing Heuristic (SWH) pedagogical approach to encourage critical thinking skill development. This writing-to- learn pedagogy provides students with a scenario from which student groups collaboratively formulate questions to investigate, engage in data collection, analyze and interpret the data, and then use evidence-based reasoning to justify claims and reflect upon the process. While pre-test scores on the CCTST did not differ, students in the SWH course had significantly higher post- test scores than those in the traditional general chemistry laboratory course. These results supported the conclusion that incorporating more student-centered inquiry-based learning approaches can result in development of critical thinking skills. Other inquiry-based methods that have been proposed to support development of problem-solving and critical thinking skills in the chemistry laboratory include project-based and problem-based learning, although these studies relied on self-reporting measures that only touched briefly on development of these skills (Quattrucci, 2018; Weaver et al., 2016). Part 3. Learning in the Laboratory – A Brief History of the Evolution of Its Purpose & Desired Outcomes Over the years, the role of laboratory courses in the curriculum, particularly what the student learning outcomes should be and whether students are actually achieving these learning goals, has been a debated (Bretz, 2019; Hofstein & Lunetta, 1982, 2004; Hofstein & Mamlok-Naaman, 2007; Reid & Shah, 2007). Despite many efforts to transform laboratory courses and implement new and diverse pedagogical practices, a lack of evidence about the effectiveness of laboratory courses, student disinterest, and mounting costs has led to questions about the necessity of offering these course (Bretz, 2019). To better understand the purpose behind the laboratory in chemistry education, it is pertinent to discuss the evolution of learning objectives throughout the years. In the early 1800’s, Thomas Thomson, at the University of Edinburgh, and Justus von Liebig, at the University of Giessen, introduced the laboratory to university chemistry education (Hegarty-Hazel, 1990; Klickstein, 1948; Reid & Shah, 2007). This acted as a catalyst for the emergence of both institutionally and privately run laboratory courses (Hegarty-Hazel, 1990), in which professors would instruct students in technical skills (e.g., laboratory skills and techniques) to prepare them for industry professions of the time (Reid & Shah, 2007). Skills were taught by having students perform experiments that confirmed theoretical concepts learned 21 in lecture. By the mid 1900’s, inquiry-based learning approaches, which eschewed the strict reliance on the laboratory manual to carry out confirmatory experiments, began to appear in laboratory instruction (Hofstein & Lunetta, 1982). Hofstein & Lunetta defined inquiry as: [. . .] diverse ways in which scientists study the natural world, propose ideas, and explain and justify assertions based upon evidence derived from scientific work. It also refers to more authentic ways in which learners can investigate the natural world, propose ideas, and explain and justify assertions based upon evidence and, in the process, sense the spirit of science (Hofstein & Lunetta, 2004). Inquiry was seen as a way to familiarize students with the process of scientific discovery and the work of scientists. Further, the desire to engage students in learning that enables them to think and act as scientists do led to defining eight scientific and engineering practices in A Framework for K-12 Science Education (Table 2.2) (National Research Council & Committee on a Conceptual Framework for New K-12 Science Education Standards, 2012). These eight practices were developed to get students acquainted with the world of science and engineering and give them ways to apply their knowledge. Although the scientific and engineering practices were developed to support science learning in primary and secondary schools, they are applicable to higher education. Table 2.2. Scientific & engineering practices (National Research Council & Committee on a Conceptual Framework for New K-12 Science Education Standards, 2012). 1. Asking questions (science) & defining problems (engineering) NRC’s 8 Scientific & Engineering Practices 2. Developing and using models 3. Planning & carrying out investigations 4. Analyzing & interpreting data 5. Using mathematics & computational thinking 6. Constructing explanations (science) and designing solutions (engineering) 7. Engaging in argumentation from evidence 8. Obtaining, evaluating, & communicating information 22 While there are varying levels of inquiry based on the level of guidance and freedom provided to students (Buck et al., 2008), implementing any level of inquiry-based learning into laboratory courses have been reported to be beneficial to student learning through increased motivation, interest, engagement, critical thinking, persistence in STEM, engagement with the scientific practices, and nourishing student curiosity (Carmel et al., 2017; Kai et al., 2017; Laredo, 2013; President’s Council of Advisors on Science and Technology, 2012; Quattrucci, 2018). Amid these positive outcomes, not all studies support an increase in factors such as student interest, enjoyment, or motivation when compared to more traditional methods (George- Williams et al., 2018). Although engaging students in the process of inquiry has been a desired goal in laboratory learning for quite some time (Hofstein & Lunetta, 1982), an analysis of laboratory manuals by Buck et al. found that most instructors were still using traditional methods, characterized by low levels of inquiry (Buck et al., 2008). Additionally, the focus on building laboratory skills and techniques, a cornerstone of traditional laboratory curriculum, remains as evidenced by two studies that probed faculty goals for undergraduate chemistry laboratory courses (Bretz et al., 2013; Bruck et al., 2010). While it is evident that these technical skills remain an important aspect of student experiences in the laboratory, these two studies also saw the emergence of student learning goals centered around developing valuable EDCs, such as critical thinking and teamwork skills. Development of EDCs has emerged as a learning goal in chemistry laboratory courses (Reid & Shah, 2007). The American Chemical Society’s Committee on Professional Training (ACS CPT) oversees ACS approval of undergraduate chemistry programs and provides learning goals that must be met to receive approval (American Chemical Society, 2015, 2023). The most recent guidelines (2023) include development of communication (including teamwork) and scientific reasoning (including problem-solving and critical thinking) skills. Supporting growth in communication and teamwork skills were identified as critical requirements for any ACS- approved curriculum to better prepare students for workplace success. To foster development of communication skills, the ACS CPT requires the curriculum to include opportunities for students to develop proficiency in both written and oral communication supported by instructor feedback. To advance teamwork skills, the ACS CPT requires team-based activities that include assessments to measure performance. 23 Others in chemistry education have recognized the value of building skills in higher education to support workplace success (Chadwick et al., 2018; Galloway, 2017; Kondo & Fair, 2017; Yasin & Yueying, 2017). The ways in which these skills have been explored, integrated, or assessed within the context of undergraduate chemistry curriculum will be further explored in Chapter IV. Part 4. Remote Learning: Its Origins, Benefits, and Challenges Significant portions of the studies presented in this dissertation were carried out during the COVID-19 pandemic, which forced college courses to operate remotely for an extended period of time (Centers for Disease Control and Prevention, 2023; Kathy Katella, 2021). Therefore, the influence of online courses on student learning was an important factor to explore. Learning away from and outside of traditional brick-and-mortar educational institutions is referred to by many names – distance-learning, remote learning, e-learning, and online learning (OECD, 2020). This learning environment was new to many students and instructors during the pandemic (Means et al., 2020); however, distance-learning courses have been available for quite some time (Kentnor, 2015) and have been used by many to further their education (Allen et al., 2016). Defined as “a method of teaching where the student and teacher are physically separated” (Kentnor, 2015), distance-learning has come a long way from its 18th-century predecessor, the correspondence course (Clark, 2020; Kentnor, 2015). Originally delivered through the mail, remote learning has evolved with technological advances. Distance-learning has progressed from the delivery of course content through the mail to widely available educational radio and television programs to the offering of online college courses as the World Wide Web became accessible to the masses (Clark, 2020; Kentnor, 2015). Throughout this dissertation, the different names used in the literature for distance-learning will be used interchangeably and carry the same meaning. However, one clear distinction must be made – distance or remote laboratory learning in the studies reported in this dissertation pertains only to a learning environment where the instructor and learner interacted using web-based applications, including videoconferencing, while in different locations. It does not include data collection from a distance using remote- controlled instruments (Ma & Nickerson, 2006). Even prior to the emergency implementation of online learning in response to COVID-19 in the early months of 2020, remote learning was recognized as an important area of education that needed to be explored and integrated into higher education (Allen & Seaman, 2013; 24 Kentnor, 2015) as the number of students enrolled in online education courses has increased (Allen et al., 2016). Done intentionally and with careful planning, remote learning can provide a meaningful learning experience, which was not the situation with the rushed transition to online learning in early 2020 (Hodges et al., 2020). Remote learning has been praised for the accessibility that it offers to those who are unable to attend and engage in traditional higher education due to time or location constraints of work and home life, as well as its low cost both to students and institutions (Bharadwaj et al., 2023; Darby & Lang, 2019; Kentnor, 2015; OECD, 2020; Twigg, 2003). Today’s college courses operate in one of four modalities as described by Allen et al. - 1) traditional courses which meet synchronously in a designated location, 2) web-facilitated courses where the course primarily operates in person but incorporates elements of online learning to support or enhance in-person instruction, 3) hybrid (or blended) learning courses that meet in- person but allot significant portions of time to online learning, and 4) online (or remote) courses where all learning occurs outside of a physical campus location either synchronously or asynchronously (Allen et al., 2016). While remote learning can be beneficial for providing larger populations of students with educational opportunities, there are barriers to accessing and engaging in these types of courses. Students must have access to needed technology (e.g., computers) and adequate internet connectivity, along with the skills needed to use these resources (Bharadwaj et al., 2023; OECD, 2020). Making course resources accessible (e.g., closed captioning on videos) and easy to use (e.g., using online resources with user-friendly interfaces that require little effort to navigate) is another important consideration in online course design to ensure students with disabilities can fully engage in course activities (Reyes et al., 2023). Various studies also emphasize the need for students to be self-regulated and motivated to succeed in online learning environments (Jansen et al., 2020; OECD, 2020; Wang et al., 2013). Additionally, a study surveying undergraduate and graduate students from a variety of countries, including the United States, conducted by Aguilera-Hermida et al. found that students who preferred face-to-face instruction were less engaged in the learning process and struggled to adapt to online courses (Aguilera-Hermida et al., 2021). Courses conducted remotely can also be perceived as inferior to courses that operate fully in-person and on-campus (Hodges et al., 2020). Nevertheless, academic leaders see the 25 value in offering online learning opportunities both for colleges long-term plans and meeting or exceeding student learning outcomes (Allen et al., 2016). Beyond the impact that online learning can have on student experiences, instructors can find these types of courses hard to implement due to a lack of resources and knowledge about the best pedagogical practices for this learning environment (Kentnor, 2015). Although online courses became the primary modality during the pandemic, many instructors had no prior experience teaching this way and had to act as pioneers in constructing methods to best facilitate online learning (Darby & Lang, 2019). When creating remote courses, instructors must decide on how students meet (e.g., synchronously or asynchronously) and ideally should select from evidence-based pedagogical practices for online learning. Choosing whether students engage in online courses synchronously, at a specified time each week, or providing students with materials to complete asynchronously, at a time that fits within their schedule, is a key consideration when constructing a remote course (Castelli & Sarvary, 2021). Synchronous courses provide increased opportunities for collaboration and social interactions that further contribute to feelings of acceptance within the community (Bharadwaj et al., 2023) and the ability to ask questions and engage with instructors in real-time (Petillion & McNeil, 2021). In a study investigating the effect of online learning on student experiences, Kinsky et al. observed that students operating remotely felt a decrease in involvement with their peers and instructors and desired synchronous or live lectures to facilitate these interactions (Kinsky et al., 2021). Francescucci & Rohani found that students enrolled in a synchronous online marketing course had no significant differences in midterm or final exam grades when compared to students enrolled in the face-to-face option (Francescucci & Rohani, 2019). Although grades did not appear to be affected, students enrolled in the face-to-face course reported higher levels of engagement, based on pre- and post- surveys assessing factors such as interest in the course, attention span, expected attendance, and participation. Asynchronous courses offer increased flexibility for student engagement and have been observed to allow for greater reflection and more carefully crafted responses in class discussions (Hrastinski, 2008). Provision of course materials asynchronously can also overcome some of the limitations to participating synchronously including slow internet connections and older computers (Bharadwaj et al., 2023) and accommodating students in different time zones (Agopian, 2022). Hrastinski found that both synchronous and asynchronous online methods of 26 instruction have specific benefits and suggested incorporating elements used in both synchronous and asynchronous methods within the classroom environment, highlighting that synchronous components allow for more social engagement and participation that are lacking in the asynchronous environment (Hrastinski, 2008). However, combining these modalities can result in student frustrations (e.g., feeling overwhelmed when engaging with course materials) that should be anticipated and addressed (Watts, 2016). Creating structured active and collaborative learning environments has been shown to support student learning online (Bharadwaj et al., 2023; Castelli & Sarvary, 2021), although active learning is used as a broad term that lacks a clear definition and understanding (Cooper, 2016). A curriculum focused on building a community of practice that engages students in collaboration with peers and instructors can result in an increased sense of belonging and encourage development of interpersonal relationships (Hall et al., 2022). However, collaboration can be hard to facilitate online due to students not wanting to use cameras or microphones (Castelli & Sarvary, 2021). Additionally, getting full participation in group work can also be challenging (Dietrich et al., 2020). The emergency transition to remote learning added fuel to an already heated discussion regarding the importance of laboratory learning in a student’s education (Arnaud, 2020) and the necessity of adapting laboratory instruction for remote student participation during the pandemic presented a unique opportunity to study online chemistry laboratory courses. Numerous studies and reports concerning remote delivery of chemistry laboratory courses during the pandemic have been published (Accettone, 2022; Sansom, 2020; Schultz et al., 2020; Wild et al., 2020; Williams et al., 2022; Youssef et al., 2020). These accounts highlight the various benefits and challenges, from both the student and instructor perspective, that were encountered when administering laboratory courses online, which will be discussed more fully in Chapter V. Often observed throughout instructor reports documenting the ways in which they adapted methods of laboratory instruction to the online learning format was the time and effort needed to construct a learning environment reflective of instructor’s beliefs and practices. This was especially prevalent in Sansom’s reflection of transitioning a general chemistry laboratory course to remote learning, where they stated how they recognized a greater satisfaction and alignment of pedagogical beliefs with teaching practices when given time to explore the best practices for implementing a course online in comparison to the rapid transition to distance- 27 learning in response to the remote learning mandate (Sansom, 2020). Not only do instructors need time to craft a course reflective of their beliefs, but they also need time to find the best materials to facilitate learning, especially being that there are a plethora of online resources currently available. With online courses continuing to gain momentum in the higher education sphere (Seaman et al., 2018) and advances in technology offering useful resources to support student learning (Williams et al., 2022), embracing technology is advantageous to advancing higher education. Even prior to the transition, chemistry laboratory courses had begun to integrate a variety of digital and online components. Examples include a) digital badging with badges awarded for successful demonstration of a specified task or skill (Hensiek et al., 2016; Hill et al., 2022) b) electronic laboratory notebooks (Dood et al., 2018), and c) remote control of laboratory instrumentation to collect and analyze data in a different location (Ma & Nickerson, 2006). The literature on online laboratory teaching and learning before, during, and after the pandemic can support making informed decisions on the best pedagogical practices to use when designing laboratory courses with remote student participation and integrating online resources into laboratory courses. Summary A driving factor for students pursuing higher education is the belief that the return on investment will be access to better jobs and higher pay. The skills gap, in which employers believe college graduates are entering the workforce ill-prepared for success in the 21st-century job market, threatens these beliefs. In response, there has been an effort to integrate valuable workforce competencies into both college and K-12 curricula, although who should be responsible for teaching students these skills (educational institutions or employers) remains unclear. There are arguments for and against higher education having a curricular focus on enhancing employability of undergraduates. Supporters view the classroom as a place in which the right pedagogical practices can promote EDC development, while those against provide evidence that higher education and the workforce are two separate entities with distinct roles in a learner’s education and that one cannot wholly prepare them for the other. While the existing evidence for transfer of EDCs is limited and suggests that application of EDCs is contextually dependent, providing students with a variety of opportunities to develop these skills throughout their college career may still be beneficial and offer a foundation upon 28 which to build. Collaborative inquiry-based laboratory learning offers students the ability to apply their knowledge through practice and this learning environment may act as a catalyst to further encourage growth and development of EDCs. 29 REFERENCES Accettone, S. L. W. (2022). Student Perceptions of Remote Chemistry Laboratory Delivery Models. Journal of Chemical Education, 99(2), 654–668. https://doi.org/10.1021/acs.jchemed.1c00757 Agopian, T. (2022). Online Instruction During the Covid-19 Pandemic: Creating a 21st-Century Community of Learners Through Social Constructivism. The Clearing House: A Journal of Educational Strategies, Issues, and Ideas, 95(2), 85–89. https://doi.org/10.1080/00098655.2021.2014774 Aguilera-Hermida, A. P., Quiroga-Garza, A., Gómez-Mendoza, S., Del Río Villanueva, C. A., Avolio Alecchi, B., & Avci, D. (2021). Comparison of Students’ Use and Acceptance of Emergency Online Learning Due to COVID-19 in the USA, Mexico, Peru, and Turkey. Education and Information Technologies, 26(6), 6823–6845. https://doi.org/10.1007/s10639-021-10473-8 Allen, I. E., & Seaman, J. (2013). Changing Course: Ten Years of Tracking Online Education in the United States. https://files.eric.ed.gov/fulltext/ED541571.pdf Allen, I. E., Seaman, J., Poulin, R., & Straut, T. T. (2016). Online Report Card: Tracking Online Education in the United States. https://files.eric.ed.gov/fulltext/ED572777.pdf American Association of Colleges and Universities. (2009a). Oral Communication VALUE Rubric. https://www.aacu.org/initiatives/value-initiative/value-rubrics/value-rubrics-oral- communication American Association of Colleges and Universities. (2009b). Teamwork VALUE Rubric. https://www.aacu.org/initiatives/value-initiative/value-rubrics/value-rubrics-teamwork American Association of Colleges and Universities. (2009c). Written Communication VALUE Rubric. https://www.aacu.org/initiatives/value-initiative/value-rubrics/value-rubrics-written- communication American Chemical Society. (2015). ACS Guidelines and Evaluation for Bachelor’s Degree Programs. https://www.acs.org/education/policies/acs-approval-program/guidelines.html American Chemical Society. (2023). 2023 ACS Guidelines for Undergraduate Chemistry Programs: Approved. https://www.acs.org/education/policies/acs-approval- program/guidelines.html American Management Association. (2012). Critical Skills Survey Executive Summary. https://www.amanet.org/assets/1/6/2012-critical-skills-survey.pdf Arnaud, C. (2020). Questioning the Value of General Chemistry Labs. Chemical & Engineering News. https://cen.acs.org/education/undergraduate-education/Questioning-value-general- chemistry-labs/98/i18 30 Association of American Colleges and Universities. (2009). Valid Assessment of Learning in Undergraduate Education (VALUE). https://www.aacu.org/initiatives-2/value Bharadwaj, P., Shaw, C., NeJame, L., Martin, S., Janson, N., & Fox, K. (2023). Time for Class 2023: Bridging Student and Faculty Perspectives on Digital Learning. https://tytonpartners.com/time-for-class-2023-bridging-student-and-faculty-perspectives-on- digital-learning/ Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining Twenty-first Century Skills. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and Teaching of 21st Century Skills (pp. 17–66). Springer. https://doi.org/DOI 10.1007/978-94-007-2324-5 Bowen, R. S. (2022). Student Perceptions of “Critical Thinking”: Insights into Clarifying an Amorphous Construct. Chemistry Education Research and Practice, 23(3), 725–741. https://doi.org/10.1039/d2rp00097k Boyer 2030 Commission. (2022). The Equity/Excellence Imperative: a 2030 Blueprint for Undergraduate Education at U.S. Research Universities. https://wac.colostate.edu/docs/books/boyer2030/report.pdf Bretz, S. L. (2019). Evidence for the Importance of Laboratory Courses. Journal of Chemical Education, 96(2), 193–195. https://doi.org/10.1021/acs.jchemed.8b00874 Bretz, S. L., Fay, M., Bruck, L. B., & Towns, M. H. (2013). What Faculty Interviews Reveal About Meaningful Learning in the Undergraduate Chemistry Laboratory. Journal of Chemical Education, 90(3), 281–288. https://doi.org/10.1021/ed300384r Bridges, D. (1993). Transferable Skills: A Philosophical Perspective. Studies in Higher Education, 18(1), 43–51. https://doi.org/10.1080/03075079312331382448 Broekemier, G. M. (2002). A Comparison of Two-year and Four-year Adult Students: Motivations to Attend College and the Importance of Choice Criteria. Journal of Marketing for Higher Education, 12(1), 31–48. https://doi.org/10.1300/J050v12n01_03 Bruck, L. B., Towns, M., & Bretz, S. L. (2010). Faculty Perspectives of Undergraduate Chemistry Laboratory: Goals and Obstacles to Success. Journal of Chemical Education, 87(12), 1416–1424. https://doi.org/10.1021/ed900002d Buck, L. B., Lowery Bretz, S., & Towns, M. H. (2008). Characterizing the Level of Inquiry in the Undergraduate Laboratory. Journal of College Science Teaching, 38(1), 52–58. Bullock, D. S., Kitchen, N., & Bullock, D. G. (2007). Multidisciplinary Teams: A Necessity for Research in Precision Agriculture Systems. Crop Science, 47(5), 1765–1769. https://doi.org/10.2135/cropsci2007.05.0280 31 Burrus, J., Jackson, T., Xi, N., & Steinberg, J. (2013). Identifying the Most Important 21st Century Workforce Competencies: An Analysis of the Occupational Information Network (O*NET). Carmel, J. H., Ward, J. S., & Cooper, M. M. (2017). A Glowing Recommendation: a Project- based Cooperative Laboratory Activity to Promote Use of the Scientific and Engineering Practices. Journal of Chemical Education, 94(5), 626–631. https://doi.org/10.1021/acs.jchemed.6b00628 Castelli, F. R., & Sarvary, M. A. (2021). Why Students Do Not Turn on Their Video Cameras During Online Classes and an Equitable and Inclusive Plan to Encourage Them to Do So. Ecology and Evolution, 11(8), 3565–3576. https://doi.org/10.1002/ece3.7123 Centers for Disease Control and Prevention. (2023). CDC Museum COVID-19 Timeline. https://www.cdc.gov/museum/timeline/covid19.html Chadwick, S., De La Hunty, M., & Baker, A. (2018). Developing Awareness of Professional Behaviors and Skills in the First-Year Chemistry Laboratory. Journal of Chemical Education, 95(6), 947–953. https://doi.org/10.1021/acs.jchemed.8b00025 Cho-Baker, S., Kell, H. J., & Fishtein, D. (2022). Factors Considered in Graduate School Decision-Making: Implications for Graduate School Application and Acceptance. ETS Research Report Series, 2022(1), 1–18. https://doi.org/10.1002/ets2.12348 Clark, J. T. (2020). Distance Education. In E. Iadanza (Ed.), Clinical Engineering Handbook (2nd ed., pp. 410–415). Academic Press. https://doi.org/10.1016/B978-0-12-813467- 2.00063-8 Cooper, M. M. (2016). It Is Time to Say What We Mean. Journal of Chemical Education, 93(5), 799–800. https://doi.org/10.1021/acs.jchemed.6b00227 Covaleskie, J. F. (2014). What Good Is College? The Economics of College Attendance. Philosophical Studies in Education, 45, 93–101. Czajka, D., Reynders, G., Stanford, C., Cole, R., Lantz, J., & Ruder, S. (2021). A Novel Rubric Format for Providing Feedback on Process Skills to STEM Undergraduate Students. Journal of Science College Teaching, 50, 48–56. Danczak, S. M., Thompson, C. D., & Overton, T. L. (2017). “What Does the Term Critical Thinking Mean to You?” A Qualitative Analysis of Chemistry Undergraduate, Teaching Staff and Employers’ Views of Critical Thinking. Chemistry Education Research and Practice, 18(3), 420–434. https://doi.org/10.1039/c6rp00249h Danczak, S. M., Thompson, C. D., & Overton, T. L. (2020). Development and Validation of an Instrument to Measure Undergraduate Chemistry Students’ Critical Thinking Skills. 32 Chemistry Education Research and Practice, 21(1), 62–78. https://doi.org/10.1039/c8rp00130h Darby, F., & Lang, J. M. (2019). Small Teaching Online: Applying Learning Science in Online Classes. Jossey-Bass. Dietrich, N., Kentheswaran, K., Ahmadi, A., Teychene, J., Bessiere, Y., Alfenore, S., Laborie, S., Bastoul, D., Loubiere, K., Guigui, C., Sperandio, M., Barna, L., Paul, E., Cabassud, C., Line, A., & Hebrard, G. (2020). Attempts, Successes, and Failures of Distance Learning in the Time of Covid-19. Journal of Chemical Education, 97(9), 2448–2457. https://doi.org/10.1021/acs.jchemed.0c00717 Disis, M. L., & Slattery, J. T. (2010). The Road We Must Take: Multidisciplinary Team Science. Science Translational Medicine, 2(10), 22–31. https://www.science.org Dood, A. J., Johnson, L. M., & Shorb, J. M. (2018). Electronic Laboratory Notebooks Allow for Modifications in a General, Organic, and Biochemistry Chemistry Laboratory to Increase Authenticity of the Student Experience. Journal of Chemical Education, 95(11), 1922– 1928. https://doi.org/10.1021/acs.jchemed.8b00140 Education Dynamics. (2023). Survey of the Higher Education Landscape Report Identifying and Supporting Enrollment Growth Opportunities. https://news.gallup.com/poll/508352/americans-confidence-higher-education-down- sharply.aspx Fermin, B., & Pope, M. L. (2003). The Perceptions of College Students Regarding the Factors Most Influential in Their Decision to Attend Postsecondary Education. College and University, 78(4), 19–25. Finley, A. P. (2023). The Career-Ready Graduate: What Employers Say About the Difference College Makes. American Association of Colleges & Universities . https://dgmg81phhvh63.cloudfront.net/content/user-photos/Research/PDFs/AACU-2023- Employer-Report.pdf Fountain, J. H. (2018). The Higher Education Act (HEA): A Primer. https://crsreports.congress.gov Francescucci, A., & Rohani, L. (2019). Exclusively Synchronous Online (VIRI) Learning: The Impact on Student Performance and Engagement Outcomes. Journal of Marketing Education, 41(1), 60–69. https://doi.org/10.1177/0273475318818864 Fuell Jr., L. (2009). Understanding the Enemy as a Complex System: A Multidisciplinary Analytic Problem Requiring a Multidisciplinary Team Approach. Air & Space Power Journal, 23(2), 19–24. Galloway, K. W. (2017). Undergraduate Perceptions of Value: Degree Skills and Career Skills. 33 Chemistry Education Research and Practice, 18(3), 435–440. https://doi.org/10.1039/c7rp00011a George-Williams, S. R., Soo, J. T., Ziebell, A. L., Thompson, C. D., & Overton, T. L. (2018). Inquiry and Industry Inspired Laboratories: The Impact on Students’ Perceptions of Skill Development and Engagements. Chemistry Education Research and Practice, 19(2), 583– 596. https://doi.org/10.1039/c7rp00233e Hager, P., & Hodkinson, P. (2009). Moving Beyond the Metaphor of Transfer of Learning. British Educational Research Journal, 35(4), 619–638. https://doi.org/10.1080/01411920802642371 Hall, J. L., Whitaker, K. R., Seals, S. R., & Benz, P. P. (2022). Building a Community of Practice: A Case Study of Introductory College Chemistry Students. Journal for STEM Education Research, 5(3), 458–478. https://doi.org/10.1007/s41979-022-00073-7 Halonen, J. S. (1995). Demystifying Critical Thinking. Teaching of Psychology, 22(1), 75–81. Hanson, S., & Overton, T. (2011). Skills Required by New Chemistry Graduates and Their Development in Degree Programmes. https://edu.rsc.org/download?ac=509473 Hart Research Associates. (2018). Fulfilling the American Dream: Liberal Education and the Future of Work. https://files.eric.ed.gov/fulltext/ED602450.pdf Hegarty-Hazel, E. (1990). The Student Laboratory and the Science Curriculum. Routledge. Hensiek, S., Dekorver, B. K., Harwood, C. J., Fish, J., O’Shea, K., & Towns, M. (2016). Improving and Assessing Student Hands-On Laboratory Skills Through Digital Badging. Journal of Chemical Education, 93(11), 1847–1854. https://doi.org/10.1021/acs.jchemed.6b00234 Hill, M. A., Overton, T., Kitson, R. R. A., Thompson, C. D., Brookes, R. H., Coppo, P., & Bayley, L. (2022). ‘They Help Us Realise What We’re Actually Gaining’: The Impact On Undergraduates and Teaching Staff of Displaying Transferable Skills Badges. Active Learning in Higher Education, 23(1), 17–34. https://doi.org/10.1177/1469787419898023 Hodges, C., Moore, S., Lockee, B., Trust, T., & Bond, A. (2020). The Difference Between Emergency Remote Teaching and Online Learning. Educause. https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remote-teaching- and-online-learning Hofstein, A., & Lunetta, V. N. (1982). The Role of the Laboratory in Science Teaching: Neglected Aspects of Research. Review of Educational Research Summer, 52(2), 201–217. Hofstein, A., & Lunetta, V. N. (2004). The Laboratory in Science Education: Foundations for the Twenty-First Century. Science Education, 88(1), 28–54. https://doi.org/10.1002/sce.10106 34 Hofstein, A., & Mamlok-Naaman, R. (2007). The Laboratory in Science Education: The State of the Art. Chemistry Education Research and Practice, 8(2), 105–107. Hrastinski, S. (2008). Asynchronous & Synchronous E-Learning: A Study of Asynchronous and Synchronous E-learning Methods Discovered that Each Supports Different Purposes. Educause Quarterly, 4, 51–55. Jackson, D. (2016). Modeling Graduate Skill Transfer from University to the Workplace. Journal of Education and Work, 29(2), 199–231. https://doi.org/10.1080/13639080.2014.907486 Jackson, D., Fleming, J., & Rowe, A. (2019). Enabling the Transfer of Skills and Knowledge Across Classroom and Work Contexts. Vocations and Learning, 12(3), 459–478. https://doi.org/10.1007/s12186-019-09224-1 Jansen, R. S., van Leeuwen, A., Janssen, J., Conijn, R., & Kester, L. (2020). Supporting Learners’ Self-regulated Learning in Massive Open Online Courses. Computers and Education, 146. https://doi.org/10.1016/j.compedu.2019.103771 Kai, S., Chu, W., Reynolds, R. B., Tavares, N. J., Notari, M., Wing, C., & Lee, Y. (2017). 21st Century Skills Development Through Inquiry-Based Learning: From Theory to Practice. Springer Nature. Kathy Katella. (2021). Our Pandemic Year—A COVID-19 Timeline. https://www.yalemedicine.org/news/covid-timeline Kentnor, H. (2015). Distance Education and the Evolution of Online Learning in the Distance Education and the Evolution of Online Learning in the United States United States. Curriculum and Teaching Dialogue, 17(1/2), 21–34. Kinsky, E. S., Merle, P. F., & Freberg, K. (2021). Zooming Through a Pandemic: An Examination of Marketable Skills Gained by University Students During the COVID-19 Crisis. Howard Journal of Communications, 32(5), 507–529. https://doi.org/10.1080/10646175.2021.1965927 Klickstein, H. S. (1948). Thomas Thomson: Pioneer Historian of Chemistry. Chymia, 1, 37–53. https://about.jstor.org/terms Kondo, A. E., & Fair, J. D. (2017). Insight into the Chemistry Skills Gap: The Duality Between Expected and Desired Skills. Journal of Chemical Education, 94(3), 304–310. https://doi.org/10.1021/acs.jchemed.6b00566 Kozlowski, S. W., & Bell, B. S. (2004). Work Teams. In C. Spielberger (Ed.), Encyclopedia of Applied Psychology (Vol. 3, pp. 725–732). Elsevier Ltd. 35 Laker, D. R., & Powell, J. L. (2011). The differences between hard and soft skills and their relative impact on training transfer. Human Resource Development Quarterly, 22(1), 111– 122. https://doi.org/10.1002/hrdq.20063 Lamri, J., & Lubart, T. (2023). Reconciling Hard Skills and Soft Skills in a Common Framework: The Generic Skills Component Approach. Journal of Intelligence, 11(6). https://doi.org/10.3390/jintelligence11060107 Laredo, T. (2013). Changing the First-Year Chemistry Laboratory Manual to Implement a Problem-based Approach That Improves Student Engagement. Journal of Chemical Education, 90(9), 1151–1154. https://doi.org/10.1021/ed300313m Lee, Y. A. (2019). Understanding Parents’ View on US Higher Education Marketing Communication. Educational Research for Policy and Practice, 18(3), 279–299. https://doi.org/10.1007/s10671-019-09249-2 Loughry, M. L., Ohland, M. W., & Dewayne Moore, D. (2007). Development of a Theory-based Assessment of Team Member Effectiveness. Educational and Psychological Measurement, 67(3), 505–524. https://doi.org/10.1177/0013164406292085 Ma, J., & Nickerson, J. V. (2006). Hands-on, Simulated, and Remote Laboratories: A Comparative Literature Review. ACM Computing Surveys, 38(3), 1. https://doi.org/10.1145/1132960.1132961 Means, B., Neisler, J., & with Langer Research Associates. (2020). Suddenly Online: A National Survey of Undergraduates During the COVID-19 Pandemic. https://files.eric.ed.gov/fulltext/ED610781.pdf Moore, T., & Morton, J. (2017). The Myth of Job Readiness? Written Communication, Employability, and the ‘Skills Gap’ in Higher Education. Studies in Higher Education, 42(3), 591–609. https://doi.org/10.1080/03075079.2015.1067602 National Academies. (2011). Assessing 21st Century Skills. In Assessing 21st Century Skills. National Academies Press. https://doi.org/10.17226/13215 National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, Board on Higher Education and Workforce, Policy and Global Affairs, Leshner, A. I., & Scherer, L. A. (2018). Graduate STEM Education for the 21st Century. National Academies Press. https://nap.nationalacademies.org/catalog/25038/graduate-stem-education-for-the-21st- century National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, Board on Science Education, Cognitive Board on Behavioral, & Committee on How People Learn II: The Science and Practice of Learning. (2018). How People Learn II: Learners, Contexts, and Cultures. In How People Learn II: Learners, 36 Contexts, and Cultures. National Academies Press. https://doi.org/10.17226/24783 National Association of Colleges and Employers. (2023). Job Outlook 2023. https://www.naceweb.org/store/2022/job-outlook-2023 National Research Council, & Committee on a Conceptual Framework for New K-12 Science Education Standards. (2012). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. In A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. National Academies Press. https://doi.org/10.17226/13165 National Research Council, Committee on Defining Deeper Learning and 21st Century Skills, Division of Behavioral and Social Sciences and Education, Board on Science Education, Center for Education, Board on Testing and Assessment, Pellegrino, J. W., & Hilton, M. L. (2012). Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. In Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. National Academies Press. https://doi.org/10.17226/13398 National Research Council Committee on Developments in the Science of Learning, NetLibrary Inc, Bransford, J., Brown, A. L., & Cocking, R. R. (1999). How People Learn: Brain, Mind, Experience, and School. National Academy Press. National Research Council, Committee on the Assessment of 21st Century Skills, Center for Education, Board on Testing and Assessment, & Koenig, J. A. (2011). Assessing 21st Century Skills: Summary of a Workshop. In Assessing 21st Century Skills. National Academies Press. https://doi.org/10.17226/13215 Nuckols, W., Bullington, K. E., & Gregory, D. E. (2020). Was it Worth it? Using Student Loans to Finance a College Degree. Higher Education Politics & Economics, 6(1), 1–19. https://doi.org/10.32674/hepe.v6i1.1358 OECD. (2020). The Potential of Online Learning for Adults: Early Lessons from the COVID-19 Crisis. https://www.oecd.org/coronavirus/policy-responses/the-potential-of-online-learning- for-adults-early-lessons-from-the-covid-19-crisis-ee040002/ Olesen, K. B., Christensen, M. K., & O’Neill, L. D. (2020). What Do We Mean by “Transferable Skills”? A Literature Review of How the Concept is Conceptualized in Undergraduate Health Sciences Education. Higher Education, Skills and Work-Based Learning, 11(3), 616–634. https://doi.org/10.1108/HESWBL-01-2020-0012 Partnership for 21st Century Learning. (2019). Framework for 21st Century Learning Definitions. https://static.battelleforkids.org/documents/p21/P21_Framework_DefinitionsBFK.pdf Perkins, D. N., & Salomon, G. (1992). Transfer of Learning. In T. Husen & T. N. Postlethwaite (Eds.), The International Encyclopedia of Education (2nd ed., pp. 425–441). Permagon 37 Press. Persky, A. M., & Robinson, J. D. (2017). Moving from Novice to Expertise and Its Implications for Instruction. American Journal of Pharmaceutical Education, 81(9), 72–80. https://doi.org/10.5688/ajpe6065 Petillion, R. J., & McNeil, W. S. (2021). Student Satisfaction with Synchronous Online Organic Chemistry Laboratories: Prerecorded Video vs Livestream. Journal of Chemical Education, 98(9), 2861–2869. https://doi.org/10.1021/acs.jchemed.1c00549 President’s Council of Advisors on Science and Technology. (2012). Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and Mathematics. https://files.eric.ed.gov/fulltext/ED541511.pdf Purdue University. (2024). CATME: Smarter Teamwork Homepage . https://info.catme.org Quattrucci, J. G. (2018). Problem-Based Approach to Teaching Advanced Chemistry Laboratories and Developing Students’ Critical Thinking Skills. Journal of Chemical Education, 95(2), 259–266. https://doi.org/10.1021/acs.jchemed.7b00558 Randles, C. A., & Overton, T. L. (2015). Expert: Vs. novice: Approaches used by chemists when solving open-ended problems. Chemistry Education Research and Practice, 16(4), 811– 823. https://doi.org/10.1039/c5rp00114e Reid, N., & Shah, I. (2007). The Role of Laboratory Work in University Chemistry. Chemistry Education Research and Practice, 8(2), 172–185. Reyes, J. I., Meneses, J., & Xavier, M. (2023). Suitability of Online Higher Education for Learners with Disabilities: The Students’ Voices. Journal of Special Education Technology, 38(3), 370–383. https://doi.org/10.1177/01626434221131772 Reynders, G., Suh, E., Cole, R. S., & Sansom, R. L. (2019). Developing Student Process Skills in a General Chemistry Laboratory. Journal of Chemical Education, 96(10), 2109–2119. https://doi.org/10.1021/acs.jchemed.9b00441 Sansom, R. L. (2020). Pressure from the Pandemic: Pedagogical Dissatisfaction Reveals Faculty Beliefs. Journal of Chemical Education, 97(9), 2378–2382. https://doi.org/10.1021/acs.jchemed.0c00657 Schmitt, J., & Boushey, H. (2012). Why Don’t More Young People Go to College? Challenge, 55(4), 78–93. https://doi.org/10.2753/0577-5132550405 Schultz, M., Callahan, D. L., & Miltiadous, A. (2020). Development and Use of Kitchen Chemistry Home Practical Activities during Unanticipated Campus Closures. Journal of Chemical Education, 97(9), 2678–2684. https://doi.org/10.1021/acs.jchemed.0c00620 38 Seaman, J. E., Elaine Allen, I., & Seaman, J. (2018). Grade Increase: Tracking Distance Education in the United States. https://files.eric.ed.gov/fulltext/ED580852.pdf Stephenson, N. S., & Sadler-McKnight, N. P. (2016). Developing Critical Thinking Skills Using the Science Writing Heuristic in the Chemistry Laboratory. Chemistry Education Research and Practice, 17(1), 72–79. https://doi.org/10.1039/c5rp00102a Stowe, R. L., & Cooper, M. M. (2017). Practicing What We Preach: Assessing “Critical Thinking” in Organic Chemistry. Journal of Chemical Education, 94(12), 1852–1859. https://doi.org/10.1021/acs.jchemed.7b00335 The Economist Intelligence Unit Limited. (2014). Closing the Skills Gap: Companies and Colleges Collaborating for Change. https://www.luminafoundation.org/files/publications/Closing_the_skills_gap.pdf Twigg, C. A. (2003). Improving Learning and Reducing Costs: Lessons Learned from Round I of the Pew Grant Program in Course Redesign. https://www.thencat.org/PCR/Rd1Lessons.pdf U.S. Bureau of Labor Statistics. (2022). Education Pays : U.S. Bureau of Labor Statistics. https://www.bls.gov/emp/chart-unemployment-earnings-education.htm Wang, C. H., Shannon, D. M., & Ross, M. E. (2013). Students’ Characteristics, Self-Regulated Learning, Technology Self-efficacy, and Course Outcomes in Online Learning. Distance Education, 34(3), 302–323. https://doi.org/10.1080/01587919.2013.835779 Watson, C. E. (2019). Federal Financing of Higher Education at a Crossroads: The Evolution of the Student Loan Debt Crisis and the Reauthorization of the Higher Education Act of 1965. Michigan State Law Review, 2019(4), 883–978. Watts, L. (2016). Synchronous and Asynchronous Communication in Distance Learning: A Review of the Literature. The Quarterly Review of Distance Education, 17(1), 23–32. Weaver, M. G., Samoshin, A. V., Lewis, R. B., & Gainer, M. J. (2016). Developing Students’ Critical Thinking, Problem Solving, and Analysis Skills in an Inquiry-Based Synthetic Organic Laboratory Course. Journal of Chemical Education, 93(5), 847–851. https://doi.org/10.1021/acs.jchemed.5b00678 Wild, D. A., Yeung, A., Loedolff, M., & Spagnoli, D. (2020). Lessons Learned by Converting a First-Year Physical Chemistry Unit into an Online Course in 2 Weeks. Journal of Chemical Education, 97(9), 2389–2392. https://doi.org/10.1021/acs.jchemed.0c00678 Williams, N. D., Gallardo-Williams, M. T., Griffith, E. H., & Bretz, S. L. (2022). Investigating Meaningful Learning in Virtual Reality Organic Chemistry Laboratories. Journal of Chemical Education, 99(2), 1100–1105. https://doi.org/10.1021/acs.jchemed.1c00476 Yasin, N. Y. B. M., & Yueying, O. (2017). Evaluating the Relevance of the Chemistry 39 Curriculum to the Workplace: Keeping Tertiary Education Relevant. Journal of Chemical Education, 94(10), 1443–1449. https://doi.org/10.1021/acs.jchemed.7b00296 Youssef, M., McKinstry, E. L., Dunne, A., Bitton, A., Brady, A. G., & Jordan, T. (2020). Developing Engaging Remote Laboratory Activities for a Nonmajors Chemistry Course During Covid-19. Journal of Chemical Education, 97(9), 3048–3054. https://doi.org/10.1021/acs.jchemed.0c00792 40 CHAPTER III: THEORETICAL & METHODOLOGICAL FRAMEWORKS This chapter presents the theoretical and methodological frameworks supporting the studies reported in this dissertation. Part 1 will explore the theoretical underpinnings of social constructivism by outlining the ways in which knowledge is believed to be developed to establish the foundation for the theoretical framework employed in this study. Part 2 will first outline the study design of using a convergent mixed-methods approach and then focus on the qualitative methodological framework of transcendental phenomenology to set the stage for how the study was conducted. Without reporting in-depth details of data collection, which will be provided in later chapters, this section brings forth the surrounding philosophy and purpose behind the methodology chosen for this study. Part 1.Theoretical Framework: Construction of Knowledge In seeking to understand how students perceive development of EDCs in these studies, theories about how people learn or construct knowledge must be considered. Learning within these studies is believed to occur through social engagement with others, as proposed in the social constructivist theory of learning. Four key players have largely contributed to development of constructivism, namely Jean Piaget, Lev Vygotsky, Jerome Bruner, and John Dewey. Not only did their beliefs change the way in which learning was perceived, but their work acted as a catalyst for educational transformations that challenged traditional teacher-centered methods of instruction (Herman & Pinard, 2015). It was through their work that inquiry-based learning methods emerged, with their influence continuing to be seen in curriculum development and pedagogical methods used today. A common theme shared in the work of these four scholars is the belief that rote learning and didactic teaching methods rooted in memorizing and reciting disconnected, unrelated, and segmented conceptual fragments of information was far removed from the actual learning processes students engage in where meaningful learning is produced. This section will introduce the contributions these scholars have made to the theory of constructivism and conclude with discussion of how social constructivism relates to the project- based learning pedagogy used in the courses studied. Jean Piaget - Constructivism & Cognitive Development The origin of constructivism is largely attributed to Jean Piaget’s work positing that knowledge is actively constructed by the learner and is not gained via passive instruction. This theory of knowledge construction draws upon the philosophical beliefs of Aristotle and Kant (Boisvert, 41 1998). Piaget postulated that knowledge is developed through experiences and internal cognitive processes as the learner interacts with the world around them. This theory was in complete opposition to the objectivist epistemological belief, brought forth by B.F. Skinner, who saw knowledge as an external entity that was situated in an objective truth to be transmitted to the learner (Jonassen, 1991). This theory of learning which views knowledge as transmissible elements, sometimes called behaviorism, emphasizes the role of the teacher/educator in cognitive development (Murphy, 1997; Wadsworth, 1996). Being at the helm, the educator fills the unknowing student, who is perceived as a "blank slate,” with the knowledge needed to master a subject. This theory of learning is the foundation of traditional classrooms with lecture-based instruction, a method of teaching still used in many college courses today. Whereas proponents of objectivism saw knowledge as truths to be given by an instructor and taken up by students, advocates for constructivism saw knowledge as being a product of an individual’s prior knowledge and experiences. This can be seen in Piaget’s theory of cognitive development. Within this theory, Piaget believed in a “dual nature of intelligence as something both biological and logical” (Piaget, 1947). Drawing upon this, organisms strive for equilibrium, a process in which the organism (e.g., learner) adapts to the environment based on stimuli (e.g., incoming information) and the tendency for organisms to want to achieve this state. This self- regulated process involves the interplay between assimilation and accommodation of schema to achieve equilibrium. Schema, also known as the “building blocks” for constructing knowledge, are organized and related units of information that can have simple relationships in earlier stages or form complex systems in more advanced stages of cognition (Oogarah-Pratap et al., 2021). When new knowledge is introduced, a learner integrates this information through either assimilation or accommodation. Assimilation involves the learner incorporating new information into predefined schemas, a process that allows for the learner to remain in an equilibrium state where prior knowledge remains unchallenged. Incoming information is processed and essentially filed or placed into existing schemas, causing the schema to grow but not be modified (Wadsworth, 1996). In contrast, accommodation occurs when the learner enters a disequilibrium state, also known as cognitive dissonance, where new knowledge contradicts or challenges what is already known. To once again reach equilibrium, the learner must modify their previously defined schemas or generate new ones to accommodate this conflicting information. After accommodation occurs, the previously incongruous information is then assimilated into the new 42 or modified schema (Wadsworth, 1996). This process of learning is cyclical, whereby learners go back and forth between equilibrium and disequilibrium states as new information is processed and incorporated. At the center of this process lies the learner, whose engagement in the sensemaking process is a hallmark of constructivism. Constructivism within the classroom places the emphasis on the learner being self-governed and responsible for integrating and incorporating new information, constructing meanings, and making meaningful connections (Rannikmäe et al., 2020), in contrast to traditional classrooms where the instructor is the primary authority tasked with imparting this knowledge. These underlying principles are the basis for all constructivist theories. Social constructivism, which is considered a subcategory of constructivism, expands this theory beyond the individual to include the impact that the learner’s engagement with their social environment has on cognitive development. Vygotsky & Bruner - Social Constructivism & the Influence of Society in Learning Like Piaget’s work, Lev Vygotsky’s theory of learning also rested in the notion that learning is constructed through active participation by the learner. However, Vygotsky did not perceive learning to occur independently of the social world in which the learner is situated. He believed that it is through the influence of society and culture that cognitive development occurs, leading to Vygotsky being identified as a founding father of social constructivism (Vasileva & Balyasnikova, 2019). Generally, his theory of cognitive development stressed the importance of addressing the system rather than its individual parts. He believed that to fully understand cognitive development, a learner’s environment and their perception of this environment must also be considered. Within Vygotsky’s theory, he posited that cognitive development is the product of social intercourse that over time, with compounding experiences, becomes internalized and adapted to serve as a function or operation for the learner (Murphy, 2022; Vygotsky, 1978). To exemplify this, he used a child’s experience of internalizing the meaning of finger pointing. This comes from a child first trying to grasp an object out of reach followed by an interaction with the mother bringing the object closer evolving the initial act to a gesture of pointing towards an object that the child wants. This process involved “an interpersonal process [being] transformed into an intrapersonal one” (Vygotsky, 1978). This sociocultural influence on learning and cognitive development is thought to play an integral role during a child’s developmental years. 43 The Bucharest Early Intervention Project (BEIP) investigated the impact of neglect during early stages of development to determine the best timing for remediation or recovery, and it provided evidence for the role of social interactions in development (Nelson III et al., 2007). Within this longitudinal study, institutionalized children between the ages of 0 and 30 months were randomly placed into two study groups, continuing care within an institution or foster care with a family. A third group of children who were never institutionalized and remained in the care of their biological parents served as a control. Measuring cognitive and developmental outcomes at various timepoints, they found that children who remained institutionalized suffered cognitive delays in relation to their foster care counterparts and those who stayed with their biological families. In a related BEIP study of children who were ~10 years old, institutionalized children were found to have impaired social communication scores and skills that improved for those who entered foster care (Levin et al., 2015). In both studies, social and cognitive deficiencies were attributed to fewer social interactions with caregivers pointing to the importance of social interactions in development (Nelson et al., 2014). Although these studies focused on the developmental progress of children reared in different environments, the role of social interactions on learning is relevant to educational settings as well. Vygotsky further expanded upon the social constructivist framework for learning by introducing the concept of the zone of proximal development (ZPD), describing it as follows: It is the distance between the actual developmental level as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance or collaboration with more capable peers. (Vygotsky, 1978) The ZPD corresponds to the problem solving that learners can do with appropriate support and is found between what students can achieve independently and what they are unable to do even with support. Social interactions with instructors and peers can mediate learning in the ZPD. The instructor can act as a mentor, guiding the student to the knowledge or skills needed to perform a task that falls within the ZPD. This act of providing guidance without explicitly telling students what to do is defined by Jerome Bruner as scaffolding; it positions the instructor (or teaching assistant) as an expert source of information to support student ideas and promote self- directed learning (Rannikmäe et al., 2020; Reiser, 2004; Sanders & Sugg Welk, 2005). 44 Scaffolding can be provided through assignment design and/or verbal coaching. Iterative feedback is integral to supporting students in building knowledge through the problem-solving process (Mergendoller et al., 2006). Beyond social interactions with instructors, collaboration with more knowledgeable peers can also aid student learning in the ZPD (van Merrienboer & de Bruin, 2014; Vygotsky, 1978). Vygotsky made it clear that learning and development viewed in this regard are not synonymous; learning precedes and results in development. He also proposed that learning arises from interacting with others (e.g., ZPD), while development is the internalization of these processes (Vygotsky, 1978). Throughout his work, social engagements played a pivotal role in learning and cognitive development, a concept that has influenced the work of others. Although credit for the theory of social constructivism is most often attributed to Vygotsky, Jerome Bruner is considered a more modern contributor to this theory of development. Inspired by Vygotsky, Bruner believed in the importance of a learner engaging with their social community and posited that it is through these experiences, influenced by language and culture, that knowledge is constructed (Rannikmäe et al., 2020). Like Vygotsky, Bruner believed that the instructor should act as a supporting structure for students to guide them through the learning process. Bruner also believed that building upon prior knowledge was a process that builds gradually, with the end goal of the student possessing autonomy over learning. John Dewey - The Social Role of the Classroom & the Emergence of Inquiry-based Learning Education reformer John Dewey’s philosophy of education and pedagogy, which centered on real world engagement by learners, aligned with social constructivist theories of learning (Boisvert, 1998; Dewey, 1938; Martin, 2003). Dewey saw education as an “important social interest,” (Dewey, 1938). He believed that schools were social institutions involving relationships of students with their peers and instructors, which differed from other social structures, such as the family (Dewey, 1938). Additionally, Dewey found the social interactions in education to be pertinent to student experiences and their learning trajectories. Dewey wrote in his book Experience and Education “all human experience is ultimately social: that involves contact and communication” and “the principle that development of experience comes about through interaction means that education is essentially a social process” (Dewey, 1938). Creating a social environment in the classroom in which all participants (e.g., students and instructors) 45 contributed was of great importance to Dewey, a stark contrast to traditional methods of instruction. For Dewey, traditional methods, such as didactic lecturing, placed learners at a disadvantage. In this paradigm, students were seen as inexperienced and ill-equipped in comparison to the expert knowledge of the instructor. To remedy this, instructors, using the textbook as a leading source of information, would impart their knowledge to the unknowing learner. This pedagogical approach frequently provided learners with isolated blocks of information that were disconnected from their everyday experiences beyond the classroom. Dewey found these traditional classrooms, focused on rote memorization, to be problematic and believed that learning should be situated within relevant social and historical contexts (Dewey, 1938). Where traditional learning seemed to be aimed at the past, Dewey wanted education to be a product of the past, present, and future. Dewey was an influential contributor to the progressive education pedagogical movement (Hofstein & Lunetta, 1982; Theobald, 2009). The following statement captured the essence of his educational philosophy: I take it that the fundamental unity of the newer philosophy [of progressive education] is found in the idea that there is an intimate and necessary relation between the process of actual experience and education. (Dewey, 1938) Learners were expected to be active agents in the learning process through engagement in experiences with instructors facilitating this process. Dewey did not fully reject traditional education, as he found the complete opposition seen in some extremely progressive classrooms during his time to be more detrimental than good. He took elements of traditional education (e.g., conceptual knowledge set in the past) and set out to marry these elements with progressive components (e.g., integrating student-centered experiences) to produce a pedagogical approach aimed at dealing with problems of the present and future. He viewed complete freedom for learners and removal of the instructor an extreme approach, instead advocating for the instructor acting as a guide rather than bringer of knowledge. Much like Vygotsky’s and Bruner’s works, the instructor maintained an important role in mentoring students but was no longer the center of learning. 46 Dewey emphasized that experiences needed to be made meaningful to learners by stimulating their enjoyment, interest, and creativity. To give further explanation of what constitutes experiences, he outlined what he called an experiential continuum consisting of past, present, and future experiences. Dewey proposed that prior experiences will inextricably set the course for future experiences and will change the way a person perceives, acts, and knows, with growth resulting from the cumulative effect of prior and current experiences. The role of the instructor is to discern whether the learner is moving in the direction intended and if not, to make adjustments to guide learning. Furthermore, Dewey expressed how all experiences are a culmination of those who came before us (e.g., access to certain technologies now makes our experiences different from societies who functioned without these technologies), highlighting the social context of learning. Putting Dewey’s educational philosophy (Dewey, 1938; Martin, 2003; Williams, 2017) into practice involves creating a learning environment with an engaging social climate and community, beneficial and captivating learning experiences, and opportunities for students to have autonomy over their learning. To study the best methods for implementing inquiry-based philosophies for teaching students in grades K-12, Dewey founded the experimental and influential Laboratory School at University of Chicago in the late 1800’s (DePencier, 1967; Durst, 2010; The University of Chicago, n.d.). Dewey’s work has inspired many, with his core beliefs underlying a variety of pedagogical methods used today in which active engagement of students in their learning has replaced traditional lectures. His work gave rise to inquiry-based learning methods found in pedagogies such as Process-Oriented Guided Inquiry Learning (POGIL), Problem-based Learning, Case-based Learning, Place-based Learning, Discovery-based Learning, and Project- based Learning (PjBL) (Chase & Abrahamson, 2018; Hofstein & Lunetta, 1982; Krajcik & Blumenfeld, 2006; Moog, 2019; Barrow, 2006; Herman & Pinard, 2015; Provenzoi, 1979). The various forms of inquiry-based learning listed above are implemented in both lecture and laboratory courses in college and university settings. With project-based learning serving as the underlying structure of the laboratory courses investigated in this study, the course structure and its relationship to the social constructivist framework is explored below. 47 Project-based Learning (PjBL) - An Open Inquiry Pedagogy with a Constructivist Approach to Learning Just as the evolution of 21st-century careers has caused a shift in the skills needed for success, project-based learning resulted from changing tides from traditional to progressive methods of education (Wilhelm et al., 2019). Through Dewey’s work and the progressive education movement, the foundation for project-based learning was established. While Dewey is primarily credited as a founding father of project-based learning, it was one of his pupils, William H. Kilpatrick, who is often cited as bringing the project-based method to the masses (Burlbaw et al., 2013). Kilpatrick penned the well-known essay, The Project Method, with the goal of defining the meaning of “project.” Within this essay, he conceptualized “project” as a “purposeful act” situated within a social context. As these questionings rose more definitely to mind, there came increasingly a belief - corroborated on many sides - that the unifying idea I sought was to be found in the conception of wholehearted purposeful activity proceeding in a social environment, or more briefly, in the unit element of such activity, the hearty purposeful act. It is to this purposeful act with the emphasis on the word purpose that I myself apply the term 'project'. (Kilpatrick, 1918) A project embodied an act that had significance to the person carrying it out with genuine purpose and interest. Kilpatrick used examples of a girl making a dress or a boy “making a kite that will fly.” These acts were relevant to the individual’s present-day life, reflective of the philosophies of both Dewey and Kilpatrick, in contrast with traditional education of the time, which consisted of delivery of disconnected facts situated in the past as “preparation for future living” (Dewey, 1938; Kilpatrick, 1918). Although this essay made significant waves in education, Kilpatrick’s broad and ambiguous definition was not praised by all, including Dewey (Knoll, 2010, 2012). Kilpatrick’s approach, although met with enthusiasm, was not easy to implement because it lacked boundaries and a set curriculum. Students had free will to pursue activities deemed purposeful or cease demotivating tasks (Knoll, 2010, 2012). As a result of multiple failed attempts at implementation and backlash from the community, Kilpatrick 48 recanted his support for the use of project methods. Furthermore, Knoll has argued against crediting Kilpatrick as the founder of modern-day project-based learning methods, instead recognizing its historical roots in architecture, engineering, and agriculture, where theory and application were combined (Knoll, 2012). The concept of a “project” in conjunction with education appeared in Italian vocational schools as early as the late 16th century (Knoll, 1997). Originally implemented as a teaching method in the mid 18th century at a French architectural school, the project-based method allowed students to apply theoretical knowledge learned in lecture in practice through creating their own plans. This method was adapted by European and American engineering vocational schools, where students would apply principles learned to create a product. Instances of project- based learning can be found implemented within the United States in the mid 19th century at the Massachusetts Institute of Technology. Subsequently, project-based learning was introduced to agriculture and science education in the early 1900’s, eventually leading to its implementation across a variety of disciplines and classrooms that we see today. Modern day project-based learning is influenced by Dewey’s learning theories and pedagogical practices – the importance of the social environment combined with learning through doing. The social environment is an integral part of PjBL with the role of the teacher and the engagement of the student differing from traditional didactic classrooms. A project-based learning environment promotes social engagement and collaboration within a learning community consisting of learners and a teacher. A key feature in PjBL is the role of the teacher. Within this learning environment, the teacher takes a step back from being “all knowing” and becomes an active participant in the learning process through seeking answers to questions that they themselves and the students may not yet know (Wilhelm, 2019). This showcases the constructivist roots of project-based learning, with the teacher stepping down from the position of centralized “bringer of knowledge” to a mentor role that engages in the learning process with students. Careful attention is required to ensure that the teacher acts as a scaffold to guide students productively. This includes encouraging students’ creativity and self-guided learning, but at the same time recognizing when students are struggling and providing immediate guidance as needed, while making sure not to spoon-feed students information. Placing teachers in this role shifts the culture of the classroom away from a hierarchical structure with the teacher at the top to a collaboration between students and teacher. Collaboration with teammates or peers is 49 another key component of PjBL. Through engaging with peers, students brainstorm and share ideas, gather information, and communicate the outcomes of a project (Krajcik & Shin, 2014). Looking beyond the importance of social engagements in the PjBL classroom to the curriculum, each project is constructed with specific components to engage students in learning by doing and to give them agency in their learning. Projects 1) provide a problem and driving question with real-world significance as the starting point, 2) engage students in research and investigation into the problem and driving question provided, 3) ask students to design procedures, collect evidence, and analyze/interpret data, 4) have students report findings and propose solutions to the problem, and 5) use relevant technology to aid students throughout the process. (Krajcik & Shin, 2014; Wilhelm et al., 2019) To kickstart a project, students are given a driving question, although in some classrooms students are given the freedom to generate their own. This question is generally introduced with a brief synopsis of a problem situated within a real-world context (e.g., you work for a beverage company that produce generic sports drinks and have been asked to recreate the color profile of a name-brand beverage), followed by a question for students to answer or problem to solve (e.g., how can you use various tools and techniques in the lab to determine the color profile and successfully recreate the color profile of the name-brand beverage?). Both the scenario and driving question act as the focal point for the entire project and can continually be used as a tool to refocus students’ attention, if they get off course, as they navigate through the project (Bender, 2012). Once students are acquainted with the driving question, they engage in research by investigating the topic and problem provided. This can be facilitated by using carefully scaffolded activities that guide students through key concepts and/or techniques that may be relevant to understanding and exploring solutions to the problem. Students then generate plans and develop procedures that are then carried out to collect and analyze data to provide the evidence needed to support claims and answer to the driving question. To conclude the project, student teams compile their work on the project components (e.g., background information on problem, methods, results, conclusions) and report findings in presentations or reports. Because projects can be approached using a variety of methods, as proposed by students, the emergence of unanticipated problems or errors can occur (Wurdinger, 2016). Students are strongly encouraged to engage with their teachers and peers in a collaborative effort to overcome 50 difficulties and progress towards successful project completion. An additional component present in PjBL is the use of relevant technology to access information, collect data, analyze and model data, present results, and collaborate with others. The general chemistry laboratory courses in this study follow a cooperative learning format that incorporates the components of project-based learning (Carmel et al., 2017; Cooper, 1990, 1994). Project-based learning has been suggested as an approach to support development of highly valued workplace skills, also known as 21st-century skills (Bell, 2010; Bender, 2012; Wurdinger, 2016). Some of the skills reported as being developed within courses implementing this pedagogy include communication, teamwork & collaboration, problem solving & critical thinking, creativity, and time management. Although these skills are reported to be a product of students engaging in project-based learning, the problem of undergraduates being unable to articulate how they have developed these skills to potential employers remains. A mixed- methods approach has been followed to investigate what skills students perceive as being developed in project-based general chemistry laboratory courses at a large Midwestern University as well as their perceptions of how these skills are being developed. Part 2. Study Design & Qualitative Methodological Framework Study Design To understand student perceptions of the skills needed for their planned career and how the general chemistry laboratory courses may have supported development of these skills, a convergent mixed-methods approach was used. This approach entails concurrent collection and analyses of both qualitative (e.g., interviews) and quantitative (e.g., surveys) data. While this approach generally addresses both methods with equal priority (Creswell & Plano Clark, 2011), this study placed emphasis on qualitative methods. The aim of adopting this study design was twofold – to provide context and enhancement (Bryman, 2006). Context was provided by conducting interviews to provide a nuanced picture with the inclusion of surveys to offer generalizable results. Surveys were included within this study as an alternative extra credit opportunity for students who were unable to participate in interviews. As a result, survey data were available to provide a more general overview of student perceptions in the course. Enhancement emerged as the study progressed with the outcomes of one method being used to further support the results of another. In this study, survey results were used to provide further weight to interview findings and answer some additional questions not 51 explored in interviews. Quantitative analysis of survey data, and interview data when appropriate, was used to explore if statistically significant differences were present in how students with different student characteristics responded to questions. In this dissertation, Chapters IV, V, and VII report results from interview-based studies, while Chapter VI presents quantitative analysis of survey responses, which complement the qualitative studies found in Chapters IV and V. Two acronyms will appear frequently in discussion of the results from these studies, SPCs representing student-perceived competencies and EDCs corresponding to employer-desired competencies. Although these acronyms may seem to be interchangeable, they are distinct. SPCs refer to the skills that students report as necessary for success in their future career and will often appear in reporting and discussion of student responses. In contrast, EDCs are the skills identified as necessary for workplace success from surveys of individuals directly involved in or with knowledge of the hiring process, which have been reported in the literature. Comparison of SPCs with EDCs provides insight on the alignment of student perceptions with what employers claim to be looking for in new hires. The remainder of this chapter will explore the qualitative methodological framework used in the studies reported in Chapters IV, V, and VII. In-depth exploration of the quantitative methods used in analysis of participant data will be introduced in Chapter VI. Qualitative Methodological Framework: Transcendental Phenomenology – Origins & Practice Transcendental phenomenology was used as the methodological framework for analysis of student and graduate teaching assistant interviews. To better understand the foundation of this methodology, the philosophy and origin of this framework is reviewed. First appearing in the works of Johann Heinrich Lambert and Georg Wilhelm Friedrich Hegel, the term phenomenology is derived from two Greek words - phenomenon meaning “that which is experienced” and logos meaning “the study of” (Biemel & Spielgelberg, 2023; Williams, 2021). Thus, phenomenology is the study of that which is experienced or referred to in qualitative methodology as investigating and deriving meaning from human experiences (Creswell, 2013; Moerer-Urdahl et al., 2004). Under the umbrella of phenomenology, there are two rival schools – namely hermeneutic and transcendental phenomenology. While the two approaches have the same primary goal of gaining insight into lived experiences, where the researcher is situated within the study differs. 52 Phenomenology as a philosophical tradition, which seeks to find meaning and understanding in human experiences surrounding a particular phenomenon, is attributed to early 20th century philosopher Edmund Husserl. Husserl did not believe in the complete opposition of objective and subjective realities but recognized the interconnected nature of the two entities – with objective reality being the manifestation of one’s subjective experiences or perceptions, and meanings ascribed to them (Creswell, 2013; Emmanuel Levinas, 1998). Husserl believed that in order to see the true meaning behind these experiences, one must “bracket”, or set aside, their own beliefs and experiences (Creswell, 2013; Eddles-Hirsch, 2015; Moerer-Urdahl et al., 2004; Moustakas, 1994). This set the stage for the emergence of transcendental phenomenology, which laid the foundation for the field of phenomenology as a whole and as an investigative practice. Transcendental phenomenology relies on three philosophical tenets - intentionality, intuition, and epoché (Cerbone, 2006; Moustakas, 1994). Intentionality is the conscious mind building meanings about world around us through experiences directed towards external objects, material or immaterial, leading to perception (Creswell, 2013; Kidd, 2014). Intentionality is the product of two correlated philosophical tenets, noema and noesis. Creswell states that noema refers to an individual’s perception of a phenomenon, based on thoughts and feelings, considered to be “what” is being perceived, while noesis constitutes the experiences that contribute to “how” these perceptions were developed (Creswell, 2013). These two entities are inextricably linked and combine to form the experiences and meanings individuals associate with a phenomenon (Cerbone, 2006). Intuition refers to the ability to see what is before oneself and derive the true meaning of what is through a reflective process (Moustakas, 1994). From Husserl’s perspective this occurred through describing the essence of a phenomenon. To capture the essence of a phenomenon in a research study, participant experiences are summarized into an all-encompassing description shared by participants of how the phenomenon was experienced. To see this picture clearly, the researcher must shelve their own perceptions regarding a phenomenon, a practice known as epoché. Epoché embodies acknowledgment of past experiences or preconceived notions surrounding the phenomenon under study and places these aside to perceive what is present with clear or unburdened eyes (Moustakas, 1994). Additionally, transcendental phenomenology recognizes that meanings are constructed through experiences, with no one true reality but our own. 53 In opposition to transcendentalism, Martin Heidegger, a student of Husserl’s, proposed a hermeneutic approach, creating a new branch of phenomenology also known as interpretive phenomenology. This approach contrasted with transcendentalism by acknowledging the researcher’s experiences regarding a phenomenon. The aim within this branch of phenomenology was to incorporate the researcher’s perceptions and interpretations instead of the researcher distancing themselves from the subject under investigation (Cerbone, 2006; Eddles- Hirsch, 2015). Heidegger thought it important to include the researcher’s interpretation in understanding and shaping the true meaning behind lived experiences. Including the thoughts of the researcher and moving away from simply describing experiences as presented by participants introduces the potential for biases and judgments that Husserl aimed to remove. In an effort to primarily present interview results in the voice of the undergraduate students and graduate teaching assistants, transcendental phenomenology was used as the qualitative methodological approach for this study (Creswell, 2013; Moustakas, 1994) with a heavy reliance on providing descriptive accounts and confronting personal biases. To illustrate, application of transcendental phenomenology in the study of student perceptions of 21st-century skill development, reported in Chapter IV, is described. First, the phenomenon to be investigated was identified (e.g., what skills students believe are necessary for their career goals and how students perceive development of these skills in project-based general chemistry laboratory courses) and the sample for data collection was established (e.g., students from general chemistry laboratory 1 and 2). Data were then collected from the sample through interview and open answer survey responses. Once interviews were transcribed, the researcher became familiar with the data, isolating and noting statements that were relevant to investigating the phenomenon (e.g., when asked what skills were needed for their future career goal, Devan said the following “[. . .] for optometry, one of the things I love about it is that it's more of a low stress medical field, so you're really not very stressed about your work. So I feel like a very calm and warming personality is always going to be one of the greater aspects of an optometrist.”). These excerpts were used to identify broad categories of experiences present in the data (e.g., different aspects of working within teams emerged from interviews as a course component that contributed to skill development and were aggregated under the category of Collaborative Environment). These categories were then used to identify broader or overarching themes (e.g., relating course components to development 54 of specific skills). These themes were then presented using rich descriptions of participant’s experiences and quotes as evidence. Finally, themes were used to generate a description of the overall essence of participant experiences, or a description of how the collective sample experienced the phenomenon (e.g., how development of transferable skills was perceived by study participants). Throughout this process researchers engaged in epoché by making their experiences, biases, and judgements transparent and putting them aside as much as possible as they interacted with the data and derived meanings surrounding the phenomenon. Summary Project-based learning is rooted within the constructivist theory of knowledge formation. The strong emphasis on collaborative engagement in PjBL led to the adoption of social constructivism as the primary theoretical framework employed in this study. To investigate the potential of this learning environment to promote the development of the skills necessary to be successful in the 21st-century workforce, a transcendental phenomenological methodology was used to examine student responses to interview and open-ended survey questions. In this primarily qualitative and exploratory study, this methodology allowed the researcher to explore participant experiences. While navigating through this study, it is crucial to recognize that we did not aim to define the skills participants report as being needed for their career goal. The meaning that each participant attributes to a skill can vary and is outside of the scope of this study but could be the focus of future work. A recent study exploring student perceptions of critical thinking and the meanings they attribute to this skill provides an example of such work (Bowen, 2022). 55 REFERENCES Bell, S. (2010). Project-based Learning for the 21st Century: Skills for the Future. The Clearing House: A Journal of Educational Strategies, Issues, and Ideas, 83(2), 39–43. https://doi.org/10.1080/00098650903505415 Bender, W. N. (2012). Project-based Learning: Differentiating Instruction for the 21st Century. In Project-based Learning: Differentiating Instruction for the 21st Century. Corwin. Biemel, W., & Spielgelberg, H. (2023). Phenomenology. In Encyclopaedia Britannica. Boisvert, R. D. (1998). John Dewey: Rethinking Our Time. In John Dewey: Rethinking Our Time . State University of New York Press. Bowen, R. S. (2022). Student Perceptions of “Critical Thinking”: Insights into Clarifying an Amorphous Construct. Chemistry Education Research and Practice, 23(3), 725–741. https://doi.org/10.1039/d2rp00097k Bryman, A. (2006). Integrating Quantitative and Qualitative Research: How Is It Done? Qualitative Research, 6(1), 97–113. https://doi.org/10.1177/1468794106058877 Burlbaw, L. M., Ortwein, M. J., & Williams, J. K. (2013). The Project Method in Historical Context. In R. M. Capraro, M. M. Capraro, & J. R. Morgan (Eds.), STEM Project-Based Learning: An Integrated Science, Technology, Engineering, and Mathematics (STEM) Approach, Second Edition (pp. 7–14). Sense Publishers: Imprint: SensePublishers. https://doi.org/10.1007/978-94-6209-143-6_2 Carmel, J. H., Ward, J. S., & Cooper, M. M. (2017). A Glowing Recommendation: a Project- based Cooperative Laboratory Activity to Promote Use of the Scientific and Engineering Practices. Journal of Chemical Education, 94(5), 626–631. https://doi.org/10.1021/acs.jchemed.6b00628 Cerbone, D. R. (2006). Understanding Phenomenology. In Understanding Phenomenology. Routledge. Chase, K., & Abrahamson, D. (2018). Searching for Buried Treasure: Uncovering Discovery in Discovery-based Learning. Instructional Science, 46(1), 11–33. https://doi.org/10.1007/s11251-017-9433-1 Cooper, M. M. (1990). Cooperative Chemistry Laboratories. Journal of Chemical Education, 71(4), 307. Cooper, M. M. (1994). Cooperative Chemistry Laboratories. Journal of Chemical Education, 71(4), 307. https://pubs.acs.org/sharingguidelines Creswell, J. W. (2013). Qualitative Inquiry and Research Design: Choosing Among Five Approaches (3rd ed.). Sage Publications, Inc. 56 Creswell, J. W., & Plano Clark, V. L. (2011). Designing and Conducting Mixed Methods Research (2nd ed.). Sage Publications. DePencier, I. B. (1967). The History of the Laboratory Schools: The University of Chicago 1896- 1965. Quadrangle Books. Dewey, J. (1938). Experience and Education. Collier Macmillan. Durst, A. (2010). Women Educators in the Progressive Era: The Women behind Dewey’s Laboratory School. In Women Educators in the Progressive Era: The Women behind Dewey’s Laboratory School. Palgrave Macmillan. Eddles-Hirsch, K. (2015). Phenomenology and Educational Research. International Journal of Advanced Research, 3(8), 251–260. Emmanuel Levinas. (1998). The Theory of Intuition in Husserl’s Phenomenology (2nd ed.). Northwestern University Press. Herman, W. E., & Pinard, M. R. (2015). Critically Examining Inquiry-Based Learning: John Dewey in Theory, History, and Practice. In P. Blessinger & J. M. Carfora (Eds.), Inquiry- based Learning for Multidisciplinary Programs: A Conceptual and Practical Resource for Educators (Vol. 3, pp. 43–62). Emerald Publishing Limited. https://doi.org/10.1108/s2055- 364120150000003016 Hofstein, A., & Lunetta, V. N. (1982). The Role of the Laboratory in Science Teaching: Neglected Aspects of Research. Review of Educational Research Summer, 52(2), 201–217. Jonassen, D. H. (1991). Objectivism versus Constructivism: Do We Need a New Philosophical Paradigm? Educational Technology Research and Development, 39(3), 5–14. Kidd, C. (2014). Husserl’s Phenomenological Theory of Intuition. In L. Osbeck & B. Held (Eds.), Rational Intuition (pp. 131–150). Kilpatrick, W. H. (1918). THE Project Method: The Use of the Purposeful Act in the Educative Process. Teachers College, Columbia University. Knoll, M. (1997). The Project Method: Its Vocational Education Origin and International Development. Journal of Industrial Teacher Education, 34, 59–80. Knoll, M. (2010). “A Marriage on the Rocks” An Unknown Letter by William H. Kilpatrick about his Project Method. https://files.eric.ed.gov/fulltext/ED511129.pdf Knoll, M. (2012). “I Had Made a Mistake”: William H. Kilpatrick and the Project Method. Teachers College Record, 114, 1–45. 57 Krajcik, J. S., & Blumenfeld, P. C. (2006). Project-Based Learning. In The Cambridge Handbook of the Learning Sciences (pp. 317–334). Cambridge University Press. Krajcik, J. S., & Shin, N. (2014). Project-based Learning. In The Cambridge Handbook of the Learning Sciences, Second Edition (pp. 275–297). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.018 Levin, A. R., Fox, N. A., Zeanah, C. H., & Nelson, C. A. (2015). Social Communication Difficulties and Autism in Previously Institutionalized Children. Journal of the American Academy of Child and Adolescent Psychiatry, 54(2), 108–115. https://doi.org/10.1016/j.jaac.2014.11.011 Martin, J. (2003). The Education of John Dewey: A Biography. In The Education of John Dewey: A Biography. Columbia University Press. Mergendoller, J. R., Markham, T., Ravitz, J., & Larmer, J. (2006). Pervasive Management of Project Based Learning: Teachers as Guides and Facilitators. In C. M. Everston & C. S. Weinstein (Eds.), Handbook of Classroom Management: Research, Practice, and Contemporary Issues (1st ed., pp. 583–615). Lawrence Erlbaum Associates. Moerer-Urdahl, T., Creswell, J., & Creswell, J. W. (2004). Using Transcendental Phenomenology to Explore the “Ripple Effect” in a Leadership Mentoring Program. International Journal of Qualitative Methods, 3(2). https://doi.org/10.1177/160940690400300202 Moog, R. S. (2019). Origins of POGIL: Process Oriented Guided Inquiry Learning. In S. R. Simonson (Ed.), POGIL: An Introduction to Process Oriented Guided Inquiry Learning for Those Who Wish to Empower Learners (pp. 23–41). Stylus. Moustakas, C. (1994). Phenomenological Research Methods. Sage Publications. Murphy, C. (2022). Vygotsky and Science Education Research. In Vygotsky and Science Education (pp. 48–49). Springer International Publishing. https://doi.org/10.1007/978-3- 031-05244-6 Murphy, E. (1997). Constructivism: From Philosophy to Practice. https://files.eric.ed.gov/fulltext/ED444966.pdf Nelson, C. A., Fox, N. A., & Zeanah, C. H. (2014). Romania’s Abandoned Children : Deprivation, Brain Development, and the Struggle for Recovery. In Romania’s abandoned children : deprivation, brain development, and the struggle for recovery. Harvard University Press. Nelson III, C. A., Zeanah, C. H., Fox, N. A., Marshall, P. J., Smyke, A. T., & Guthrie, D. (2007). Cognitive Recovery in Socially Deprived Young Children: The Bucharest Early Intervention Project. Science, 318, 1937–1940. https://doi.org/10.1126/science.1149774 58 Oogarah-Pratap, B., Bholoa, A., & Ramma, Y. (2021). Stage Theory of Cognitive Development - Jean Piaget. In B. Akpan & T. J. Kennedy (Eds.), Science Education in Theory and Practice: An Introductory Guide to Learning Theory (pp. 133–135). Springer International Publishing : Imprint: Springer. http://www.springer.com/series/13812 Piaget, J. (1947). The Psychology of Intelligence. Harcourt Brace. Rannikmäe, M., Holbrook, J., & Soobard, R. (2020). Social Constructivism - Jerome Bruner. In B. Akpan & T. J. Kennedy (Eds.), Science Education in Theory and Practice: An Introductory Guide to Learning Theory (pp. 260–261). Springer. Reiser, B. J. (2004). Scaffolding Complex Learning: The Mechanisms of Structuring and Problematizing Student Work. The Journal of the Learning Sciences, 13(3), 273–304. Sanders, D., & Sugg Welk, D. (2005). Strategies to Scaffold Student Learning. Nurse Educator, 30(5), 203–207. The University of Chicago. (n.d.). About Lab. Retrieved September 19, 2023, from https://www.ucls.uchicago.edu/about-lab Theobald, P. (2009). Education Now: How Re-thinking America’s Past Can Change Its Future. Paradigm Publishers. van Merrienboer, J. J. G., & de Bruin, A. B. H. (2014). Research Paradigms and Perspectives on Learning. In J. M. Spector, M. D. Merill, J. Elen, & M. J. Bishop (Eds.), Handbook of Research on Educational Communications and Technology (4th ed., pp. 21–29). Springer. Vasileva, O., & Balyasnikova, N. (2019). (Re)Introducing Vygotsky’s Thought: From Historical Overview to Contemporary Psychology. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.01515 Vygotsky, L. S. (1978). Mind in Society: Development of Higher Psychological Processes. Harvard University Press. Wadsworth, B. J. (1996). Piaget’s Theory of Cognitive and Affective Development (5th ed.). Longman. Wilhelm, J., Wilhelm, R., & Cole, M. (2019). Creating Project-Based STEM Environments The REAL Way. Springer. Williams, H. (2021). The Meaning of “Phenomenology”: Qualitative and Philosophical Phenomenological Research Methods. Qualitative Report, 26(2), 366–385. https://doi.org/10.46743/2160-3715/2021.4587 Williams, M. K. (2017). John Dewey in the 21 st Century. Journal of Inquiry & Action in 59 Education, 9(1), 91–102. Wurdinger, S. D. (2016). The Power of Project-based Learning: Helping Students Develop Important Life Skills. In The Power. Rowman & Littlefield 60 CHAPTER IV: 21st-CENTURY SKILL DEVELOPMENT IN PROJECT-BASED GENERAL CHEMISTRY LABORATORY COURSES: A MIXED-METHODS APPROACH Introduction Employers’ perceptions that college students are not building the knowledge and skills required to enter and maintain success in the 21st-century workforce persist (Carlson, 2022). In particular, employers point to deficiencies in soft skills (aka 21st-century skills), such as communication, teamwork, problem-solving, and professionalism (Binkley et al., 2012; National Association of Colleges and Employers, 2023; National Research Council et al., 2011), which are distinct from disciplinary knowledge and technical skills (Lamri & Lubart, 2023). It could be that many college students are not being given adequate opportunities through curricular and co-curricular experiences to develop these needed skills. Although even if such opportunities are offered, students may not recognize these experiences as important to their success following graduation because they may not understand that employers expect more than good grades, which are often the primary focus of student attention. However, the following quote from a study participant suggests another possibility. [. . .] they tell us to put these skills on our resume for college, and you’re not technically supposed to use these words. You’re supposed to show through your experience of [. . .] what you’ve done to show that you can do this. And that’s the, I think, even the harder part, because you can say, “Oh, I’m a really good communicator,” but without an experience to prove it you’re kind of stuck [. . .] (Devan, study participant discussing skills needed for their planned career) Students may be aware of 21st-century skills desired by employers and the importance of being able to describe experiences to demonstrate that they have specific skills, but they encounter difficulty when trying to connect experiences to specific skills. We report results from a mixed-methods study that explored students’ perceptions of the skills required for their planned career and opportunities to develop these skills in two project- based general chemistry laboratory courses at a large Midwestern R1 institution. Alignment of students’ perceptions of needed skills with skills identified by employers as critical to success in the 21st-century workforce is also examined. Study participants were predominately first- and 61 second-year undergraduate STEM students. To provide context for this study, this chapter begins with an overview of how goals and pedagogical approaches for laboratory instruction have evolved over time, the gap between employer expectations and their perceptions of recent college graduates’ preparation for the workforce, and the role of chemistry education in career preparation. Laboratory Learning in Higher Education The value of laboratory courses in higher education relative to their cost has been questioned, at least in part because of limited evidence that they support the student learning provided as a rationale for why they are necessary (Bretz, 2019; Hofstein & Lunetta, 1982). Integrated into college curricula in the 19th century, general chemistry laboratory courses were originally designed to train bench chemists for industry positions, which are in stark contrast to today’s multidisciplinary and technologically advanced careers in STEM fields (Hofstein & Lunetta, 1982; Reid & Shah, 2007). Traditionally, the curriculum centered on technical skill development (e.g., laboratory techniques) by providing structured step-by-step laboratory instructions for experiments with known outcomes and focused on reproducible results, an instructional approach that remains prevalent today (Buck et al., 2008; Russell & Weaver, 2011). These traditional laboratory courses are often viewed by students as disconnected rote exercises that do not stimulate interest or promote creativity (Lagowski, 1990; Reid & Shah, 2007; Wink & Weaver, 2011). The model of student-centered inquiry-based learning set forth by John Dewey in the early 1900’s, marked a pivotal turning point in education that would set the foundation for future laboratory transformation (Herman & Pinard, 2015; Krajcik & Blumenfeld, 2006; Provenzoi, 1979). Inquiry-based learning, realized in the laboratory environment, provides opportunities for students to propose and test hypotheses, design experiments, and make claims supported by evidence (Hofstein & Mamlok-Naaman, 2007). Four levels of inquiry-based learning have been characterized for undergraduate science laboratories, ranging from structured inquiry, where most of the information needed to conduct an investigation from defining the problem to analysis is provided, to authentic inquiry, where students determine all aspects of an investigation from formulation of a scientific question to reporting findings (Buck et al., 2008). In progressing from structured inquiry to authentic inquiry, the next level is guided inquiry, where procedures are provided but students have agency in deciding how to analyze data and communicate results. In 62 open inquiry, the student’s role in directing laboratory activities is increased even further. Students are given background information and an initial problem or question, but they are responsible for designing the rest of the investigation from generating experimental procedures to communicating results. Project-based learning (PjBL) is a form of open inquiry. Students are given a driving question to answer through investigation carried out in a collaborative environment supported by scaffolded activities (Krajcik & Blumenfeld, 2006; Robinson, 2013; Wilhelm et al., 2019). Projects investigating a particular phenomenon can range in duration from several weeks to multiple semesters and typically conclude with student teams reporting findings. A key feature of PjBL is the centrality of student engagement with peers and instructors. Through these interactions, students can consult and rely on others to bridge gaps in knowledge, brainstorm and share ideas, and collaboratively engage in problem-solving and critical thinking to work towards a common goal (Capraro, Capraro, & Morgan, 2013; Capraro, Capraro, & Slough, 2013; Wurdinger, 2016). If one takes a more expansive view of the purpose of laboratory courses, this type of learning environment can offer students the opportunity to develop career-related skills amidst a growing concern that college graduates entering the job market today are lacking important skills, commonly known as “the skills gap” (Bell, 2010; Jollands et al., 2012; Wurdinger, 2016). Employer-desired Competencies and the Skills Gap For decades, a sizeable gap has existed between the skills possessed by recent college graduates entering the workforce and those expected by employers. Employers report that many college graduates lack the skills needed for success in the 21st-century workplace, emphasizing deficiencies in soft skills as a root cause (Association for Talent and Development, 2022; Goodman et al., 2015; National Association of Colleges and Employers, 2018; Organisation for Economic Co-operation and Development, 2019; The Association of American Colleges and Universities by Hart Research Associates, 2013; Warner et al., 2011; World Economic Forum & in collaboration with The Boston Consulting Group, 2015). These soft skills, which will be referred to as employer-desired competencies (EDCs) in this study, are commonly characterized as generic or transferrable skills that are independent of a graduate’s field of study and are applicable across disciplines (Rockwood, 2021). They include communication, teamwork, problem-solving, critical thinking, and professionalism, which span 63 the cognitive, interpersonal, and intrapersonal domains. EDCs contrast with job-specific knowledge and technical skills often described as hard skills (Lamri & Lubart, 2023; National Academies of Sciences et al., 2017). This perceived deficit in EDCs has attracted the attention of academic institutions and governmental agencies since the early 1990’s because of its implications for competitiveness on a national and global level (Secretary’s Commission on Achieving Necessary Skills, 1991). The job market has undergone rapid evolution during the first two decades of the 21st- century as technological advances have created new career opportunities. Automation in the workplace replaced the need for rote technical skills, and the desire from employers for advanced skill sets grew exponentially. No longer did a resume lined with proficiencies in technical skills suffice. This new era, recognized as the fourth industrial revolution, moved at a markedly faster pace than its predecessors (Schwab, 2016). This shift affected college graduates, who were often left without the skills needed to adapt in this evolving job market, while hiring managers were faced with decreasing pools of qualified candidates, leaving many job openings unfilled (American Society for Training & Development, 2012; Association for Talent and Development, 2022; Rider & Klaeysen, 2015; The Association of American Colleges and Universities by Hart Research Associates, 2013). In addition to applicants being turned away from job openings because they lacked the necessary skills, employers reported that many current employees were deficient in EDCs, indicating a larger and more profound problem. Furthermore, perceptions of those in higher education regarding how well their institutions are preparing students for the working world are often at odds with the views of business professionals and the general public. In a recent survey, 95% of academic officers indicated that their institution adequately prepares students for the workforce, in stark contrast to a mere 11% of business leaders and 13% of the general public believing that students were adequately prepared with the skills needed (Carlson, 2022). Universities face challenges in integrating EDC development within curricular and co- curricular experiences because the definitions of EDCs and associated expectations can be vague or overly general (Olesen et al., 2020). In addition, these skills are hard to measure through traditional assessments and can require the use of methods that are both time-consuming and costly (National Research Council et al., 2011). Various organizations have put forth frameworks in a call to action, suggesting potential strategies for addressing this problem 64 (Crompton & Sykora, 2021; International Society for Technology in Education, 2019; Jones & King, 2012; Metiri Group, 2003; Organisation for Economic Co-operation and Development, 2019; Partnership for 21st Century Learning: A Network of Battelle for Kids, 2019; Secretary’s Commission on Achieving Necessary Skills, 1991). The Framework for 21st Century Learning developed by P21 provides a vision for K-12 education that integrates learning in key subjects and 21st-century themes with life & career skills, learning and innovation skills - 4C’s (critical thinking, communication, collaboration, and creativity), and information, media & technology skills (Partnership for 21st Century Learning: A Network of Battelle for Kids, 2019).The Common Core State Standards (CCSS) in English Language Arts and Mathematics address competency development to build career readiness and college preparedness, detailing learning milestones for each grade in K-12 (Jones & King, 2012). The enGauge 21st-century skills offer another perspective on competencies essential for “success in a digital age” (Metiri Group, 2003). The Metiri Group, which contributed to the enGauge project, subsequently defined individual 21st-century skills and described corresponding levels of attainment to guide teachers in measuring student progress (Metiri Group, 2017). International organizations, such as the Organization for Economic Co-operation and Development (OECD) and the International Society for Technology in Education (ISTE), have developed standards for creating globally- competent citizens (International Society for Technology in Education, 2019). Problem-solving, critical thinking, communication, and collaboration stand out as competencies found across these initiatives. Unfortunately, despite the attention given to identifying competencies required for workplace success and incorporating these competencies in education standards, the skills gap remains. In 2019, the Society for Human Resource Management (SHRM) published the results of a 2018 survey of US human resource professionals involved in the recruitment of new hires (Burner et al., 2019). Three-quarters of those reporting difficulty filling positions attribute it to inadequate technical field-specific skills and/or soft skills, while 51% reported that educational institutions are not doing enough to close the skills gap, underscoring the importance of educational institutions making a greater effort to prepare students for today’s workforce. The investment in time and money required to earn a college degree carries expectations that higher education will deliver on the promise of providing a path to a fulfilling career and upward economic mobility (Gallup Inc. & Lumina Foundation, 2022). Academic institutions have a 65 responsibility to assess if their graduates have the discipline-specific knowledge and skills as well as EDCs required to enter the workforce and a foundation for further professional growth. Chemistry Education & Career Preparation Expectations in the chemical industry for graduates with B.S. Chemistry degrees have also evolved, although the undergraduate chemistry curriculum has not necessarily adapted. In a survey of individuals working in the chemical industry serving as a proxy for employers who hire recent graduates with chemistry degrees, Kondo & Fair found that respondents rated interpersonal skills (93%), teamwork (83%), strong work ethic (79%), and problem-solving skills (68%) as very important (Kondo & Fair, 2017). At the same time, while over 80% of respondents indicated that they expected formal instruction, practical application, and evaluation/feedback in undergraduate chemistry programs to include technical skills, analytical and quantitative skills, written communication skills, problem-solving skills, and oral communication skills, only 55% expected explicit training in teamwork and 47% in development of interpersonal skills. However, among respondents who didn’t expect formal instruction, 40% indicated that they would like to see training in teamwork and 50% in interpersonal skills. The fact that roughly half the respondents expected formal instruction in teamwork and interpersonal skills with many of the remaining respondents identifying such training as desirable suggests a need for undergraduate chemistry programs to intentionally incorporate opportunities to build these skills in the curriculum. The American Chemical Society Committee on Professional Training (ACS CPT) recognized the need for going beyond building technical skills and content knowledge to prepare chemistry bachelor’s degree graduates for the workforce. This 2015 report recommended incorporating the development of problem-solving, communication, and team skills into the undergraduate chemistry curriculum (American Chemical Society, 2015). Recent guidelines released by ACS show that these skills continue to be identified as important skills to be incorporated into laboratory curriculum (American Chemical Society, 2023). Many of the skills identified by individuals working in the chemical industry and the ACS are broadly applicable to fields beyond chemistry, which is important in the context of this study as the majority of students enrolled in the general chemistry laboratory courses studied do not aspire to careers in chemistry. 66 Studies and interventions conducted in several countries offer additional perspectives on career preparation in the context of undergraduate chemistry programs and courses.  Yasin & Yueying found in surveys of undergraduate chemistry majors and prospective employers in Singapore that the transferable skills of communication and teamwork were highly ranked by both groups; however, less than half of the students reported acquiring these skills, 27% for communication and 38% for teamwork (Yasin & Yueying, 2017). From questionnaires administered to undergraduates pursuing a chemistry degree at a UK institution, Galloway found that over 90% of respondents planning chemistry-focused careers as well as over 90% of those planning careers in other areas identified problem-solving, teamwork, time management, and independent learning ability as very useful or useful for their planned career (Galloway, 2017). Where the two groups differed most significantly was in their perceptions of the usefulness of chemistry content knowledge and safe chemical handling practices, with students planning careers where chemistry is not central finding these aspects of their undergraduate program less useful. Interestingly, even among respondents planning careers that were not chemistry-focused, significant numbers reported that analytical techniques (over 80%) and manipulative practical skills (over 90%) as very useful or useful for chemistry-related aspects of their planned career. Chadwick and co-workers introduced assessment of organizational skills (preparation and organization), interpersonal skills, and work-based skills (teamwork, initiative, independence, and working safely) to a first-year chemistry laboratory for STEM students in Australia to promote development of professional skills (Chadwick et al., 2018). Over half the students reported that assessment of these professional skills helped them understand expectations and built their confidence. Instructors reported improvement in professional skills and engagement as well as greater alignment between student and instructor evaluations over time. A project-based general chemistry laboratory course (Carmel et al., 2019) modeled on the Cooperative Chemistry project-based curriculum (Cooper, 1990) that implements an open inquiry learning approach and is centered on scientific practices can potentially offer an inclusive opportunity for large numbers of STEM students to begin developing EDCs early in their undergraduate careers. Semi-structured interviews elicited an in-depth and nuanced picture of participants’ experiences with EDC development in these courses, while statistical analyses was used as supporting evidence when identifying significant differences between samples (e.g., comparing if there were differences between a student’s career goal and the skills they perceive 67 as necessary for success in their future career). The perspective on student awareness of EDCs and how they may associate lab activities with building EDCs provided by this study can potentially inform the design of laboratory curricula to explicitly promote development of EDCs to begin addressing the skills gap. Research Questions This mixed-methods study, carried out in two project-based general chemistry laboratory courses, sought to address the following research questions: 1. (RQ1) What skills do students believe are needed for their career goal, and how do these student-perceived competencies (SPCs) align with employer-desired competencies (EDCs)? 2. (RQ2) How do students relate course components and activities in a project-based general chemistry laboratory learning environment to skills perceived as necessary for their planned career? Theoretical Framework This study is guided by social constructivism, a theory of learning proposed by Vygotsky which posits that learners actively build knowledge through interactions with their social environments (Vygotsky, 1978). As a result, learning cannot be separated from the social interactions and influences that are part of the environment in which learning takes place. Construction of knowledge in this paradigm requires active participation by learners rather than passive transfer of information. Learners also draw upon prior experiences to make meaning. Development of job-related skills or competencies based on a social constructivist framework has been used previously to guide the design of employee training programs. Kraiger proposed that social constructivism is central to workplace training, stating the “knowledge and skill base required to perform our jobs is socially negotiated” (Kraiger, 2008). Cooper et al. studied the impact of a small-scale training program developed using the principles of social constructivism and found from pre- and post-assessment that the experimental group showed gains in the soft skills of creativity, adaptability, and self-acceptance/confidence relative to the control group (Cooper et al., 2006). Active construction of understanding and social interaction are two of the pillars of project-based learning (Krajcik & Blumenfeld, 2006). The project-based general chemistry laboratory courses in this study create an environment where students are encouraged to “do 68 what scientists do” through actively engaging in scientific practices while interacting with the learning community. This community includes multiple teams of 3-4 students and graduate teaching assistants, where each member of the community can provide guidance and support to aid students’ learning and development throughout the course (Carmel et al., 2019; Krajcik & Blumenfeld, 2006). The centrality of social interactions to the collaborative learning environment found in the general chemistry laboratory courses in this study influenced the adoption of a social constructivist framework as a lens for characterizing student experiences and perceptions. Methods Methodological Framework This study used a mixed-methods approach, incorporating qualitative and quantitative elements (Plano Clark & Ivankova, 2018), to explore EDC development in project-based general chemistry laboratory courses. The focus of this chapter will be on the qualitative methods employed, using transcendental phenomenology to guide this area of research, accompanied by statistical methods to aid in interpretation of comparisons between samples. The primary aim of transcendental phenomenological research is to explore a particular phenomenon (e.g., EDC development) through providing rich and nuanced descriptions of “what” and “how” individual participants experienced a phenomenon (Creswell, 2013). Moustakas and Creswell define transcendental phenomenology as a methodological approach where researchers engage in the central philosophical tenet of “epoché” by addressing preconceived judgements or experiences to be set aside prior to investigating the phenomenon in order to see the true essence of what is, a concept proposed by the founder of phenomenology Edmund Husserl (Creswell, 2013; Moerer-Urdahl et al., 2004; Moustakas, 1994). Individuals who experience the chosen phenomenon are selected and data is collected, commonly through interviews. Themes from transcripts are identified and defined from excerpts and then presented by describing what was experienced with contextual descriptions of how something was experienced (Eddles-Hirsch, 2015; Moerer-Urdahl et al., 2004; Moustakas, 1994). Although acknowledging researcher biases via epoché was not initially integral to this research, the importance of addressing the primary author’s (BE) personal experiences with the courses and consciously setting aside any preconceived ideas to allow themes to emerge from the experiences of participants became evident as the study progressed. 69 Researcher Bias Statement Important to every study is acknowledgment of an author’s positioning in the research process and any potential for bias this may introduce. Author BE, who led data collection and analysis, held instructional positions in the two courses studied. Prior to and during data collection, the primary author (BE) was a graduate teaching assistant (GTA) for both general chemistry laboratory courses over 4 semesters. Additionally, author BE served as a senior GTA for 1 semester following data collection. As a senior GTA, BE supervised undergraduate learning assistants and assisted the laboratory coordinators with GTA training. These experiences give author BE a unique insider perspective but could also potentially introduce bias in collection, synthesis, and interpretation of data. Steps were taken to mitigate against the potential for bias associated with author BE’s involvement in these courses. Interviews were conducted by three different researchers, including the primary author BE. No interview participant was a student of the primary author. Interview transcripts were deidentified prior to analysis. In addition to BE being an instructor, a second author (Priya Patterson-Lee - PP-L) was an undergraduate student enrolled in these courses while analysis was underway for this study. Author PP-L served as the second coder, and it was imperative during coding meetings to address and recognize any potential biases and let the data and participant experiences speak for themselves. An additional second author (Lynmarie A. Posey - LAP) on this study was not directly involved in the courses studied and participated in analysis and interpretation of interviews and refinement of the coding scheme. Acknowledgment of all authors positionally in the study is intended to provide transparency to readers. Course Contexts This study was conducted in general chemistry laboratory 1 (GCL1) and general chemistry laboratory 2 (GCL2) courses at a large Midwestern university. These general chemistry laboratory courses follow a cooperative, project-based curriculum (Cooper, 2012; Sandi-Urena, Cooper, & Stevens, 2011; Sandi-Urena et al., 2012). Teams of 3 or 4 students work together, leveraging the benefits of constructing knowledge in a social environment often going beyond what they can achieve individually, to solve a problem or answer a scientific question based on a project scenario (e.g., reproduce the color profile of a name brand beverage for a generic soda company or conduct experiments using artificial kidney stones to formulate a recommendation on the best way to dissolve and prevent kidney stones). Graduate teaching assistants serve as 70 mentors. Time is allotted during lab for teams to collaborate on planning documents consisting of a series of scaffolded questions. These questions are used to guide teams in making decisions about the experiments to perform and data to collect by prompting students to design experimental procedures and divide tasks among team members. The following week, students perform planned experiments, share data with team members, collaboratively analyze data, and report preliminary findings in their laboratory notebook. The multi-week format for projects gives student teams the flexibility to make mistakes, adapt their plan after collecting some data, or even explore a different approach. At the conclusion of each project, students synthesize their findings and construct arguments using experimental evidence to support their claims. They communicate their results in formats that include oral presentations, poster presentations, and written reports. Although the origins of project-based learning predate A Framework for K-12 Science Education, the cooperative project-based laboratory curricula in these courses embed many of the scientific practices in laboratory activities and assessments (Carmel et al., 2019; National Research Council & Committee on a Conceptual Framework for New K-12 Science Education Standards, 2012). Learning outcomes provided to students in the laboratory manual, that each student is required to obtain, outline the scientific practices that are most prevalent to the laboratory activities (e.g., designing experiments, analyzing and interpreting data, and constructing explanations and engaging in argumentation), alongside outcomes that could be related to EDCs such as collaboration and communication (Cooper et al., 2023). The course structure for GCL1 and GCL2 is identical; however, the complexity of projects increases in GLC2. Students are not required to take the labs concurrently with the corresponding general chemistry lecture course. Enrollment in these courses consists of predominantly of first- and second-year STEM students. Total enrollments in GCL1 and GCL2 during this study were 2,627 and 1,103, respectively (breakdown by semester provided in Appendix A.IV.1). Students in most engineering majors only take GLC1, and enrollment in GCL2 is dominated by students pursuing degrees in the biological sciences. Under 4% of the students enrolled in these courses during this study were chemistry majors (Appendix A.IV.2). This study was conducted in Spring 2020, Summer 2020, and Fall 2020. Due to the COVID-19 pandemic, these courses transitioned to emergency remote instruction in Spring 2020 and were conducted entirely online in Summer 2020 and Fall 2020. Exploring the influence of online 71 learning on student experiences in developing EDCs is outside the scope of this chapter but will be explored further in Chapter VI. Data Collection This study was determined to be exempt after review of study protocols by the institution’s IRB office. During three consecutive semesters, Spring 2020 (Sp20), Summer 2020 (Su20), and Fall 2020 (Fa20), students enrolled in GCL1 and GCL2 were invited to participate in remote interviews. Extra credit was offered as an incentive for participation. Surveys were administered as an alternative extra credit opportunity for students who declined to participate, were unable to participate, or were not selected for interviews; however, these data will not be examined in this chapter. Students were not eligible to participate in both a survey and interview during the same semester and were only permitted to participate in the interview process once for the duration of this study. Selection of Participants Interview participants were selected using a purposeful sampling method (Mathison, 2005) based on declared major in order to represent experiences from a diverse sample of majors and planned careers as described in Appendix A.IV.3. Interviews were conducted remotely on the Zoom video conference platform, with interviews lasting from 16 to 63 minutes and having an average length of 44 minutes. A total of 54 participants were interviewed (Sp20 n = 24, Su20 n = 12, Fa20 n = 18). One interview from Sp20 was excluded from the data set because a poor internet connection degraded the quality of the audio recording resulting in significant gaps in the transcript. In addition, the participant stated that they were attending university to pursue personal interests rather than to obtain a degree related to a career goal. This resulted in a final sample size of 53 participants. The interview sample was compared to the course population for each semester using SAT composite scores. No significant differences were found (Appendix A.IV.4). Participant Demographics Demographic data for the 53 participants whose interviews were analyzed was obtained from a student records data request. No demographic data were available for one Su20 participant; as a result, this individual is not represented in the characterization of the interview sample that follows, except for course enrollment. Across the three semesters, the sample consisted of roughly equal numbers of participants from GCL1 (n = 27, 51 %) and GCL2 (n = 26, 49%) who 72 were primarily freshman and sophomores (n = 39, 75%). The mean age of participants was 18.9. The breakdown by legal sex was female (n = 30, 57%) and male (n = 22, 42%), which may not accurately capture participants’ identities. Approximately a fifth of the sample self-reported as first generation (n = 10, 19%) with the balance assumed to be continuing generation (n = 42, 79%). The distribution by race and ethnicity across the three semesters was Asian (n = 1, 1.9%), Black or African American (n = 7, 13.2%), Hispanic/Latinx (n = 6, 11.3%), International (n = 3, 5.7%), Two or More Races (n = 1, 1.9%), White (n = 28, 52.8%), and not reported (n = 3, 5.7%). Additional demographic information, including a breakdown by semester is provided in Appendix A.IV.5. Interview Protocol The interview protocols used in this study, which were refined over the three semesters, are provided in Appendix (A.IV.6, A.IV.7, and A.IV.8). Participants were provided with a brief overview of the study and informed of their rights at the start of the interview. Interviewers verified that participants were 18 years or older and obtained signed consent. Interviews followed a semi-structured format using predetermined questions but allowing interviewers to follow up on comments from participants for clarification or to elicit additional information (Herrington & Daubenmire, 2014). The interview protocol was divided into unprompted and prompted sections. In the initial unprompted segment, students were asked to first identify skills they thought would be valuable for their planned career goal. Subsequently, they were asked about development of these skills through experiences within and outside the general chemistry laboratory courses. In the prompted portion of the interview, students were shown a list of skills that employers desire in new hires (Appendix A.IV.9) and were asked to identify skills necessary for their career goal and discuss any experiences in their general chemistry laboratory course that may have contributed to skill development. Qualitative Analysis of Interviews Interview recordings and transcripts were assigned unique identification numbers and associated aliases prior to analysis. Nonbinary aliases were assigned intentionally to avoid indicating participant gender. Audio recordings of interviews were transcribed verbatim using a web-based machine transcription service (OTTER.ai). The researchers reviewed the machine-generated transcripts and corrected errors before dividing the transcripts into unprompted and prompted subsections based on the interview protocol design prior to analysis. 73 Inductive thematic analysis was utilized to identify emergent themes for four main areas of inquiry; a) anticipated career goal, b) prior experience and exposure to career, c) skills needed to be successful in planned career, and d) experiences associated with development of these skills within the courses (Nowell et al., 2017). Researchers began by getting acquainted with the interview transcripts and generating initial codes that were then organized into broader themes to represent a preliminary coding scheme. Coding was carried out using MAXQDA (VERBI Software). Once the preliminary coding scheme was constructed, two independent coders individually applied the scheme to a set of interviews to determine its reliability. Until inter-rater reliability (IRR) was established, discrepancies were discussed in full at weekly meetings. The coding scheme continued to evolve until complete understanding of the codes was reached and Brennan Prediger’s Kappa (𝜅BP) values of 0.80 or greater were achieved. Landis & Koch interpret 𝜅BP = 0.80–1.0 as near perfect or very good agreement (Landis & Koch, 1977; Mabmud, 2012; Sim & Wright, 2005). After IRR was reached, one independent coder coded the remaining interview transcripts. Qualitative coding of human responses is inherently messy. Development of a coding scheme can require many iterations to establish a structure that can be consistently applied by multiple researchers to identify overall themes, while still capturing the variety and richness of participant responses. Throughout the results from this study, rich descriptions and quotes of student responses are provided with the hope of maintaining complete transparency between the researcher and the reader, while capturing the student voice. Statistical Analysis for Comparisons Between Samples Of interest in this study was determining whether a student’s anticipated career goal or prior experiences in the field of their career goal had a relationship to whether a skill was reported as needed for success in their future career without prompting. Skills that became emergent in participant responses were quantified and displayed using bar charts for initial investigation of differences between samples. To determine if observed visual differences were statistically significant, chi-square tests of association (𝜒2) was run for each individual skill using dichotomous variables to signify if a participant either 1 = reported skill or 0 = did not report skill as needed for their career goal without prompting during the first half of the interview. Subsequent p-values were adjusted using the Benjamini-Hochberg method of adjustment to control for multiple comparisons (Benjamini & Hochberg, 1995) and adjusted values that 74 remained significant, post adjustment, were to be followed by use of Phi or (φ) or Cramer’s V (V) to report effect sizes (Cohen, 1988; Reid, 2022). Findings The results presented will focus on 1) characterization of the skills that students perceived as needed for their planned careers (student-perceived competencies or SPCs), 2) the alignment of student-perceived competencies with employer-desired competencies (EDCs), and 3) students’ perspectives on opportunities to develop these skills in their GCL1 or GCL2 course. Prior to investigating instances of perceived skill development in GCL 1 or GCL 2, student career goals and prior experiences (e.g., internships, shadowing), were qualitatively coded to provide context for coding of skills required for intended career and skills developed in courses. Findings will be introduced by first presenting a general overview of broader themes, followed by exploring specific examples nested within each theme as supporting evidence. Related statistical analyses will be interwoven with corresponding qualitative results. The terms skills and competencies will be used interchangeably. Interview data from students enrolled in GCL1 and GCL2 is reported in aggregate, because while the project scenarios differ, the course structures are the same. Participants in GCL2 who were also previously enrolled in GCL1 were encouraged to expand on and share experiences of skill development within both courses. Co-occurring codes were analyzed to represent overarching themes using MAXQDA software (e.g., instances of perceived development and corresponding course elements related to development of skills). Numbers reported in this study indicate the number of individual interview transcripts or participants where themes presented were observed. Themes reported throughout this study are presented through vignettes of participants discourse on the various topics explored in this study. For clarity, occurrences of “um”, “like”, or pauses have been excluded from participant quotes. Undergraduate Career Goals and Prior Experiences We start by categorizing participants’ planned careers and prior exposure to their planned career to offer context (Table 4.1). Students reported a range of career goals ranging from forensic scientist to optometrist. Three mutually exclusive themes emerged for career goals, Health & Medical Professions, Engineering & Subspecialties, and Other Careers. Descriptions of each career goal and examples of corresponding student responses can be found in Table 4.1. 75 Table 4.1. Classification of participant career goals. Career Goal Health & Medical Professions Engineering & Subspecialties Other Careers Description of Careers Included Doctor (any specialty), nurse, physician assistant, occupational therapist, nurse practitioner, dentist, orthodontist, optometrist, pharmacist, veterinarian. Engineer in any discipline (e.g., electrical, automotive, food engineering, etc.), and computer scientist (e.g., artificial intelligence). Research & development, medical laboratory scientist, forensic scientist, astronaut, and undecided or does not know. n (%) 33 (62) 10 (19) 10 (19) Examples of Participant Responses “[. . .] my goal is to either become a pediatric orthopedic surgeon, or, um, do cosmetic plastic surgery.”(Fenix) “[. . .] right now I plan to go to pharmacy school after undergrad, and therefore become a pharmacist.” (Corey) “[. . .] I hope to be a drilling engineer in an oil company.” (Bailey) “And through robotics, I decided that I wanted to go into more of an AI or machine learning field.”(Jordan) “. . . I definitely want to just do neuroscience research and then maybe just be a professor at a university is what I'm looking at right now.”(Sunny) “Well, interestingly enough, I'm not, I'm not necessarily [. . .] planning to do any kind of direct engineering work [. . .] but my main, my major, huge, big picture, dream goal is to be an astronaut.” (Stevie) The majority (62%) of participants aspired to careers in a medical field and engineering was the second most reported career area (19%). Other careers (19%) encompassed career areas mentioned by only 1 or 2 participants as well as participants without defined career goals. The three career goals will provide a contextual lens as participant perceptions of skills needed for their planned career and development in their general chemistry laboratory course are examined. 76 Participants were asked to discuss prior experiences and exposure related to the field they planned to pursue post-graduation. Many participants (n = 28, 53%) had direct experiences that included volunteering, shadowing, internship, work, and/or research opportunities in their specific field or related occupations. Others explicitly mentioned not yet having a chance to gain hands-on experiences in their chosen field (n = 20, 38%) for reasons including reduced opportunities during the pandemic, just starting their college career, or recent changes in their career goal and/or major. Although not all participants had a chance to engage in hands-on experiences, some had learned about their planned career through conversations with family members and friends working in the field (n = 5, 25%)1, interviews with professionals (n = 4, 20%)1, club activities (n = 3, 15%)1, exposure via personal experiences (n = 3, 15%)1(e.g., interacting with doctors as a patient or conducting research into the field), and career fairs (n = 1, 10%)1. Experiences beyond traditional shadowing or internship experiences were also shared by those who had prior hands-on experience, showcasing how participants had a wide range of experiences contributing to perceptions of what their career goal entails. A small number of participants (n = 5, 9%) were either not asked if they had prior experience in their field due the interviewer skipping this prompt, missing areas of text in the transcript due to technological problems when recording the interview, or because the participant did not elaborate on their experiences. An additional line of inquiry that was explored within interviews was asking students to elucidate what they think day-to-day would look like within their future career goal. This line of inquiry provided further insight the prior knowledge participants had of their planned occupation and returned two main findings – 1) the perception that day-to-day activities will vary depending on specialty or project (n = 6) and 2) uncertainty in what day-to-day activities would look like (n = 9). These findings were shared both by participants who reported having prior hands-on experiences (e.g., internships) as well as those who had not yet had those opportunities. When asked what their day-to-day would look like as a physician, Grey encompassed both themes of being unsure of what a typical day would entail along with believing these activities would change based on what specialty they planned to go into. 1 Percentage represents the number of participants who cited an instance of exposure to career goal out of the n = 20 participants who reported they had no internship/shadowing experience. 77 [. . .] I'm not exactly sure just because [. . .] my goal is medical school, but at the same time, I'm not sure about like a specialty that I would want to go into right now and that changes it up so much. Every doctor has a day-to-day difference even, if even within specialties depending on whether or not you own your own practice, you work in like a hospital and stuff like that, all the day to day is going to be like completely different. So, I'm not exactly sure to be honest. (Grey) These observations show the inexperience of some participants regarding what to expect within their field, but also offered insights into how aware students are that the workplace does not remain consistent and will change according to the profession and/or project they are working on. Through the variety of experiences and thoughts explored above, participants were able to share insights into skills they believed would be of high value for their future career. Skills Perceived as Needed for Future Career In the absence of prompting, students discussed many skills that they believed were necessary for their planned career, which resulted in an expansive set of initial codes for student-perceived competencies (SPCs). Thematic analysis of these codes identified seven overarching career skill sets with defined subcategories (Appendix A.IV.11), which are represented in Figure 4.1. The diameter of the circle for each skill set corresponds to the number of participants (n) who mentioned at least one skill from the main skill set as needed for their career. The seven SPC skill sets are highlighted in bold, with the respective skills listed underneath. 78 Figure 4.1. SPC skill sets (highlighted in bold) identified by participants (n = 53) as required for their career without prompting. The diameter of each circle corresponds to the number of participants who named skills from the skill set. Subcategories are listed if space permits or as follows: Problem-solving & Critical Thinking: data analysis, analytical thinking, creativity & innovation. Prioritization & Time Management: organization, punctuality, multi-tasking, efficiency with time. Education & Learning skills: curiosity & willingness to learn, general knowledge. Other Skills: attention to detail/observation skills, decision making skills. Interpersonal Skills were defined as the skills needed to work and interact with others in the work environment which are reliant on social interactions. Intrapersonal Skills embodied competencies possessed by the individual that contributed to a person’s work ethic and personality traits or characteristics of an individual. Although Problem-solving & Critical Thinking Skills and Prioritization & Time Management Skills have both interpersonal and intrapersonal elements, they were characterized as distinct skill sets in this study. Problem- solving & Critical Thinking Skills encompassed skills related to solving problems with creativity and innovation or using data analysis and interpretation to seek or justify solutions. Prioritization & Time Management contained skills surrounding effective management of tasks and time. Occupation-specific Skills pertained to field-specific knowledge or technical skills required for a specific job or task. Education & Learning Skills included competencies needed to learn, acquire, develop, or obtain general knowledge. The Other Skills category was defined by skills that did not fit into previously defined skill sets and did not occur with sufficient frequency in the data set to warrant a separate category. 79 In the absence of prompting, nearly all participants (n = 48, 91%) identified interpersonal skills as necessary for entering the workforce, followed by intrapersonal skills (n = 31, 58%), occupation-specific skills (n = 27, 51%), problem-solving & critical thinking (n = 19, 36%), and prioritization & time management (n = 17, 32%). Only 9% (n = 5) of participants mentioned education & learning skills. Subskill categories from interpersonal (communication, teamwork, social & people skills, miscellaneous interpersonal skills), intrapersonal (personality & character traits, work ethic), occupation-specific (technical skills, field-specific knowledge, miscellaneous occupation- specific skills), education & learning (curiosity & willingness to learn, acquiring knowledge), and other (working in a diverse environment, navigating across boundaries, miscellaneous other skills) skill categories were disaggregated and compiled alongside problem-solving & critical thinking and prioritization & time management to compare differences in skills reported as valuable career competencies, without prompting, based on students’ career goals or prior experiences (A.IV.12). Comparison of these skills, using chi-square tests, revealed no statistically significant differences to indicate a relationship between either variable and the skills students recognized as valuable to their careers without prompting. During the second half of the interview, participants were shown a visual aid listing EDCs (A.IV.9). They were asked to identify any skills from the list that would be needed for their career and expand upon why skills not discussed previously would be relevant. Figure 4.2 above shows the percentages of students in each semester who identified a skill as important to their career during the interview, either with or without prompting. This figure shows six skills that were prevalent in all semesters: communication skills, teamwork skills, work ethic (primarily self-motivation), problem-solving & critical thinking, prioritization & time management, and technical skills. Although social & people skills were identified by a majority of participants as important career competencies in Sp20 and Su20 semesters, less than 40% of Fa20 participants recognized these skills as necessary leading to exclusion of this skill in further investigation. This noticeable decrease in Fa20 participants reporting social & people skills as relevant to their planned career goals may be due to the lack of inclusion of compassion & empathy skills present in the Sp20 and Su20 skills list that was categorized under social & people skills. The six most frequently identified skills will be our focus in exploring instances of perceived development in laboratory courses. 80 s t n a p i c i t r a P f o t n e c r e P ) % ( r e t s e m e S n i h t i w 100 90 80 70 60 50 40 30 20 10 0 Sp20 Su20 Fa20 Figure 4.2. Percentages of interview participants who identified competencies as valuable for their career goal before and/or after prompting reported by semester (%). For Fall 2020 data, an * signifies skills that were not included in the skills list prompt, and the percentages reported correspond to mentions without prompting only. An additional line of inquiry added to Fa20 interviews was asking participants to rank skills in order of importance to their career goal, with 1 representing the highest value (A.IV.13). The most prevalent finding within this line of questioning was that participant’s believed technical skills to be of least importance. While a small set of Fa20 participants (n = 4) were not included in analysis for ranking of skills due to time constraints or this line of inquiry being absent from discussion, most participants (69%, n = 9)2 ranked technical skills as fifth or sixth on the spectrum of importance in comparison to the other five EDCs (A.IV.9). Giving participants a reason to expand on why they ranked skills in such a manner, participants provided insights into 2 Percentage is taken from a sample of n = 13 participants. One participant did not report a ranking for technical skills. 81 the trends explored above. Some interview participants (n = 3) spoke of ranking technical skills last due to the skill being perceived to be learned or taught on the job as reflected in Pax’s comment below. [. . .] I'm kind of leaning towards putting technical skills near the bottom, because I feel like regardless, anyone can just be taught how to work a machine [. . .] (Pax, Technical Skills) Others (n = 2) who ranked technical skills lowest reported these skills to be a) something their career can do without (referring to use of technical skills in the context of technology), and b) acquiring these skills are to be done prior to obtaining a job in their chosen career goal alluding to skills that are already obtained through schooling should become second nature and any gaps can be learned through pursuing more education, leading to these skills being perceived as less important in comparison to soft skills. For one interview participant who did not feel technical skills ranked among the rest, they reported that they felt these skills do not require focus to learn, leading to these skills being excluded from ranking. Examining the remaining skills, communication skills were ranked the highest, with 79% (n = 11) of participants assigning this skill a value in first or second place and teamwork was ranked primarily in second place by 43% (n = 6) of participants. No discernable trends were observed for prioritization & time management or self-motivation (also referred to as work ethic) skills. As shown in Table 4.2, the student-perceived competencies (SPCs) that emerged from the interviews (both not-prompted and prompted) overlap significantly with the core competencies required for success in the 21st-century workplace identified by several organizations and authors (Kondo & Fair, 2017; National Association of Colleges and Employers, 2021, 2023; National Research Council et al., 2011; Organisation for Economic Co- operation and Development, 2019; World Economic Forum & in collaboration with The Boston Consulting Group, 2015). Within and across these employer-desired competencies (EDCs) is the strong presence of interpersonal skills (e.g, teamwork, communication, social/cultural awareness, empathy, leadership), problem-solving & critical thinking (e.g., analytical skills, creativity, systems thinking), technical skills (e.g, information/ communications technology, computer skills), intrapersonal skills (e.g., work ethic, persistence/grit, responsibility, adaptability, self- 82 efficacy), and education & learning skills (e.g., curiosity/learning to learn, self development), skills that are also clearly evident in participant reflection of skills needed prior to and with prompting. Interpersonal skills, such as teamwork and communication, and cognitive skills that include problem-solving & critical thinking are just as prevalent in student responses as they are in employer reports and academic studies. Additionally, when ranking EDCs, students were able to recognize value of soft skills over hard technical skills. Table 4.2. Comparison of student-perceived competencies (SPCs) with employer-desired competencies (EDCs). Listed based on prevalence: Student-perceived Competencies (SPCs) from Study 1. Interpersonal Skills (e.g., teamwork & collaboration, communication skills, social & people skills, leadership, networking) 4. Problem-solving & Critical Thinking (e.g., creativity & innovation, analytical thinking – data analysis) 2. Intrapersonal Skills (e.g., work ethic - motivation, focus, determination, ability to work under pressure; personality & characteristic traits - patience, flexibility & adaptability, brave, open-minded) 3. Occupation-specific Skills (e.g., technical skills, field specific knowledge, training in field) 5. Prioritization & Time Management (e.g., organization, punctuality, multi-tasking, efficiency with time) 6. Education & Learning Skills (e.g., curiosity & willingness to learn, general knowledge) 7. Other Skills (e.g., attention to detail/observation skills, decision making skills) Kondo & Fair’s Skills Desired in Chemical Industry Listed based on importance: Interpersonal skills Teamwork Strong work ethic Problem-solving skills Initiative 1. 2. 3. 4. 5. 6. Analytical/qualitative skills Technical Skills Oral communication Written communication 7. 8. 9. 10. Organizational skills 11. Computer skills 12. Leadership Employer-desired Competencies (EDCs) World Economic Forum’s 21st Century Skills National Research Councils 21st Century Skills Listed in no order: • Foundational Literacies (e.g., literacy in science, numeracy, information/communications technology, financial, cultural & civic) • Competencies (e.g., critical thinking & problem-solving, creativity, communication, & collaboration) • Character Qualities (e.g., curiosity, initiative, persistence/grit, adaptability, leadership, social cultural awareness) Listed in no order: • Cognitive skills (e.g., non-routine problem-solving, critical thinking, & systems thinking) Interpersonal skills (e.g., complex communication, social skills, teamwork, cultural sensitivity, dealing with diversity) Intrapersonal skills (e.g., self management, time • • 83 management, self development, self regulation, adaptability, executive functioning) Table 4.2 (cont’d) Organisation for Economic & Cooperative Development’s Skills for 2030 National Association of Colleges & Employers (NACE) 2023 Core Competencies* Listed based on importance: 1. Communication 2. Critical Thinking 3. Teamwork 4. Equity & Inclusion 5. Professionalism 6. Technology 7. Career & Self Development 8. Leadership Listed in no order: • Cognitive & Metacognitive Skills (e.g., critical thinking, creativity, learning to learn, & self regulation) • Social & Emotional Skills (e.g., empathy, self efficacy, responsibility,& collaboration) • Practical & Physical Skills (e.g., using new information & communication technology devices) * NACE’s 2022 list of core competencies had a notable switch of critical thinking being the top rated career competency and communication being the second more important. The remaining skills were rated in the same order. †All competencies listed under EDCs represent exact wording extracted from the associated resources. Over 80% of the students interviewed for this study were freshmen or sophomores. These results demonstrate that even in the early stages of their undergraduate studies, students are aware of the skills that are needed to be successful in today’s workforce in the absence of any prompting, a finding also observed in other studies (Hill et al., 2019; Ntola et al., 2024; Yasin & Yueying, 2017). These findings suggest that the perceived skills gap is not simply a matter of students’ lack of awareness, and furthermore, opportunities may exist to leverage students’ existing awareness to build EDCs within undergraduate curricula. Additionally, comparisons were performed to determine if students more frequently mentioned the top six skills categories with or without prompting (Appendix A.IV.14). These comparisons were made without use of statistical testing and filtering for students who recognized a skill category as important and further disaggregating results into 1) those who mentioned a skill without prompting and 2) the additional participants who recognized a skill post-prompting, showed that a majority of participants recognized work ethic, problem-solving & critical thinking, prioritization & time management, and technical skills as valuable to their career goal after being introduced to a skills list. These observations suggest that while students may not identify some of these skills as important to their career on their own, they do recognize specific EDCs as relevant when given a list, pointing to the importance of drawing students’ 84 attention to the skills needed in the 21st-century workforce. Building awareness is a necessary first step in helping students connect their college experiences to EDC development. Based on these results, several possible explanations exist for the gap reported by employers between desired competencies and the skills recent college graduates bring to the workplace. These include 1) students have not had opportunities to develop EDCs in college, 2) students do not recognize experiences as helping them develop EDCs, 3) students cannot articulate how they developed EDCs despite opportunities to do so, or 4) students cannot translate these experiences to the workplace. The next section will provide an overview of students’ perceptions of the relationship between experiences in the introductory general chemistry laboratory courses (GCL1 and GCL2) and skill development. Course Components Contributing to SPC Development Codes exploring perceived development were generated through the focus of skills that were both reported as being needed for participants’ future career goal (SPCs) and perceived as developed in the course. If a participant discussed development of a skill in the course but did not relate the skill to their career goal, coding did not occur. Students had varying perceptions of skill development within the laboratory courses and reported instances in which the course provided support or lacked opportunities to develop skills, as represented in Figure 4.3. Some participants may have perceived aspects of the course as contributing to development, while seeing the online modality as negatively impacting development. For this sample and the purposes of this chapter, those who mentioned online learning as a hinderance to development, while also mentioning instances of the course supporting development, are only presented in the number of students citing the course as supporting development. A small subset of participants may be represented as both citing instances of the course supporting development as well as viewing the course as lacking opportunities for the same skill. Because of this a single participant could have mentioned development of one competency (e.g., self-motivation) and at the same time report no opportunities for development of another (e.g., focus) within the same skill category (e.g., work ethic), meaning that this participant could be represented under both development and lack of development. This occurred for work ethic (n = 5), problem-solving & critical thinking skills (n = 1), and technical skills (n = 3). 85 Communication Skills Teamwork Skills 2 1 Work Ethic 11 19 Problem-solving & Critical Thinking Prioritization & Time Management 5 4 26 Technical Skills 14 13 36 37 40 Course Lacked Opportunities for Development Course Supported Development Figure 4.3. Study participants citing opportunities for development of SPCs in the general chemistry laboratory courses or an absence of such opportunities. If a participant mentioned that they had not yet developed a skill during their college career, it was assumed that this perception included the general chemistry laboratory courses, and as such, they were included in the group of students who perceived the course to be lacking opportunities. Students also could have spoken of not perceiving development in earlier portions of the interview transcripts and revisited the skill to reflect on instances in which the course aided development later on in which they were coded as experiencing development. If instead a participant first mentioned they experienced development and later reflected that the course did not provide opportunities for development, they were coded as the course lacking opportunities for development. Because of the various factors involved in the coding scheme listed above, it is crucial when interpreting perceptions of development to keep in mind certain categories are not mutually exclusive. Instances which are pertinent to results that include a recurring participant citing instances of both the course supporting or lacking opportunities for development for a specific skill will be outlined to provide clarity. An additional aspect to consider is that due to students guiding the conversation or time not permitting, interviewers were unable to follow up on every skill mentioned as important to a participant’s career goal and subsequent development within the context of the course. 86 Consequently, the numbers of students who perceived the course as supporting development or lacking such opportunities may be underreported. Most participants identified opportunities for skill development in these courses (Figure 4.3). Over 80% of participants identified skills from the communication (82%), teamwork (80%), and problem-solving & critical thinking (83%) categories as valuable workplace competencies that they also developed in their course. Prioritization & time management was recognized by over half of the participants (62%), while work ethic (41%) and technical skills (36%) categories were reported less frequently. Only one student reported no opportunities for SPC development. Because of their prevalence, communication, teamwork, problem-solving & critical thinking, and prioritization & time management will be the primary focus when exploring course elements that contributed to development. While many participants viewed the courses as providing instances for building skills (that will be explored later on), not all participants shared this perception; however, for most skills these occurrences were relatively low in comparison to instances in which the courses were thought to support development, with technical skills being the exception. A brief synopsis of themes exploring explicit and implicit reasons why students did not believe the course to provide development for communication, teamwork, work ethic, problem-solving & critical thinking, and prioritization & time management are included in Appendix A.IV.15. In the case of technical skills, participants had polarizing views, with approximately equal numbers of participants perceiving the course as lacking opportunities to develop specific skills and supporting skill development.3 Among the participants who said the course did not support development of technical skills, 64% (n = 9) of them attributed it to the absence of hands-on experiences and opportunities to carry out experiments in an online laboratory course. Many participants who did not perceive development of particular technical skills also talked about technical skills being context-dependent or learned in specific places, such as on the job, during continued schooling (e.g., medical school), or within certain college courses. Examples given included medical laboratory or computer science skills, which are outside the scope of a general chemistry laboratory course. Some participants were able to recognize the job-specific nature of technical 3 Note that a small number of students (n = 3) identified a skill from the technical skills category that they developed while also commenting on another technical skill perceived as important that they did not. 87 skills that is cited throughout the literature (Lamri & Lubart, 2023; National Academies of Sciences et al., 2017). Additionally, one participant noted how development was not only class specific, but they had to have interest in a topic or course in order for technical skill development to occur. Although this was only shared by one participant, seeing value and having interest in material is cited as a motivation for learning (National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, et al., 2018). Even when this cooperative, project-based laboratory curriculum is implemented fully in-person, technical skills are not the primary focus of these courses. In addition to being asked what skills were developed, students were asked to elaborate on how these skills were developed through their general chemistry laboratory experiences. Codes for course components that emerged from inductive analysis are summarized in section A.IV.16 of Appendix. The resulting themes connecting course experiences to skill development were classified as either major themes when over 60% of participants associated a course component with a particular skill set/skill category or minor themes when 20 - 60% of participants connected a course component with a skill set. Major and minor themes will be explored in greater depth using representative participant comments. Three major themes, and associated minor themes, emerged from participants’ association of skill development with course components: 1. Major Theme 1: The collaborative nature of the courses (e.g., working with teammates) contributed to development of communication and teamwork skills. 1.1.Minor theme 1.1: Various course activities aided communication and teamwork development. 1.2.Minor Theme 1.2: Problem-solving & critical thinking along with prioritization & time management was related to collaboration within the course as supporting development. 2. Major Theme 2: Problem-solving & critical thinking skill development was related to the open inquiry learning structure (e.g., students designing and planning experiments and having less guidance). 2.1.Minor Theme 2.1: Other course components that contributed to problem-solving & critical thinking skill development included working with data (e.g., data 88 analysis and interpretation), using course resources (e.g., lab manual), conceptual learning and application (e.g., applying conceptual knowledge), and troubleshooting problems that arise. 2.2.Minor Theme 2.2. Course scenarios provided at the start of each project acted as a tool of encouragement from problem-solving & critical thinking skill development. 3. Major Theme 3: Prioritization & time management development came from students having to independently manage projects, assignments, and tasks. 3.1.Minor Theme 3.1: Formal reports and presentations facilitated the use of prioritization & time management skills. A fourth primary theme surfaced that was independent of the specific skills categories - students discussed professional skill development as a continual learning process that starts prior to, within, and beyond the general chemistry laboratory courses. Tables in A.IV.16 provide additional detail on course element subthemes and the number of participants who cited them. Additionally, to include experiences surrounding development of all six top SPCs, course elements reported by students to aid building self-motivation and technical skills are explored in A.IV.17. It is necessary to acknowledge that because this study was conducted during the COVID pandemic, students’ perceptions of opportunities to develop SPCs were impacted by the delivery of a laboratory course being partially or fully online. Although it is impossible to fully disentangle the online modality from the themes presented herein, this chapter will only focus on themes surrounding course components integral to the course design that are considered independent of modality. The impact of the online experience will be explored in a separate chapter. Major Theme 1. The collaborative nature of the courses contributed to development of communication and teamwork skills. The collaborative environment was most frequently reported as supporting development of communication and teamwork skills from the interpersonal skills category. Within the project- based learning environment students are encouraged to collaborate with those around them. This includes interacting with teammates, neighboring teams, and their teaching assistant. From these 89 opportunities for collaboration, students interviewed predominantly discussed how interactions within their semester-long teams contributed to communication and teamwork skill development. These interactions included delegation and coordination of tasks, learning to work with others, and listening and sharing ideas. Projects in GCL 1 and GCL 2 are intentionally designed so that they cannot be successfully completed by a single student in the time allotted (Carmel et al., 2019), prompting many students to associate delegation and coordination of team roles and tasks with building communication and teamwork skills. Students talked about having to determine which tasks each team member was responsible for during lab, scheduling time outside of the lab period, taking the lead and kickstarting group work, and making sure that everyone worked together as a cohesive unit through communicating what was being done and confirming satisfactory completion of work. An additional aspect of team coordination was ensuring that all team members had the information and resources needed to complete their individual tasks. Morgan and Rowan illustrate how making sure the team is functioning in an efficient manner supported development of these skills. So, certainly teamwork because you know, you're not performing the experiments by yourself and you know, delegating work to say, okay, you do this, you do this, you do this, and together, we're all gonna collectively, you know, do a good job. (Morgan, Teamwork Skills) [. . .] I think that communication is a big one. Because without communication, your team doesn't work as well. And then you eventually you won't do as well. You won't get everything done as efficiently as you'd like to and I think that in [GCL 1] you get, I mean you learn about communication very fast. (Rowan, Communication Skills) Students also perceived learning how to work with others and the act of being a team player as aspects of developing communication and teamwork skills. This was done through learning how to navigate a team environment through a variety of experiences. Participants touched on learning to work with diverse groups of students and provided examples of 90 collaborating with teammates who learned or worked differently or came from different cultural backgrounds as contributing to building communication skills, as exemplified by Blake. [. . .] in University all of your team members come from different backgrounds [. . .] So, being able to communicate and understand and help each other in this [. . .] aspect by doing groups or some, or some way to connect with each other, that really helps in how we develop our communication skills. (Blake, Communication Skills) Participants also cited figuring out the team dynamic by learning and playing to each other’s strengths and weaknesses, engaging in conflict resolution, and overcoming differences of opinion as encouraging skill development. As students articulated that their team’s success carrying out projects depended on the contributions of all team members, they discussed making sure that everyone was contributing to the team, relying on and trusting each other to complete assigned tasks, and picking up the slack for members when needed. Participants also talked about helping each other and being respectful of every team member. Finally, some reflected introspectively on their role in the team as illustrated by Charlie who points to learning when to lead and when to follow. [. . .] that's the one class [GCL 1] where I have to work with other people [. . .] and that's definitely one where I have to work with different people that aren't all exactly like me. So, I have to get used to working with people who don't see things exactly how I do who don't operate at the same speed as I do. But we all have to work together for that final goal and to have a final complete finished project. We have to keep the work equal, we have to rely on each other and that's definitely something that has allowed me to better myself in terms of working more as a team player versus a leader or someone who has to take control over things. (Charlie, Teamwork Skills) Sharing ideas with teammates was another component of collaboration that contributed to development of communication and teamwork skills. Students spoke of how their groups would come together to share their opinions on the activities or tasks they were working on and how 91 they had to listen and actively contribute their own thoughts to the discussion. Shiloh shared their thoughts on how this led to teamwork skill development. [. . .] it improved my skills when, so it's just not me working, it'll be [. . .] four people working on the same thing. So, they'll be giving some ideas, and I'll be, I'll be giving some ideas and we, we need to come up, I mean, choose one particular idea, I need to write down that idea. So, I mean, doing that will really help me, you know, managing something [. . .] (Shiloh, Teamwork Skills) Minor Theme 1.1. Various course activities aided communication and teamwork development. When examining how students associated course activities with SPC development (A.IV.16) it was found the communication and teamwork skill development was minorly associated with planning documents and presentations, while communication also included completion of laboratory reports as encouraging development. Project planning by student teams was facilitated by planning documents, which consisted of a series of scaffolded questions to guide development of a procedure to investigate a question or problem presented in the project scenario. When speaking of communication and teamwork skill development, organization of the team was critical to the process of developing the planning document as well as the final product. I think for every project, really just like getting started on the first procedure sheet. We kind of need someone to step up and kind of see, like, plan out which part like, which people are going to do which part. Yeah, I think these projects all need some sort of solid leadership in order to get the teamwork started. But once it gets started, it's totally fine. (Jordan, Teamwork Skills) Palmer recognized that the need for strong communication skills that extended beyond sharing ideas, while planning an investigation, to the written documentation of the plan. [. . .] With the procedure part of the planning documents you need to, you need to be specific with it ‘cause [. . .], basically the planning document, you gave it to somebody else, somebody else could do the whole experiment. That's the 92 whole purpose of it. So that was one of the big parts of it. (Palmer, Communication Skills) At the conclusion of each project, students present their results and make claims based on experimental evidence. This is done as a team presentation to the class through oral or poster presentations or individually in informal or formal written reports submitted to their GTA. Engaging in the process of completing oral and poster presentations offered a springboard for developing communication and teamwork skills. Students spoke about working with team members to synthesize and practice presentations or having to speak publicly in front of other students to communicate scientific ideas. We recently had an oral presentation, that was very much about team, we practiced the oral presentation. So, yeah, just I think being, relying on each other [. . .] But yeah, almost every week you're doing something [. . .] teamwork related and the oral presentation was just an example of that. (Riley, Teamwork Skills) Preparing reports through seeking help via GTA office hours, conveying information professionally through synthesizing the various lab components into a written report, and displaying information via charts, students were able to advance their communication skills. [. . .] writing skills, professionally making data charts to make it easier for readers and stuff [later in the interview expanding on the same skill] Like kind of condensing your ideas is how I put it. So, you're saying everything you want to in the least amount of words. So, that was a big thing that I had to work on. And doing that like over and over, because it was every week I got the hang of it and it was better. (Sam, Communication Skills) Even though teamwork and communication skills lacked a clear connection between development and project scenarios (A.IV.17), the concept of projects with high difficulties lending itself to development occurred in Lyrics response. Definitely. I feel like I definitely did it during [GCL1], not so much [GCL2]. I think we were doing the spectrometry lab. And that, that lab, it was, it probably 93 was one of the harder labs that I've probably done. So, that's why I really had, had to really rely a lot on my teammates, and really communicate with them about what's going on. And just talk to them some, a lot. (Lyric, Communication Skills, along with Teamwork Skills) Minor Theme 1.2. Problem-solving & critical thinking along with prioritization & time management was related to collaboration within the course(s) as supporting development. Evidence of the richness of the collaborative environment for supporting SPC development beyond communication and teamwork came from connections made by students to problem-solving & critical thinking and prioritization & time management. Among various aspects that will be explored below, participants associated delegation of responsibilities and coordination among team members with developing prioritization & time management and problem-solving & critical thinking skills. Kai offered a perspective on the relationship between prioritization & time management skill development that occurred through assigning tasks and making sure everyone is performing their role. And with some of our labs that we're working on, you definitely want to make sure [. . .] everyone has their assigned task and they're working accordingly. Or else you'll have something left over 10 minutes before it's due and it won't be done quality work. (Kai, Prioritization & Time Management) Often when students related delegating responsibilities to problem-solving & critical thinking skill development, they discussed it through maneuvering as a team around problems that arose and making sure everyone was on the same page. Reign presented an interesting take on problem-solving & critical thinking when a team member was unable to complete an assigned task and the other team members had to adapt to find a solution. [. . .] we were working on a poster presentation for one of our labs. And one of someone in the team wasn't able to make it to one of our sessions [ . . .] And it was kind of important that they would be there because they were bringing like supplies that we needed. So, I guess it was a problem-solving moment where we had to quickly figure out where we would be able to print something out to put on the poster [. . .] . (Reign, Problem-solving & Critical Thinking) 94 Some students discussed exchanging ideas with team members to plan and carry out their project as aiding development of problem-solving & critical thinking. Pax reported that the course structure made them reliant on communication with teammates to solve problems. Problem-solving, definitely, for [GCL1], [. . .] they didn't provide a perfect answer or steps that you needed to follow in order to solve the problem. So it [. . .] involved, communicating with the team to problem solve and figure out a solution to finishing the experiment. (Pax, Problem-solving & Critical Thinking) For another participant, these moments of listening to others gave them a chance to learn, evaluate new information, and incorporate others knowledge into their own as seen through Lev’s experience. It was through brainstorming with teammates that Lev related collaboration in the course as helping them learn problem-solving & critical thinking skills. So, a big thing with us like solving like, "Hey, what's this unknown plastic?" is just getting ideas from other people [. . .] bouncing off ideas from one another [. . . ] So, I think definitely the, that big shift from like high school to college, that laboratory process is a big way of why, the solving problems, I think, the biggest contributor to what I learned from the lab. (Lev, Problem-solving & Critical Thinking) Students discussed prioritization & time management skill development in the context of meeting responsibilities to their team that included being punctual out of respect for teammates, being motivated to contribute time outside of class to complete activities, and learning how to work together and adapt to roles as needed to finish tasks efficiently. For example, Alex spoke about their team relying on them to be on time. Yeah, I've always appear[ed] on time because [. . .] we do work in groups, so if I'm not [on] time, or one of my [. . .] group members, not [on] time, might affect our grade in overall, because we are given all different responsibilities.” (Alex, Prioritization & Time Management) 95 Sam added the element of prioritization & time management to problem-solving & critical thinking when discussing how their team arrived at an approach for measuring density. [. . . ] So, we were finding the mass [density] of a plastic, or we were going to before we got sent home. But we had to come up with different ways to measure them [. . .] And for me personally, I was sitting there stuck, I was like, "Wait, how do you like, measure the mass of this?" like I, or volume, like I don't know. And someone in my group was like, "Oh, you do the water displacement thing," you put in water or some substance and know the volume or mass of that, and [. . .] putting those ideas together, thinking critically and, it helps. Especially when you're [. . .] managing time, because you gotta work fast in a way, but also work smart, but work fast if that makes sense. (Sam, Problem-solving & Critical Thinking, Prioritization & Time Management) Teammates were not the only source of collaboration in the laboratory environment, where having the teaching assistants to guide and answer questions for students was another aspect of the collaborative environment that aided communication and problem-solving & critical thinking skill development. These experiences, recounted by students, capture how the collaborative and social aspect of these project-based general chemistry laboratory courses can promote development of interpersonal skills that are highly valued in today’s workplace. Engaging students in collaborative work that results in development of teamwork skills has been defined as a learning goal in general chemistry laboratory courses (Bruck et al.,2010) and the intentional design of projects that cannot be completed by one person in the time available makes interpersonal skills, such as teamwork and communication, essential to successful completion of projects. Students recognize this, but they also identify other SPCs whose development is supported by the collaborative environment. The benefits of working with others is further supported by the literature, where working in an environment that fosters meaningful collaboration can be associated with the creation of better solutions and problem-solving capabilities (National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, et al., 2018) and an increased sense of belonging (Corwin et al., 2015). The social nature of learning is salient in the 96 examples presented above, where working with others aided not only in development of expected interpersonal skills, but other valuable SPCs such as problem-solving & critical thinking and prioritization & time management as well. Major Theme 2. Open inquiry learning promoted development of problem-solving & critical thinking skills. The open inquiry learning environment, that is integral to the general chemistry laboratory courses in this study, supported development of problem-solving & critical thinking skills by requiring students to design and plan investigations associated with scenarios representing real-world problems. Arbor associated the freedom to design methods, evaluating options before deciding on the approach, assessing the effectiveness of the method chosen, and identifying possible improvements with thinking critically. I would say I feel like given a, being given a scenario, and having the independence to think about any way you deem fit to solve a problem, involve thinking critically. You'd have to assume the positives and negatives of each method that you're going to choose to solve a problem, which I also enjoy to see if the method I devise is the most effective, or what improvements could be made to my method, or to my team's method to make it more effective, get more precise answer. I also think it's rewarding to get the correct way to do it. (Arbor, Problem-solving & Critical Thinking) Within the project-based general chemistry laboratory courses, teaching assistants serve as mentors, who promote student thinking through asking guiding questions without providing direct answers. Some participants commented on the lack of stepwise directions or explicit instruction from teaching assistants, but they acknowledged the value in having to figure out how to the answer the question or solve the problem presented in the project scenario, as exemplified by Lyric and Blaine’s comments. And I believe a good thing with [GCL2] and [GCL1] does is the fact that they let you design the experiments yourself. It's not like a guided experiment, you kind of just go about it in a way that you will want to do it. And I really like the TAs, they, they'll tell you what they want you to do, but then it's up to you 97 after that. It's a lot of, I feel like it's a lot of freedom. I definitely feel a lot of freedom in [GCL1]. (Lyric, Problem-solving & Critical Thinking) I think problem-solving's really, especially 'cause I feel like we're not really given that much direction it's kind of just like, "Here's like the lab scenario," it was kind of do it yourself. So, just really important to be able to think about what you're doing. (Blaine, Problem-solving & Critical Thinking) Another aspect of problem-solving & critical thinking that students touched upon was performing research and finding additional sources of information to support designing their investigations and reporting results in oral and written reports. So, that aspect was actually very challenging for me, and I actually appreciated the experience because in having to create your own procedure kind of forces you to really think about what you're doing. And you have to investigate fully like the topic yourself. (Perry, Problem-solving & Critical Thinking) We had to do a oral presentation on it and I think when I actually had to go more in depth versus not just doing it on a lab notebook, I think physically like, going out of my way to know more information about it, and then not only just reading it, but then having to explain it to a class helped me a lot. (Raiden, Problem-solving & Critical Thinking) Participant responses highlighted awareness of opportunities to engage in development of what was perceived as problem-solving & critical thinking skills. The absence of stepwise instructions for carrying out experiments, which is a key feature of open inquiry (Buck et al., 2008), was central in their discussion of these experiences. This is further captured by a small set of students who contrasted their more traditional laboratory experiences in high school or college courses with the open-inquiry nature of the project-based general chemistry laboratory courses. The aspects explored above of having less guidance, the ability to generate procedures, and having to investigate the topic prior to writing a procedure were perceived as more beneficial in promoting problem-solving & critical thinking 98 skill development than being provided step-by-step instructions. This is reflected in Dylan and Ember’s comments below. [. . .] right now I'm taking a lab at [lists university they are taking a lab course at], and its, they just give you the lab, and you do it on your own. So, I think I like the way State does it better, making up your own lab. (Dylan, Problem- solving & Critical Thinking) At first it was like, kind of difficult, obviously, because I had never been in a lab where we had to design our own experiments. I'd also- always been in those where they just give you all the steps and you don't really have to think about why you're doing each thing, very much. So, it was helpful. But it could also be a little difficult because you have to, you know, do all that work between. But yeah, I mean, it's like a combination of it's harder work, but it's also more rewarding, you know, so, I would say that it's enjoyed, I could enjoy it. (Ember, Problem-solving & Critical Thinking) Building problem-solving & critical thinking skills that are transferrable beyond the contexts in which they are learned is a goal of student-centered inquiry-based learning (Constantinou et al., 2018). Although we cannot report on the transferability of these skills, students were able to articulate and connect their experiences within these courses to the problem-solving & critical thinking skills that were viewed as beneficial for their career goals. Although open inquiry does not provide as much freedom as authentic inquiry, it is more scalable and allows the instructor to have greater control over the learning that is occurring while still providing students with opportunities to develop valuable problem-solving & critical thinking skills. Minor Theme 2.1. Additional course components frequently mentioned as supporting development of problem-solving & critical thinking skills. Minor themes surrounding problem-solving & critical thinking could be found through troubleshooting, learning and applying conceptual knowledge, working with data, and using course resources as supporting development. Conceptual learning and application emerged through participants having to apply prior or newly acquired conceptual knowledge to laboratory 99 activities or gaining new knowledge on topics introduced in the courses. Casey exemplified both aspects of acquiring and applying knowledge when working on both the food dyes UV-visible spectroscopy and identifying unknown plastics labs to develop problem-solving & critical thinking skills. [. . .] so within most of the prompts, again, we're going in with stuff that we know a little bit about, we had a little intro into, but you know, before this class, I'd not really heard of spectroscopy, I knew in general it kind of existed, but never used it. Same thing with the different ways of evaluating plastic, very specific things that you kind of, we learned a little bit about, and then had to use what we, what little we learned to do it. So, definitely, there's some problem-solving and overcoming that knowledge gap of, okay, so we know a little bit about this thing, how can we use the little bit we know in how this thing works to figure out how to solve the problem that's given. (Casey, Problem-solving & Critical Thinking) Working with data encompassed students deciding how to display, analyze, and interpret their data. Stevie offered an example of problem-solving & critical thinking in the context of analyzing and interpreting data collected from the infrared spectrometer and learning how to draw meaning from that data. And sort of being able to make, draw conclusions from the data that we collect. Rather than just seeing it and looking at a bunch of spiky lines, we actually understand intuitively what the meaning of the data is [. . .] (Stevie, Problem- solving & Critical Thinking) Course resources involved use of the course textbook, lab manual, and working with chemical instruments as components related to development. Use of these resources was referenced in relation to problem-solving & critical thinking skill development through utilizing the lab manual to aid students in planning procedures, consulting the provided textbook to find information and fill gaps in knowledge, or working with instruments as observed in Stevie’s response above. 100 Problem-solving and critical thinking skills are often referenced as learning goals of students engaging in the laboratory (Bretz et al., 2013; Bruck et al., 2010; Hofstein & Lunetta, 1982; Reid & Shah, 2007). The major and minor themes presented within this study surrounding instances of problem-solving & critical thinking skill development support findings in the literature. When Danczak et al. asked students how they developed critical thinking skills in their chemistry courses, accompanied by questioning teaching staff on how they provided students with the opportunity to develop critical thinking skills, the aspects of a) designing experiments, b) the inquiry-based learning environment, c) applying and developing knowledge, d) engaging with experimental data, e) critiquing experimental design or data, f) performing research, and g) engaging with the learning community (e.g., other students or instructors) emerged (Danczak et al., 2017). Additionally, Danczak et al. found the laboratory environment to be one of the places identified by students and instructors as conducive to development of this skill. Development of critical thinking skills was also defined as a learning goal by general chemistry laboratory teaching faculty in a study conducted by Bruck and colleagues (Bruck et al., 2010). Faculty believed critical thinking skills to be developed by designing experimental procedures, using evidence to inform decisions, making judgements, and engaging in the problem-solving process. Additionally, some of the emergent themes that students believed contributed to problem- solving & critical thinking skill development, in the study herein, can be tied to the scientific practices as outlined by the National Research Council’s A Framework for K-12 Science Education (National Research Council & Committee on a Conceptual Framework for New K-12 Science Education Standards, 2012). The primary practices that emerged were designing and planning experiments and analyzing and interpreting data. These themes also support Carmel et al.’s findings in which it was reported that the general chemistry laboratory courses investigated in this study contained opportunities for students to engage in the scientific practices (Carmel et al., 2019). Minor Theme 2.2. Course projects aided problem-solving & critical thinking skill development. During interviews, students referred to specific projects from the courses when talking about development. Most frequently, these references were associated with problem-solving & critical thinking (A.IV.16). Participants often associated problem-solving & critical thinking skill development with projects perceived as difficult, complicated, or unfamiliar. As complexity 101 increased, it became necessary to deconstruct the project into manageable parts as explained by Blaine. [. . .] you're given like a huge assignment and you're not really sure what to do. And it's breaking it into smaller parts and thinking about how can you use them. So, I think that's what problem-solving and critical thinking is, it’s like if you're solving a problem of “How do I start this?” then the critical thinking is, “Okay, this is what I have. How can I apply it?” So, I think for sure, [Food Dyes] was one I really had to use that in. Because I think the gas and volume one in the classes were like a lot easier, just ‘cause they were smaller. And yeah, the food dye one was kind of figuring it out yourself. (Blaine, Problem- solving & Critical Thinking) Projects covering an unfamiliar topic, such as making soaps and analyzing the associated wastewater, prompted the use of problem-solving & critical thinking skills discussed by Perry. [. . .] I probably relate to the last project I did [. . .] it was on creating soaps for chemistry and I guess it's something that none of us have done before, for like my teammates and I, I mean, I guess we just kind of like we read through the manual we were given and we asked our TA some questions. And we use scientific concepts to, like draw upon basically a framework for where we would write our procedure and we have to do these planning documents. And those help a lot because we have to answer scientific related questions. [. . .] (Perry, Problem-solving & Critical Thinking - along with Communication Skills) Planning document activities that students engaged with were also seen as a providing instances in which development can occur. The quote from Perry above also associates the process of developing a procedure supported by the planning document with building problem- solving & critical thinking skills. 102 Major Theme 3. Prioritization & time management development came from students having to independently manage projects, assignments, and tasks. Although collaboration is a key component of successfully completing each project, specific tasks are carried out by individuals of the team. When participants reflected upon development of prioritization & time management skills, they commonly focused on their individual responsibilities for completing the work. They discussed learning to be prepared ahead of time for an upcoming class or assignment, splitting assignments into manageable pieces, prioritizing completion of assignments, staying organized, making sure to stay on task, and keeping track of assignment due dates and meeting deadlines. Arbor spoke of the importance of prioritizing tasks that need to be completed during the three hours in which lab meets, while Codi learned how to manage time through the experience of working on an assignment too close to the due date. So, you'd manage your time, which about two and a half, two hours, 50 minutes, 11 to 1:50. So, being able to assign priority, priorities to tasks is really important. Because I think it's easy to get lost in the details and just forget what you're doing [. . .] So, managing priorities is really important. And time, since you don't want to spend too much time on anything. (Arbor, Prioritization & Time Management) But, I feel like managing time also, especially these past couple of weeks, because we've had lab report sections to turn in and the first night, I waited until the night before to finish the introduction. And then I was stressed so then I learned I need to start managing my time, like space it out a little bit and I definitely learned from first experience that I should manage my time better and space it out, so that has been helpful. (Codi, Prioritization & Time Management) Minor Theme 3.1. Formal reports and presentations facilitated the use of prioritization & time Organization and time management were required to compile information collected throughout course projects into a finished product and prepare for presentations. Riley reflected management skills. 103 on the team effort it took to produce their oral presentation, while Reign expanded on keeping organized in order to create either formal reports or oral reports to develop prioritization & time management skills. I actually, I definitely think the labs are very good at, I guess organizing like your, I guess your experiment as a whole. Because at the end we would do either a formal report or an oral report or some type of report that we had to turn in. And that's when the organization really comes in handy, where hopefully, you recorded and you made the proper graphs and tables, and you collected all the data, you needed to fully explain what you did in the lab. (Reign, Prioritization & Time Management) For one project each semester, students prepared a formal report following a journal article format, which is a new experience for most students in GCL1. Students prepared and submitted drafts for individual sections (introduction, methods, results and discussion) over several weeks to get feedback before submitting the final report, which required students to keep track of multiple due dates for this one assignment. Because of this, formal reports were the most frequently discussed activity that contributed to development of prioritization & time management skills. Major Theme 4: Skill development is a continual learning process. A broader theme that emerged from the interviews, which was not specific to the laboratory courses or specific SPCs, was students’ perception of skill development as an ongoing and progressive learning experience. Some students expected to progressively develop SPCs across their college career and noted that their experiences in the laboratory courses pointed to areas for growth. I'd say I have a very basic level. I wouldn't consider myself advanced at all in those skills. I think I definitely have like, a lot more learning left to do. And I'd probably expect my college experience to teach me those skills. (Perry, Social & People Skills, Communication Skills, Problem-solving & Critical Thinking, and Field-specific Knowledge skills) 104 But I mean, I think that's just something that you learn over time, and something you try and improve on [. . .](River, Communication Skills) But it [the lab course] actually has taught me that the time management skills I thought was good, was not good. And I needed to do better which I'm doing better. Now I could do even better. But, you know, I'm working and learning every day. (Oak, Prioritization & Time Management) For a small sample of students, although the course may not be viewed as a source of development for some SPCs, the course was perceived to act as a steppingstone that would provide an indirect but beneficial contribution towards their future career goal by providing the knowledge and experiences needed for advancing to future courses. Participants also mentioned experiences from their lives beyond school and college as contributing to competency growth and development. Some students reported and acknowledged that development will occur beyond their college education and continue even after they reach their intended career. Others recognized that they needed to continue developing their skill sets and expressed a desire to continue learning. [. . .] I think everything leading up to where you're gonna end up is helpful in some way. Because, you know, everything builds upon your past experiences and things like that. So, I think, even if it would just help me in future chemistry classes, for example, which would then help me. You know, what I'm trying to say, everything builds upon itself. (Ember, commenting on no specific skill) Really, I would, I really feel like I would just be constantly consulting with patients, every different day, I really believe I'm, my education would not stop, it would really just be the beginning. So I really feel like I'd just be learning more every day [. . .](Lyric, commenting on no specific skill) Additionally, the top 6 SPCs were perceived by some students as having value extending to various aspects of life beyond their future career goal (Appendix A.IV.18). This included all six SPCs being viewed as important during their college career and campus life, as well as being 105 seen as general life skills needed for everyday living. Additionally, these skills were reported as being valuable across a variety of careers within (e.g., any job in healthcare) and/or outside of (e.g., any career/profession) a participant’s intended career category. These findings showcase how students view EDCs as being applicable and having importance to not only success in their career but various avenues of life, further emphasizing the value of building student awareness of these skills, how they can be applied, and instances in which development or growth can occur. These skills are widely recognized, beyond the participants of this study, as pertinent skills for success in life, learning, and career contexts (nkley et al., 2012; National Research Council et al., 2011; Organisation for Economic Co-operation and Development, 2019; Wurdinger, 2016). Areas Students Desired Development Broadening the scope beyond skills students mentioned as necessary for their career goal, students were asked in what areas they have liked to have had opportunities to develop skills within the courses but were not given a chance to. Interviewers had a slight variation on the way in which this question was asked – some maintained a more general line of questioning referring to skills students would like to continue working on before entering their career or areas they would like to continue to improve in the general chemistry laboratory courses - leading to some participants being excluded from this analysis. This is a limitation to be taken into account when assessing these results. The outcomes of this question, as discussed below, are drawing upon the data presented in A.IV.19. Skills that students wanted to work on the most, that they felt they did not have an opportunity to grow within the general chemistry laboratory courses, were occupation-specific skills (n = 10). Of these skills, most students (n = 8) wanted a chance to develop technical skills, while the remaining students desired field-specific knowledge (n = 2). Technical skills that students wish were included in the courses were - a) lab techniques that were unable to be learned due to a lack of hands-on experimentation that an in-person course would afford, b) a lack of knowledge that was only perceived to be gained through taking other courses and could not be learned in the general chemistry laboratories, c) wanting more background information behind how different areas of lab work (e.g., spectroscopy) prior to “conducting” the labs, and d) working with software (e.g., Microsoft PowerPoint) that was unable to be learned due to a lack of tutorials and resources available that was accompanied by the feeling that being in-person would have been more beneficial in learning these. When speaking of professional/technical 106 skills present on the skills list provided to students, one student separated the two and felt that technical skills of laboratory techniques were lacking due to taking the course being online but also felt they would have liked to learn more about what professional skills are needed for their career path. This is reflected in Indy’s comment below. [. . .] one of the little boxes mentioned professional skills and I feel like we do things that are for a college class but I don't know what's necessary in the professional realm and I feel like maybe if people have a wide variety of career paths that they want to go into it would be hard to merge them, but maybe to prepare us more for that. (Indy, Professional/Technical Skills) For the small number of participants who desired working on field-specific knowledge this included – a) wanting to learn scientific principles and feeling that the course was too focused on experimental aspects and b) when asked about developing physics knowledge within the courses one participant mentioned that there were no opportunities, further elaborating that taking the course online was a detriment to skill development. Other skills students wanted to work on that the course was perceived to be lacking included – a) problem-solving & critical thinking (n = 5), b) the intrapersonal skills of embracing change (n = 2, situated within personality & character traits) and self-motivation (n = 1, situated within work ethic skills), c) the education & learning skill of curiosity (n = 2), d) the interpersonal skills of leadership (n = 1, situated within miscellaneous interpersonal skills) and social & people skills (n = 1), and e) other skills that included having hands-on experiences (n = 2, without referencing technical skills) and working on understanding and comprehension of what is to be done for each project (n = 1). Not all participants felt that the course needed further integration of skills, with some (n = 8) stating that there were no further skills they could think of that they wish would’ve been present within the courses. While technical skills are not a primary learning objective within these courses, participants consistently desire opportunities to develop these skills. This often coincided with the fact that students were removed from the laboratory environment (a finding that is explored in depth in Chapter V). While technical skills may be perceived to benefit more from students being in the classroom, some students offered thoughts on how problem-solving & critical thinking skills could better be incorporated within the courses that could be implemented 107 independent of the course modality (e.g., in-person or remote). Of the n = 5 participants who wanted opportunities to develop problem-solving & critical thinking in the course, n = 3 felt that this would be better done with more guidance and direction given from their GTA from starting a project to generating lab procedures and interpreting data. For Zuri, this desire for more guidance was reminiscent of a traditional lab. So, I feel like the TA's probably could do like a better job of explaining in the beginning what the lab outcome [. . .] should be, and how you should interpret it, like what factors and properties you should look at and to think what does this mean and why does it relate to the lab. (Zuri, Problem-solving & Critical Thinking) Contrasting this, one participant found that the online labs lacked the freedom to determine one’s own decisions during experimentation, leading to the feeling that this skill was unable to be obtained from the course. While the open-inquiry structure of these project-based courses aim to provide students with less structure (e.g., giving students freedom to generate procedures versus being provided procedures), it was observed that a small number of participants believed that having more assistance could be beneficial in growth of problem-solving & critical thinking skills. However, this same lack of assistance was also seen as a beneficial opportunity for many students in developing problem-solving & critical thinking skills. This may provide an indication that some students benefit from varying levels of assistant from instructors. Implications While students are unlikely to fully develop the skills needed for success in their career within one course, courses such as project-based general chemistry laboratory courses can contribute to students’ career preparation and skill development. In addition, introductory STEM courses can set the stage for retention of STEM majors depending on students’ perceptions of their relevance to their planned major or career (Chen, 2013; Meaders et al., 2020; Seymour & Hunter, 2019). Providing opportunities to develop EDCs in introductory courses can maintain interest and demonstrate relevancy to students’ career goals. A project-based general chemistry laboratory course (Carmel et al., 2019) modeled on the Cooperative Chemistry project-based curriculum (Cooper, 1990) that implements an open inquiry learning approach and is centered on scientific 108 practices can potentially offer a more inclusive opportunity for large numbers of STEM students to begin developing EDCs early in their undergraduate careers. Ideally, as students proceed through their academic and subsequent professional careers, they can build upon these experiences and continue to develop and fine tune the skills needed to enter and maintain success in the workforce. However, in order for ECD development to be purposefully included within students’ college careers, careful consideration of how these skills are integrated into course curriculum is needed. This can be done by using evidence-based research practices, such as this study, to influence curriculum design. We offer additional suggestions for how EDC development can be introduced both within introductory courses and across a student’s college career. First, building awareness of skills early in a students’ college career through introducing the concept of EDCs and how they can be developed or applied within the context of a course may aid students in recognizing key competencies and focusing on instances in which growth can occur. Additionally, providing explicit examples of how a skill is prevalent to a students anticipated career goal through experiences relayed by professionals in their field may further enhance students’ perceptions of the importance of these skills. Although providing examples for all possible career goals within a course is not plausible, investigating the student characteristic of planned careers and using the most prevalent categories can potentially aid in this task. Second, students could greatly benefit by being given opportunities to reflect and articulate instances of perceived development. Being able to clearly demonstrate application of EDCs to potential employers is a key aspect of the interview process and gaining entry into the workforce. As such, students should be given ample opportunities to reflect on how they have gained EDCs throughout their college career and within course curriculum by using available tools such as e-portfolios. However, integrating additional opportunities for professional development of teaching assistants or instructors to assess student performance in EDCs may also be beneficial (Chadwick et al., 2018), as students have been shown to lack the ability to identify deficiencies or places they need work (Ntola et al., 2024).Third, we acknowledge that integrating additional activities beyond previously set course curriculum can be an arduous task on top of the many responsibilities of instructors. By integrating short questionnaires in the form of extra credit activities, educators can offer a small but rewarding introduction to EDCs and encourage awareness of development within their course. Building student awareness of EDCs 109 through outlining how skills can be developed and applied, situating the skills within relevant career contexts that can further motivate learning, and allowing for instances of reflection are just some of many aspects in a model proposed to aid in skill transfer (Jackson, 2016). Conclusions & Future Directions The 53 students interviewed for this study were able to identify skill sets important to their planned career without prompting that aligned with competencies desired by employers and valued in today’s workforce. In addition, they provided specific instances of experiences in project-based general chemistry laboratory courses that supported development of employer desired competencies. The above results show how rich experiences that involved various components of the courses coalesced to build these skills. Although providing opportunities for development of EDCs are not an explicit focus or objective of these courses, the structure of the introductory project-based laboratory courses provided multiple opportunities for development of interpersonal, intrapersonal, problem-solving & critical thinking, and prioritization & time management skills as reported by students. Situated in the context of a course that caters to mostly non-chemistry majors who will not go on to jobs in the chemical industry, findings of perceived EDC development are an important distinction. Students also recognized that development of the skills necessary to succeed in the workplace will be a continual learning process extending beyond their academic career. Developing and reflecting upon instances of development early in a student’s college career may be beneficial for preparing them for the 21st-century workforce. By providing students with opportunities to recognize and become aware of development early on, students could continue to build upon their skills sets throughout their college career. This in turn could potentially allow students to continually articulate and refine how they have used and applied skills, with the hope of ultimately providing students with better career preparation for their future careers. The actual transferability of EDCs across contexts remains unclear. These skills are often ill-defined and hard to measure with literature lacking evidence to support transference from educational to career contexts (National Research Council et al., 2011). Additionally, because EDCs fall outside of the confines of content knowledge, they are often not the focus in higher education curricula and are considered secondary aspects of student learning. Assessment of EDC development and whether these skills transfer across contexts were not the goals of this 110 study. We do not claim to provide evidence for development of specific skills or transference. Instead, we aimed to provide insight into how students perceive development of these skills in reference to their career goals. This study contributes rich accounts from a student perspective on their experiences in a project-based general chemistry laboratory and the skills that they think they have developed. Further studies must be conducted to determine the best method for measurement of each skill, though the downfalls of this and current measurement methods/tools can be found thoroughly defined by the NRC report (National Research Council et al., 2011). Additionally, students will have a multitude of experiences prior to and after taking these courses, both in formal academic settings and outside, that will contribute to advancement of these skills. Exploring cases in which graduates draw upon examples to demonstrate skills when entering the workforce through longitudinal studies may offer insight into instances that had the greatest impact on perceived development of EDCs and the relationship to career success. Limitations This study, based on semi-structured interviews of a sample of students from two large- enrollment general chemistry laboratory courses, was designed to capture students’ perceptions of EDCs needed for their planned career and possible connections between the experiences in the laboratory and EDC development. As such, the findings of this study cannot be generalized to the general chemistry laboratory student population at the institution sampled. Further, this study does not make claims about EDC development and career preparation in other project-based general chemistry laboratory courses, although it does indicate that this potential exists and identifies course components that students associate with EDC development. Additionally, we do not aim to provide evidence that a single college course can or should be the sole venue for EDC development but rather that an introductory laboratory course taken early in students’ undergraduate studies can provide a platform on which to continue building and enhancing skills needed to be successful in today’s workforce. The meanings participants ascribe to these skills may differ and due to the often broad definitions surrounding these skills it is possible that categorization of skills into the themes generated by this study’s researchers is not reflective of how a student would categorize a skill. It must be acknowledged that the COVID-19 pandemic created unique circumstances during the semesters when these interviews were conducted, which unavoidably impacted student experiences in the general chemistry laboratory courses and their perceptions of these 111 experiences. In two of the three semesters students planned investigations and analyzed and interpreted data but had no direct hands-on experience with carrying out investigations and collecting data. The absence of this hands-on experience may have influenced which career- related skills students chose to discuss without prompting. Although the impact of the pandemic and the necessary adjustments to instruction are not explicitly examined in the current study, they will be explored in a separate chapter. Prior to and throughout the process of conducting interviews, survey administration occurred. Because of this a sample of interview participants (n = 11) had completed an online survey on EDC development in the general chemistry laboratory courses in a previous semester. We acknowledge that priming due to prior engagement with a related survey could influence responses. However, providing students with opportunities to continually reflect on development of EDCs could also be beneficial in helping students to not only consider their experiences in a particular course but also the growth of skills as they move through their college career. 112 REFERENCES American Chemical Society. (2015). ACS Guidelines and Evaluation for Bachelor’s Degree Programs. https://www.acs.org/education/policies/acs-approval-program/guidelines.html American Chemical Society. (2023). 2023 ACS Guidelines for Undergraduate Chemistry Programs: Approved. https://www.acs.org/education/policies/acs-approval- program/guidelines.html American Society for Training & Development. (2012). Bridging the Skills Gap: Help Wanted, Skills Lacking: Why the Mismatch in Today’s Economy? American Society for Training and Development. https://www.nist.gov/system/files/documents/mep/Bridging-the-Skills- Gap_2012.pdf Association for Talent and Development. (2022). Bridging the Skills Gap: Workforce Development in Changing Times. https://www.gpstrategies.com/wp- content/uploads/2022/01/192109-GP_ATD-Bridging-the-Skills-Gap-RR-Final-x.pdf Bell, S. (2010). Project-Based Learning for the 21st Century: Skills for the Future. The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 83(2), 39–43. https://doi.org/10.1080/00098650903505415 Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society, 57(1), 289–300. Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining Twenty-first Century Skills. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and Teaching of 21st Century Skills (pp. 17–66). Springer. https://doi.org/DOI 10.1007/978-94-007-2324-5 Bretz, S. L. (2019). Evidence for the Importance of Laboratory Courses. Journal of Chemical Education, 96(2), 193–195. https://doi.org/10.1021/acs.jchemed.8b00874 Bretz, S. L., Fay, M., Bruck, L. B., & Towns, M. H. (2013). What Faculty Interviews Reveal About Meaningful Learning in the Undergraduate Chemistry Laboratory. Journal of Chemical Education, 90(3), 281–288. https://doi.org/10.1021/ed300384r Bruck, L. B., Towns, M., & Bretz, S. L. (2010). Faculty Perspectives of Undergraduate Chemistry Laboratory: Goals and Obstacles to Success. Journal of Chemical Education, 87(12), 1416–1424. https://doi.org/10.1021/ed900002d Buck, L. B., Lowery Bretz, S., & Towns, M. H. (2008). Characterizing the Level of Inquiry in the Undergraduate Laboratory. Journal of College Science Teaching, 38(1), 52–58. Burner, T., Supinkski, L., Zhu, S., Robinson, S., & Supinkski, C. (2019). The Global Skills Shortage: Bridging the Talent Gap with Education, Training and Sourcing. 113 https://www.shrm.org/topics-tools/research/skills-gap-2019 Capraro, R. M., Capraro, M. M., & Morgan, J. (2013). STEM Project-Based Learning: An Integrated Science, Technology, Engineering, and Mathematics (STEM) Approach. In Technology, Engineering, and Mathematics (STEM) Approach. Capraro, R. M., Capraro, M. M., & Slough, S. W. (2013). STEM Project-Based Learning: An Integrated Science, Technology, Engineering, and Mathematics (STEM) Approach. In STEM Project-Based Learning: An Integrated Science, Technology, Engineering, and Mathematics (STEM) Approach. Carlson, S. (2022). Building Tomorrow’s Work Force. In The Chronicle of Higher Education. The Chronicle of Higher Education. Carmel, J. H., Herrington, D. G., Posey, L. A., Ward, J. S., Pollock, A. M., & Cooper, M. M. (2019). Helping Students to “Do Science”: Characterizing Scientific Practices in General Chemistry Laboratory Curricula. Journal of Chemical Education, 96(3), 423–434. https://doi.org/10.1021/acs.jchemed.8b00912 Chadwick, S., De La Hunty, M., & Baker, A. (2018). Developing Awareness of Professional Behaviors and Skills in the First-Year Chemistry Laboratory. Journal of Chemical Education, 95(6), 947–953. https://doi.org/10.1021/acs.jchemed.8b00025 Chen, X. (2013). STEM Attrition: College Students’ Paths Into and Out of STEM Fields Statistical Analysis Report. https://nces.ed.gov/pubs2014/2014001rev.pdf Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences Second Edition. Lawrence Erlbaum Associates. Constantinou, C. P., Tsivitanidou, O. E., & Rybska, E. (2018). What Is Inquiry-Based Science Teaching and Learning? In O. E. Tsivitanidou, P. Gray, E. Rybska, L. Louca, & C. P. Constantinou (Eds.), Professional Development for Inquiry-Based Science Teaching and Learning (Vol. 5, pp. 1–23). Springer. https://doi.org/10.1007/978-3-319-91406-0_1 Cooper, J., Basson, J., & Schaap, P. (2006). A Training Programme Based on the Principles of Social Constructivism and Focused on Developing People for the Future World of Work: An Evaluation. Human Resource Development International, 9(4), 467–483. https://doi.org/10.1080/13678860601032593 Cooper, M. M. (1990). Cooperative Chemistry Laboratories. Journal of Chemical Education, 71(4), 307. Cooper, M. M. (2012). Cooperative Chemistry Laboratory Manual (5th ed.). McGraw Hill. Corwin, L. A., Graham, M. J., & Dolan, E. L. (2015). Modeling Course-based Undergraduate Research Experiences: An Agenda for Future Research and Evaluation. CBE Life Sciences 114 Education, 14(1), 1–13. https://doi.org/10.1187/cbe.14-10-0167 Creswell, J. W. (2013). Qualitative Inquiry and Research Design: Choosing Among Five Approaches (3rd ed.). Sage Publications, Inc. Crompton, H., & Sykora, C. (2021). Developing Instructional Technology Standards for Educators: A Design-based Research Study. Computers and Education Open, 2, 100044. https://doi.org/10.1016/j.caeo.2021.100044 Danczak, S. M., Thompson, C. D., & Overton, T. L. (2017). “What Does the Term Critical Thinking Mean to You?” A Qualitative Analysis of Chemistry Undergraduate, Teaching Staff and Employers’ Views of Critical Thinking. Chemistry Education Research and Practice, 18(3), 420–434. https://doi.org/10.1039/c6rp00249h Eddles-Hirsch, K. (2015). Phenomenology and Educational Research. International Journal of Advanced Research, 3(8), 251–260. Galloway, K. W. (2017). Undergraduate Perceptions of Value: Degree Skills and Career Skills. Chemistry Education Research and Practice, 18(3), 435–440. https://doi.org/10.1039/c7rp00011a Goodman, M. J., Sands, A. M., Coley, R. J., & Educational Testing Service. (2015). America’s Skills Challenge: Millennials and the Future. https://www.ets.org/research Herman, W. E., & Pinard, M. R. (2015). Critically Examining Inquiry-Based Learning: John Dewey in Theory, History, and Practice. In P. Blessinger & J. M. Carfora (Eds.), Inquiry- based Learning for Multidisciplinary Programs: A Conceptual and Practical Resource for Educators (Vol. 3, pp. 43–62). Emerald Publishing Limited. https://doi.org/10.1108/s2055- 364120150000003016 Herrington, D. G., & Daubenmire, P. L. (2014). Using Interviews in CER Projects: Options, Considerations, and Limitations. In Tools of Chemistry Education Research (Vol. 166, pp. 31–59). Oxford University Press. https://pubs.acs.org/sharingguidelines Hill, M. A., Overton, T. L., Thompson, C. D., Kitson, R. R. A., & Coppo, P. (2019). Undergraduate Recognition of Curriculum-related Skill Development and the Skills Employers Are Seeking. Chemistry Education Research and Practice, 20(1), 68–84. https://doi.org/10.1039/c8rp00105g Hofstein, A., & Lunetta, V. N. (1982). The Role of the Laboratory in Science Teaching: Neglected Aspects of Research. Review of Educational Research Summer, 52(2), 201–217. Hofstein, A., & Mamlok-Naaman, R. (2007). The Laboratory in Science Education: The State of the Art. Chemistry Education Research and Practice, 8(2), 105–107. International Society for Technology in Education. (2019). ISTE Standards. 115 https://cdn.iste.org/www-root/PDF/ISTE%20Standards-One-Sheet_Combined_09- 2021_vF3.pdf Jackson, D. (2016). Modeling Graduate Skill Transfer from University to the Workplace. Journal of Education and Work, 29(2), 199–231. https://doi.org/10.1080/13639080.2014.907486 Jollands, M., Jolly, L., & Molyneaux, T. (2012). Project-based Learning as a Contributing Factor to Graduates’ Work Readiness. European Journal of Engineering Education, 37(2), 143– 154. https://doi.org/10.1080/03043797.2012.665848 Jones, A. G., & King, J. E. (2012). The Common Core State Standards: A Vital Tool for Higher Education. https://eric.ed.gov/?id=EJ987811 Kondo, A. E., & Fair, J. D. (2017). Insight into the Chemistry Skills Gap: The Duality Between Expected and Desired Skills. Journal of Chemical Education, 94(3), 304–310. https://doi.org/10.1021/acs.jchemed.6b00566 Kraiger, K. (2008). Transforming Our Models of Learning and Development: Web-Based Instruction as Enabler of Third-Generation Instruction. Industrial and Organizational Psychology, 1(4), 454–467. https://doi.org/10.1111/j.1754-9434.2008.00086.x Krajcik, J. S., & Blumenfeld, P. C. (2006). Project-Based Learning. In The Cambridge Handbook of the Learning Sciences (pp. 317–334). Cambridge University Press. Lagowski, J. J. (1990). Entry-Level Science Courses: The Weak Link. Journal of Chemical Education, 67(7), 541. Lamri, J., & Lubart, T. (2023). Reconciling Hard Skills and Soft Skills in a Common Framework: The Generic Skills Component Approach. Journal of Intelligence, 11(6). https://doi.org/10.3390/jintelligence11060107 Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159–174. https://about.jstor.org/terms Mabmud, S. M. (2012). Cohen’s Kappa. In Encyclopedia of Research Design. SAGE Publications, Inc. https://doi.org/10.4135/9781412961288 Mathison, S. (2005). Purposeful Sampling. In Encyclopedia of Evaluation. SAGE Publications. Meaders, C. L., Lane, A. K., Morozov, A. I., Shuman, J. K., Toth, E. S., Stains, M., Stetzer, M. K. R., Vinson, E., Couch, B. A., & Smith, M. K. (2020). Undergraduate Student Concerns in Introductory STEM Courses: What They Are, How They Change, and What Influences Them. Journal for STEM Education Research, 3(2), 195–216. https://doi.org/10.1007/s41979-020-00031-1 116 Metiri Group. (2003). enGauge 21st Century Skills: Literacy in the Digital Age. https://www.researchgate.net/publication/234731444_enGauge_21st_Century_Skills_Digita l_Literacies_for_a_Digital_Age Metiri Group. (2017). 21st Century Learning Continua of Progress. https://istestandardspd.org/product/metiri-groups-continua-of-progress-for-21st-century- learning/ Moerer-Urdahl, T., Creswell, J., & Creswell, J. W. (2004). Using Transcendental Phenomenology to Explore the “Ripple Effect” in a Leadership Mentoring Program. International Journal of Qualitative Methods, 3(2). https://doi.org/10.1177/160940690400300202 Moustakas, C. (1994). Phenomenological Research Methods. Sage Publications. National Academies of Sciences, E and M., National Academy of Engineering, Division of Behavioral and Social Sciences and Education, Policy and Global Affairs, Board on Science Education and Workforce, Board on Science, T. and E.P., Committee on the Supply Chain for Middle-Skill Jobs, & E.T. and C.P. (2017). Building America’s Skilled Technical Workforce. National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, Board on Science Education, Cognitive Board on Behavioral, & Committee on How People Learn II: The Science and Practice of Learning. (2018). How People Learn II: Learners, Contexts, and Cultures. In How People Learn II: Learners, Contexts, and Cultures. National Academies Press. https://doi.org/10.17226/24783 National Association of Colleges and Employers. (2022). Job Outlook 2022. www.naceweb.org|610.868.1421 National Association of Colleges and Employers. (2018). Job Outlook 2019. https://www.luminafoundation.org/wp-content/uploads/2020/01/2019-job-outlook- survey.pdf National Association of Colleges and Employers. (2023). Job Outlook 2023. https://www.naceweb.org/store/2022/job-outlook-2023 National Research Council, & Committee on a Conceptual Framework for New K-12 Science Education Standards. (2012). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. In A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. National Academies Press. https://doi.org/10.17226/13165 National Research Council, Committee on the Assessment of 21st Century Skills, Center for Education, Board on Testing and Assessment, & Koenig, J. A. (2011). Assessing 21st Century Skills: Summary of a Workshop. In Assessing 21st Century Skills. National 117 Academies Press. https://doi.org/10.17226/13215 Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic Analysis: Striving to Meet the Trustworthiness Criteria. International Journal of Qualitative Methods, 16(1). https://doi.org/10.1177/1609406917733847 Ntola, P., Nevines, E., Qwabe, L. Q., & Sabela, M. I. (2024). A Survey of Soft Skills Expectations: A View from Work Integrated Learning Students and the Chemical Industry. Journal of Chemical Education. https://doi.org/10.1021/acs.jchemed.3c00927 Olesen, K. B., Christensen, M. K., & O’Neill, L. D. (2020). What Do We Mean by “Transferable Skills”? A Literature Review of How the Concept is Conceptualized in Undergraduate Health Sciences Education. Higher Education, Skills and Work-Based Learning, 11(3), 616–634. https://doi.org/10.1108/HESWBL-01-2020-0012 Organisation for Economic Co-operation and Development. (2019). Future of Education and Skills 2030: Conceptual Learning Framework. https://www.oecd.org/education/2030- project/teaching-and-learning/learning/skills/Skills_for_2030.pdf Partnership for 21st Century Learning: A Network of Battelle for Kids. (2019). Framework for 21st Century Learning: A Unified Vision for Learning to Ensure Student Success in a World Where Change is Constant and Learning Never Stops. https://static.battelleforkids.org/documents/p21/p21_framework_brief.pdf Plano Clark, V. L., & Ivankova, N. V. (2018). Mixed Methods Research: A Guide to the Field. In Mixed Methods Research: A Guide to the Field. Sage Publications, Inc. Provenzoi, E. F. (1979). History as Experiment: The Role of the Laboratory School in the Development of John Dewey’s Philosophy of History. The History Teacher, 12(3), 373– 382. https://doi.org/https://doi.org/10.2307/491145 Reid, H. M. (2022). Finding Differences With Nominal Data—II : The Chi-Square Test of Independence. In Introduction to Statistics: Fundamental Concepts and Procedures of Data Analysis. SAGE Publications, Inc. https://doi.org/10.4135/9781071878927 Reid, N., & Shah, I. (2007). The Role of Laboratory Work in University Chemistry. Chemistry Education Research and Practice, 8(2), 172–185. Rider, L., & Klaeysen, C. (2015). Employer Perspectives on Soft Skills: 2014 Survey Report. https://www.seattlejobsinitiative.com/wp-content/uploads/Soft-Skill-Survey-Rpt-Sum-of- Findings-032515.pdf Robinson, J. K. (2013). Project-based Learning: Improving Student Engagement and Performance in the Laboratory. Analytical and Bioanalytical Chemistry, 405(1), 7–13. https://doi.org/10.1007/s00216-012-6473-x 118 Rockwood, K. (2021). The Hard Facts About Soft Skills: Why You Should Teach Employees to Be More Resilient, Communicative, and Creative. Society for Human Resource Management. https://www.shrm.org/hr-today/news/hr-magazine/summer2021/pages/why- soft-skills-are-important.aspx Russell, C. B., & Weaver, G. C. (2011). A Comparative Study of Traditional, Inquiry-based, and Research-based Laboratory Curricula: Impacts on Understanding of the Nature of Science. Chemistry Education Research and Practice, 12(1), 57–67. https://doi.org/10.1039/c1rp90008k Sandi-Urena, S., Cooper, M. M., & Stevens, R. H. (2011). Enhancement of Metacognition Use and Awareness by Means of a Collaborative Intervention. International Journal of Science Education, 33(3), 323–340. https://doi.org/10.1080/09500690903452922 Sandi-Urena, S., Cooper, M., & Stevens, R. (2012). Effect of Cooperative Problem-based Lab Instruction on Metacognition and Problem-solving Skills. Journal of Chemical Education, 89(6), 700–706. https://doi.org/10.1021/ed1011844 Schwab, K. (2016). The Fourth Industrial Revolution. www.weforum.org Secretary’s Commission on Achieving Necessary Skills. (1991). What Work Requires of Schools: A SCANS Report for America 2000. https://eric.ed.gov/?id=ED332054 Sim, J., & Wright, C. C. (2005). The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy, 85, 257. https://academic.oup.com/ptj/article/85/3/257/2805022 Seymour, E., & Hunter, A. B. (2019). Talking about Leaving Revisited Persistence, Relocation, and Loss in Undergraduate STEM Education. Springer International Publishing : Imprint: Springer. The Association of American Colleges and Universities by Hart Research Associates. (2013). It Takes More Than a Major: Employer Priorities for College Learning and Student Success. Liberal Education, 99(2). https://eric.ed.gov/?id=EJ1094762 Vygotsky, L. S. (1978). Mind in Society: Development of Higher Psychological Processes. Harvard University Press. Warner, L., Gaters, S. L., Christeson, W., & Kiernan, M. (2011). The Skills Gap: Reversing Washington’s Lack of Skilled Workers Through Early Learning. https://readynation.s3.amazonaws.com/wp-content/uploads/WA-AE-Skills-Report.pdf Wilhelm, J., Wilhelm, R., & Cole, M. (2019). Creating Project-Based STEM Environments The REAL Way. Springer. Wink, D. J., & Weaver, G. C. (2011). Evaluation of the Center for Authentic Science Practice in 119 Education (CASPiE) Model of Undergraduate Research. https://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_07263 2.pdf World Economic Forum, & in collaboration with The Boston Consulting Group. (2015). New Vision for Education: Unlocking the Potential of Technology. https://www3.weforum.org/docs/WEFUSA_NewVisionforEducation_Report2015.pdf Wurdinger, S. D. (2016). The Power of Project-based Learning: Helping Students Develop Important Life Skills. Rowman & Littlefield. Yasin, N. Y. B. M., & Yueying, O. (2017). Evaluating the Relevance of the Chemistry Curriculum to the Workplace: Keeping Tertiary Education Relevant. Journal of Chemical Education, 94(10), 1443–1449. https://doi.org/10.1021/acs.jchemed.7b00296 120 APPENDIX A.IV.1. Course Enrollment by Semester Table 4.3. Course enrollment by semester and course (n). Semester Spring 2020 (Sp20) Summer 2020 (Su20) Fall 2020 (Fa20) GCL1 GCL2 1074 131 1442 594 73 458 Total 1668 204 1900 121 A.IV.2. Student Declared Majors of Course Population by Semester Chemical Physics majors were included in the percentage of chemistry majors because the degree program is administered by the Department of Chemistry. At the institution in this study, there is a Department of Biochemistry and Molecular Biology that oversees biochemistry degree programs, resulting in these majors not being reported in the percentage of chemistry majors within the general chemistry laboratory courses. Biochemistry majors accounted for less than 5% of the sample for each semester. Students pursuing a chemical engineering degree were also not included the percentage of chemistry majors and account for less than 4% of enrollment in these courses. Totals in Table 4.4 below are smaller than those invited to participate in interviews for two reasons: 1) Some participants invited to participate in interviews did not have registrar information on file, or 2) some participants within the data collected from the registrar office were not included on the email list. This could be due to drops and withdrawals that normally occur during a college course leading to this fluctuation being anticipated. Table 4.4. Declared majors of course population by semester, n (%). Declared Major Semester Sp20 Su20 Fa20 Chemistry 36 (2) 6 (3) 43 (2) Biochemistry & Molecular Biology 78 (5) 8 (4) 80 (4) Chemical Engineering 53 (3) 3 (2) 65 (3) Other Majors 1480 (90) 167 (91) 1680 (90) Total 1647 184 1868 122 A.IV.3. Purposeful Sampling Method Students were asked to volunteer for interviews by completing a pre-selection survey indicating their course, class standing, and declared major. Cumulatively across 3 semesters, a total of n = 3,772 students were invited to participate in interviews, and n = 781 volunteered, yielding a response rate of 21% (Table 4.5). Selection of interview participants in each semester used purposeful sampling based on major and course (GCL1 and GCL2) to provide a diverse and representative sample. During the final semester of data collection (Fa20), class standing was an additional parameter considered in defining the target sample. Note, however, that the data from students in GCL1 and GCL2 were pooled in reporting results from this study. Details of the purposeful sampling in each semester are described below. Table 4.5 below summarizes the breakdown in interview participants by semester and course. A total of n = 54 interviews were conducted across the three semesters. The resulting sample size was n = 53 after removal of one interview from Spring 2020 because internet instability made the audio recording largely unusable. Spring 2020 Respondents to the pre-selection survey were sorted by major into the following groups: human biology, biological sciences, engineering, chemistry, biomedical laboratory science, applied life sciences, physical sciences, pre-medicine/nursing/veterinarian, data science, mathematics, psychology, business, international relations, education, exploratory, and undeclared. After sorting the respondents by major and course, initial targets were established for sampling to mirror the distribution of majors and course enrollments in the volunteer population. Students were then randomly selected for interview invitations from each of the groups to achieve the numbers targeted for purposeful sampling. The target participant sample by major and course was human biology (nGCL1 = 3, nGCL2 = 3), biological sciences (nGCL1 = 3, nGCL2 = 5), engineering (nGCL1 = 4), chemistry (nGCL2 = 1), and biomedical laboratory science (nGCL2 = 1). Additional participants were added as the study progressed with the final sampling of majors being human biology (nGCL1 = 3, nGCL2 = 6), biological sciences (nGCL1 = 3, nGCL2 = 3), engineering (nGCL1 = 4), chemistry (nGCL2 = 2), and biomedical laboratory science (nGCL2 = 2) majors. The major of the participant removed following data collection because of poor quality of the audio recording was engineering. 123 Summer 2020 Initial sampling of participants in Summer 2020 was based on the distribution of declared majors for all students in each course versus the volunteers only. Majors were not condensed into further categories and were left as reported on the class roster because the total enrollments for GCL1 and GCL2 (n = 204) and number of volunteers (n = 64) were small. The target majors chosen for sampling were human biology (nGCL1 = 2, nGCL2 = 2), mechanical engineering (nGCL1 = 2), neuroscience (nGCL1 = 1, nGCL2 = 2), kinesiology (nGCL2 = 1), animal science (nGCL1 = 1), psychology (nGCL1 = 1), computer science (nGCL1 = 2), and chemistry (nGCL2 = 2). Fewer participants were interviewed than planned due to time constraints and lack of follow-up from volunteers. The final sampling of majors was human biology (nGCL1 = 3, nGCL2 = 2), mechanical engineering (nGCL1 = 2), neuroscience (nGCL2 = 1), computer science (nGCL1 = 3), and chemistry (nGCL2 = 1). Fall 2020 Coding of participants’ career goals in prior semesters informed the sorting of volunteer’s majors in Fall 2020. Majors were grouped as follows: health-based majors (e.g., human biology, kinesiology, physiology, and nursing), biological sciences (e.g., neuroscience, microbiology, and biochemistry), and engineering. To further diversify the sample and experiences being represented in this study, class standing was used as an additional characteristic when selecting participants. Taking into account course, major, and class standing yielded a target sample of 7 freshman participants in health-based majors (nGCL1 = 4) and engineering (nGCL1 = 3), 9 sophomore participants in health-based majors (nGCL1 = 2, nGCL2 = 4) and biological sciences (nGCL2 = 2) and engineering (n = 1), and 4 juniors in health-based majors (nGCL2 = 3) and biological sciences (nGCL2 = 1). Seniors were excluded from selection based on the belief that students of this class standing were taking the courses out of sequence and may have the perception that the class is not a priority. The final sample included 7 freshman participants in health-based majors (n = 3) and engineering (n = 4), 8 sophomore participants in health-based majors (n = 5), biological sciences (n = 2), and engineering (n = 1), and 3 junior participants in health-based majors (n = 2) and biological sciences (n = 1). 124 Table 4.5. Interview response rates. # of Students Invited (n) 1075 # of Students Who Volunteered (n) 147 Response Rate (%) 14 # of Students Who Participated in Interviews (n) 10 Final Sample Size (n) 9* Semester Course Sp20 GCL1 Su20 Fa20 GCL2 GCL1 GCL2 GCL1 GCL2 Total 594 131 73 1442 458 3772 108 37 27 365 97 781 18 28 37 25 21 21 14 8 4 10 8 54 14 8 4 10 8 53 *One participant was removed post data collection because the poor audio quality of the interview recording made transcription of a significant portion of the interview impossible. 125 A.IV.4. Independent Samples t-tests Composite SAT scores were used to determine if the interview sample was significantly different from the course population. This variable was chosen for testing differences because freshman and cumulative college GPAs show correlation with SAT scores (Bridgeman et al., 2008; Westrick et al., 2019). Students who participate in extra credit assignments, such as interviews or surveys, may be higher achieving students, and we aimed to avoid this potential selection bias. To test if the interview participants had different mean SAT scores from the overall course population, independent samples t-tests were run for each semester after confirming a normal distribution of scores for interview participants and the course population. Alpha levels of 0.05 were used to determine significance for all tests reported below. Using the Shapiro-Wilk test for normality, the distribution of scores for the sample and population were assessed to initially determine if the assumption of normality was met for the two samples (Table 4.6). Although the course population for Sp20 and Fa20 failed the test for normality (p < 0.05), sample sizes greater than 30 follow the central limit theorem allowing for normality to be assumed (Ghasemi & Zahediasl, 2012; Lumley et al., 2002). Table 4.6. Shapiro-Wilk test for normality by semester. Semester Sp20 Su20 Fa20 Sample Interview Participants Course Population Interview Participants Course Population Interview Participants Course Population Shapiro-Wilk Sample Size (n) 23 1582* 11* 155* 18 1788* Test Value (W) 0.975 0.997 0.932 0.988 0.931 0.998 Significance (p-value) 0.807 0.006 0.429 0.199 0.206 0.007 *Not all students in the sample had SATX scores on file. These missing values were not included in testing. With the assumption for normality being satisfied for both the interview sample and course population, independent samples t-tests were run (Table 4.7). Levene’s test was first used to assess if equal variances could be assumed, ultimately determining the appropriate t-test statistic to use in analysis. Equal variances could be assumed for Su20 allowing for use of the equal variance t-test, while this assumption failed for Sp20 and Fa20 semesters requiring use of an unequal variance t-test. To avoid Type 1 errors from occurring due to multiple testing, that includes the outcomes of these tests and additional statistical analyses that will be explored later 126 on in this data set, Benjamini-Hochberg adjustment of p-values was performed (Benjamini & Hochberg, 1995). Type 1 errors incorrectly identify significance when there is no true significance present (Liu, 2022). Independent samples t-test results returned no significant differences between mean SAT scores in the interview samples and the course populations for all semesters. Table 4.7. Independent samples t-test results by semester. Levene’s Test Independent Samples t-test Test Sample Size (n) Mean SAT Score Value Significance Value 23 1279.13 3.989 0.046 1.744 1582* 1217.29 Significance (p-value) 0.640 11* 1279.09 1.685 0.196 1.530 0.640 155* 1218.00 18 1182.22 5.570 0.018 -0.838 0.686 1788* 1215.68 Semester Sp20 Su20 Fa20 Sample Interview Participants Course Population Interview Participants Course Population Interview Participants Course Population *Not all students in the sample had SATX scores on file. These missing values were not included in testing. †First- generation student status was defined as students being first generation in which neither parent (or guardian) has a four-year degree or continuing generation in which either one or both parents (or guardians) have obtained a four- year degree. 127 A.IV.5. Participant Demographics Table 4.8. Interview participant demographics reported as number of participants, n (%). Demographic Course GCL1 GCL2 Legal Sex Female Male First Generation Status First Generation Continuing Generation Class Standing Freshman and Sophomore (< 56 credits earned) Junior and Senior (56 or more credits earned) Age Ranges 18-20 21-23 Mean SS20 (n = 23) 9 (39) 14 (61) 13 (56) 10 (44) 5 (22) 18 (78) Semester US20 (n = 12) 8 (67) 4 (33) 6 (50)* 5 (42)* 2 (17)* 9 (75)* FS20 (n = 18) 10 (56) 8 (44) 11 (61) 7 (39) 3 (17) 15 (83) Total (n = 53) 27 (51) 26 (49) 30 (57)* 22 (42)* 10 (19)* 42 (79)* 19 (83) 6 (54)* 14 (78) 39 (75)* 4 (17) 5 (46)* 4 (22) 13 (25)* 22 (96) 1 (4) 18.8 10 (91)* 1 (9)* 19.3* 18 (100) 0 (0) 18.7 50 (96)* 2 (4)* 18.9* *One US20 participant had no demographic data on file with the Registrar’s Office. 128 A.IV.6. Interview Protocol Version 1 (Spring Semester 2020) Greeting statement. Thank you for taking the time to meet with me today. Background statement. Today I am going to ask you questions about your career plans and the skills that you think are needed to be effective in your planned career. I will also ask you questions about your experiences in your general chemistry laboratory course(s), general chemistry laboratory 1 and/or general chemistry laboratory 2. This interview will be audio-recorded so that I have an accurate record of what you say. This interview is not a test. There are no right or wrong responses. I am interested in your ideas. The more you explain what you are thinking, the more helpful this interview will be to this research and improving the general chemistry laboratory courses for future students. Your participation in this interview will have no impact on your grade in your current general chemistry laboratory course, and your lab instructor will not know that you have participated in this interview. Your responses will be treated confidentially and will never be associated with you. If at any time, you do not want to answer a question, you may skip the question. You may stop participating in this interview at any time. During the interview, I may wait to make sure that you have had adequate time to think about and respond to a question and that you have finished responding. I may ask follow-up questions that ask you to elaborate to make sure that I understand what you are saying. Opening questions. Future career, prior experiences, skills needed (not-prompted), and opportunities for development 1. What is your major/ minor? What attracted you to this major? 2. What career do you plan to pursue after graduation? 3. Tell me about what you expect to do in this career? • Alternative Phrasing & Possible Follow up Questions: What does a typical day in this career look like? Have you had an internship/ shadow opportunity for said career? What do you think this job will be about? 4. What skills do you think you will need to be effective in this career? 5. Follow up: Why are these skills important? How are they used in this career? 6. Do you feel you have the qualities of [skill]? How do you feel you are at [skill]? 129 7. Have you had any experiences in your college career or college coursework that contribute to developing [skill]? 8. Have you had experiences in your general chemistry laboratory course(s) that you think will prepare you for your future career? Please elaborate. 9. How do your experiences in general chemistry laboratory 1 and/or general chemistry laboratory 2 help you gain the [skills/qualities they mentioned]? • Follow up: Were there any projects in particular that helped you gain these skills? How did those projects/ tasks help you gain those skills? Continue questions 7-8 for each skill participant listed. Break. Before continuing questioning, take this time to ask students if they want to add anything to their previous responses. Resumption of questioning. Continuation of skills needed (prompted), opportunities for development in general chemistry laboratory course, impact of emergency remote instruction, and miscellaneous closing questions Provide students with a visual aid containing list of 15 professional skills (Figure 4.4) and continue with line of questioning below. Statement to introduce list. Here is a list of 16 Essential Health and Science skills that employers look for in future college graduates. 10. Which ones do you believe are needed for you future career and how? 11. From this list, can you talk about any specific skills and/or instances that general chemistry laboratory 1/general chemistry laboratory 2 helped you gain that particular skill or set of skills? 12. Have you learned anything from your experiences in general chemistry laboratory 1 and/or general chemistry laboratory 2 about your strengths and/or weaknesses? 13. Are there any skills that you personally would like to work on developing as you prepare for your career? 14. In closing, how did moving from an in-person lab course to an online lab course affect your ability to build these skills? Closing statement. Thank you so much for this really interesting conversation/all of your 130 thoughts. Would you like to add anything else on the subject? Thanks again. Close. 131 A.IV.7. Interview Protocol Version 2 (Summer Semester 2020) Greeting statement. Thank you for taking the time to meet with me today. Background statement. Today I am going to ask you questions about your career plans and the skills that you think are needed to be effective in your planned career. I will also ask you questions about your experiences in your general chemistry laboratory course(s), general chemistry laboratory 1 and/or general chemistry laboratory 2. This interview will be audio-recorded so that I have an accurate record of what you say. This interview is not a test. There are no right or wrong responses. I am interested in your ideas. The more you explain what you are thinking, the more helpful this interview will be to this research and improving the general chemistry laboratory courses for future students. Your participation in this interview will have no impact on your grade in your current general chemistry laboratory course, and your lab instructor will not know that you have participated in this interview. Your responses will be treated confidentially and will never be associated with you. If at any time, you do not want to answer a question, you may skip the question. You may stop participating in this interview at any time. During the interview, I may wait to make sure that you have had adequate time to think about and respond to a question and that you have finished responding. I may ask follow-up questions that ask you to elaborate to make sure that I understand what you are saying. [Make sure student has pen/pencil and paper ready!] Opening questions. Future career, prior experiences, skills needed (not-prompted), and opportunities for development 1. What is your major? What is your minor? • What attracted you to this major/minor? 2. What career do you plan to pursue after graduation? 3. Tell me about what you expect to do in this career? • Alternative Phrasing: What does a typical day in this career look like? What do you think this job will be about? 4. Have you had an internship/shadowing opportunity for [career]? 132 5. I would like you to write down skills you think you will need to be effective in this career. 6. What is the first skill that you wrote down? a. How will [skill] be used in [this career]? b. Allow student to mention each skill individually, and inquire about each skill individually so that student can talk through them. c. What is your personal experience with [skill mentioned] (in any context)? d. Have you had any experiences in your college career that contributed to developing [skill]? (Mention one by one, and let them talk through each skill.) e. Have you had any experiences in your general chemistry lab course(s) that relate to [skill in list]? f. How do your experiences in general chemistry laboratory 1 and/or general chemistry laboratory 2 help you gain [skill mentioned]? g. Were there any projects, in particular, that helped you gain these skills? How did those projects/tasks help you gain those skills? h. Repeat questions a-g for each skill that participant listed. 7. What influence did your experiences in general chemistry laboratory 1 and/or general chemistry laboratory 2 have on skills needed for [future career]? Break. Before continuing questioning, take this time to ask students if they want to add anything to their previous responses. Resuming questioning. Continuation of skills needed (prompted), opportunities for development in general chemistry laboratory course, impact of online instruction, and miscellaneous closing questions. Provide students with visual aid containing list of 15 professional skills (Figure 4.4) and continue with line of questioning below. Statement to introduce list Here is a list of 15 Essential Health and Science skills that employers look for in future college graduates. I am going to ask you a series of questions about this list, and feel free to write down what you are thinking as we go. 133 8. Do you see any skills on here, that you haven’t mentioned previously, that would relate to your future career? a. How does this [skill] relate to your future career? (Repeat this question for each skill mentioned.) 9. Do you see any skills on here that you have not mentioned previously that would relate to your experience in general chemistry laboratory 1 and/or general chemistry laboratory 2? a. Which ones? b. How? c. Any specific instances? d. Provide an example if participant needs help. • For questions a and b, allow participants to talk through each skill individually. 10. Throughout the course, did you recognize any personal strengths? Ask for specific experiences as a follow up. 11. Did you recognize any personal weaknesses? Ask for specific experiences as a follow up. 12. Are there any skills that you personally would like to work on developing in your general chemistry laboratory course? 13. Additionally, what skills do you feel the course is lacking, that you would like a chance to build on? • Follow-up: Do you have any suggestions for how this skill could be integrated into the course? (This is putting students on the spot, and as such, it is totally okay to skip). 14. How did moving from an in-person lab course to an online lab course affect your ability to build [skills]? Can talk about skills individually here as well. 15. Have you had any additional experiences, outside of coursework, that impacted your skill development? Can mention extracurricular activities, job, etc. here. Closing statement. Thank you so much for this really interesting conversation/all of your thoughts. Would you like to add anything else on the subject? Thank you again, and would you be able to send either a screenshot or scanned copy of the notes you had taken during this interview? 134 Close. 135 A.IV.8. Interview Protocol Version 3 (Fall Semester 2020) Greeting statement. Hello! Thank you for taking the time to meet with me today. Background statement. Today I am going to ask you questions about your career plans and the skills that you think are needed to be effective in your planned career. I will also ask you questions about your experiences in your general chemistry laboratory course(s), general chemistry laboratory 1 and/or general chemistry laboratory 2. This interview will be audio-recorded so that I have an accurate record of what you say. This interview is not a test. There are no right or wrong responses. I am interested in your ideas. The more you explain what you are thinking, the more helpful this interview will be to this research and improving the general chemistry laboratory courses for future students. Your participation in this interview will have no impact on your grade in your current general chemistry laboratory course, and your lab instructor will not know that you have participated in this interview. Your responses will be treated confidentially and will never be associated with you. If at any time, you do not want to answer a question, you may skip the question. You may stop participating in this interview at any time. During the interview, I may wait to make sure that you have had adequate time to think about and respond to a question and that you have finished responding. I may ask follow-up questions that ask you to elaborate to make sure that I understand what you are saying. Remember. Ask student to have a piece of paper to jot things down and then collect after interview. Opening questions. Future career, prior experiences, skills needed (not-prompted), and opportunities for development. 1. What is your major? What is your minor? • What attracted you to this major/minor? 2. What is your career goal following graduation from MSU? • Alternative Phrasing: What are your future goals after you finish your undergraduate career at MSU? 3. Tell me about what you expect to do in this career? 136 • Alternative Phrasing: What does a typical day in this career look like? What do you think this job will be about? 4. Have you had an internship/shadowing opportunity or prior work experience related to your planned career? 5. Can you list some skills off the top of your head that you would need to be successful in [career]? 6. How would that skill be used in [career]? 7. What is your personal experience with [skill]? 8. Did you have any experiences in your gen chem lab course that relate to [skill]? Repeat questions 6-8 for each skill that participant listed. Break. Before continuing questioning, take this time to ask students if they want to add anything to their previous responses. Resuming questioning. Continuation of skills needed (prompted), opportunities for development in general chemistry laboratory course, impact of online instruction, and miscellaneous closing questions Provide students with visual aid containing list of 6 professional skills (Figure 4.5) and continue with line of questioning below. Statement to introduce list. Here is a list of skills that employers look for in future college graduates. I am going to ask you a series of questions based on this list. Feel free to write down what you are thinking as we go. To start, looking at these skills in general, what do these skills mean to you? How would you define these skills? 9. Do you see any skills on here, that you haven’t mentioned previously, that would relate to your future career? 10. Do you see any skills on here that you have not mentioned previously that would relate to your experience in general chemistry laboratory 1/general chemistry laboratory 2? 11. Are there any skills that you personally would like to work on developing in your gen chem course? 137 12. Have you had any additional experiences, outside of coursework, that have helped you build any of these skills? 13. Do you believe that these skills are valuable/beneficial to you? 14. Do you believe these skills would be helpful or useful in other careers? 15. Did moving from an in-person lab course to an online lab course affect your ability to build [skills]? How so? 16. In general, what has your experience been attending classes online? Closing statement. Thank you! Would you be able to send either a screenshot or scanned copy of the notes you had taken during this interview? Close. 138 A.IV.9. Visual Aids During the second half of the interview, students were shown a visual aid displaying skills that employers desire in new hires with science degrees to frame further discussion of the skills that participants perceived as relevant to their planned career and developed in their general chemistry laboratory course. The intent was to elicit discussion of skills that students may not have considered without prompting as well as further discussion of skills mentioned during the first half of the interview. In the study design, we were interested in seeing which skills students mentioned without prompting and if their focus shifted after a list of skills was provided. However, in reporting the data from this study, skills discussed with and without prompting are not distinguished. Visual Aid 1 (Figure 4.4, located on page 138) was provided during Sp20 and Su20 interviews, while Visual Aid 2 (Figure 4.5, located on page 139) was utilized during Fa20 interviews. Visual Aid 1 was based on the 16 Essential Health and Science Skill Sets (A.10) with exclusion of Balancing Work/Life because it is not a skill that could reasonably be expected to be developed within a course. This skill was excluded due to the belief that students may feel overwhelmed and not perceive a fair work/life balance during their college career. As the study progressed, the list of 15 skills was further condensed into 6 skills for Fa20 participants based on the preliminary analysis of interview and survey data from Sp20 and Su20. Figure 4.4. Visual aid used in Spring 2020 and Summer 2020 interviews. 139 Figure 4.5. Visual aid used in Fall 2020 interviews. 140 A.IV.10. 16 Essential Health and Science Skill Sets The 16 Essential Health and Science Skill Sets was developed by asking employers who interview students from Michigan State University’s College of Natural Science about the skills they look for in potential hires (Telfor, 2017). This skill set, with associated definitions, were used in development of surveys (not reported in this paper) and also framed the initial analysis of interview data. Skills are listed below with definitions from the original document. • Acquiring Knowledge: Absorbing concepts and facts in formal and informal situations. Accessing sources of information and testing their validity. Connecting related ideas. Balancing broad-based learning with a capacity for specialization in a given subject. Maintaining an active, inquisitive mind. • Balancing Work/Life: Giving time to each of the important dimensions of life: work, family, personal interests, community, spiritual. Remaining flexible when one or more dimensions need extra attention. Paying attention to personal needs and showing sensitivity to the balance needs of others. Realizing the interconnected nature of all dimensions. • Communicating Effectively: Tailoring messages to a specific audience. Interpreting messages accurately. Writing concisely. Speaking clearly. Presenting professionally and in a manner that captivates the audience. Listening well. Understanding how to craft a persuasive argument. • Contributing to a Team: Recognizing and validating the perspectives of team members. Identifying individual strengths (yours and others) and harnessing them for the group. Building consensus. Knowing when to lead, when to follow. Appreciating group dynamics. • Developing Professional/Technical Skills: Mastering tools or techniques that improve workflow. Knowing the potential and the limits of a technology or method, as well as its best application. Acquiring formal training when appropriate; respecting formal standards. Continually upgrading skills and keeping abreast of new technologies or methods. • Embracing Change: Accepting the inevitability of change; recognizing its cyclical and sometimes persistent nature. Seeing change as opportunity. Understanding how people respond to change; helping others move forward. • Managing Time & Priorities: Breaking large assignments into manageable tasks; organizing action steps in a logical sequence. Separating essential from non-essential, urgent from trivial, with regard to stakeholder interests. Developing plans; imposing structure when necessary. Staying on task; restricting distractions. Adjusting to continual changes. 141 • Navigating Across Boundaries: Comprehending the relationship between the parts and the whole. Recognizing common interests. Respecting norms and values of other domains. Acknowledging “turf” but not being deterred by it. Adjusting quickly to new environments. • Performing with Integrity: Acting consistently. Keeping one’s word. Following through. Operating from a code of ethics. Making principled decisions. Treating others fairly. Behaving in a “transparent” manner. Accepting responsibilities and admitting mistakes. • Solving Problems: Defining contexts. Gathering information from reliable sources. Viewing from multiple perspectives. Developing and testing hypotheses. Approaching with a win-win orientation. Collaborating. Envisioning resolution. • Thinking Critically: Sifting through mounds of data to identify pertinent elements; sorting data by relevance. Applying qualitative or quantitative measures to detect trends. Making sense of disparate or conflicting information. Recognizing assumptions inherent in analysis. • Working in a Diverse Environment: Valuing others. Embracing difference; regarding group diversity as a strength. Understanding one’s own identity development. Admitting and eradicating personal biases. Demonstrating commitment to diversity through supportive actions. • Self-motivation: Showing initiative to undertake a task or activity without another’s directive or supervision. Motivated to do or achieve something because of one's own enthusiasm or interest, without needing pressure from others. • Enthusiasm/Commitment: A strong internal feeling, motivation or desire which often results in goal and action specific behavior. Absorbing or controlling possession of the mind by any interest or pursuit; lively interest. • Empathy: The fact or power of sharing the feelings of another, compassion, or commiseration. An ability to recognize and describe another individual’s perspective accurately. Demonstrating the ability to actively listen and understand the experience of another. • Curiosity: The desire to learn or know about anything or something in particular; inquisitiveness. 142 A.IV.11. Qualitative Coding of Skills Needed for Future Career Goal Interview participants discussed a wide range of skills. These skills were identified in interview transcripts during the initial phase of inductive coding. Thematically related skills were first grouped into what became subcategories (e.g., Social & People Skills), which are shown in italics in the coding scheme found below, followed by grouping related subcategories into seven overarching skill sets (e.g., Interpersonal Skills) highlighted in bold. Within each subcategory, examples of specific skills are provided. During coding, more than one skill could be applied to an interview segment depending on how a participant talked about skills. For example, “[...] I had one more, it was [...] willing to learn, and [...] open mindedness [...]” was coded for both Intrapersonal Skills: Personality & Character Traits skill of “open mindedness” and Education & Learning Skills: Learning Skills & Strategies skill of “willingness to learn”. Coding Scheme - Skills Needed for Career 1. Interpersonal Skills: • Teamwork & Collaboration: includes working well with others, being cooperative, team building and organizing a team, listening to others and taking others’ ideas and perspectives into consideration, contributing to a team. • Communication Skills: includes verbal communication (public speaking and presenting), written communication (e.g., writing research papers and reports), communicating effectively. • Social & People Skills: includes being reliable, working with others in a non- collaborative way (e.g., working with patients), being charismatic and persuasive, being friendly, kind, and respectful, being caring, showing compassion and empathy. • Miscellaneous Interpersonal Skills: o Leadership Skills: overseeing decisions, guiding people in a group, making sure group is working cohesively and on track, taking initiative, delegating tasks. o Networking Skills: utilizing outside experts to answer a question and receive guidance. o General Interpersonal Skills: includes mentioning the need for interpersonal skills with no specific subskill listed. 2. Intrapersonal Skills: • Work Ethic: includes dedication, drive, determination, self-motivation, persistence, and perseverance (ability to work under pressure), focus and concentration, being responsible and accountable, performing with integrity. 143 • Personality & Character Traits: includes confidence, being outgoing, calm, patient, serious, tough, brave, being a self-advocate, being independent/showing independence (ability to work independently & isolate), being open-minded, being humble, flexible and adaptable (being versatile, managing challenges and adapting accordingly, embracing change). • Miscellaneous Intrapersonal Skills: enthusiasm, commitment. This miscellaneous category of intrapersonal skills stems from enthusiasm/commitment being present in the list provided to students and being considered to fit in both subcategories of work ethic and personality traits. 3. Occupation-specific Knowledge & Skills: • Technical Skills: o Lab Skills & Techniques: includes experience with laboratory tools. o Technological/Computer Skills: includes computer science knowledge and skills, computer modeling (3D models and simulations), coding knowledge & skills (coding software & programs and programming skills). o Hands-on Skills: includes hand-eye coordination when working with tools. o Professional/Technical Skills (from the 16 Essential Health & Science Skill Set) o Field-specific Knowledge: ▪ Medical Knowledge: includes ability to treat and diagnose patients, ability and knowledge to perform surgeries and medical procedures, knowledge of administering and prescribing medicine, drugs and drug side effects (e.g., knowing how drugs work and react within the body), conceptual medical knowledge, having a background in medical specialties, dentistry knowledge. ▪ Business Knowledge: includes business management, economics (e.g., microeconomics). ▪ Psychology Knowledge: includes psychoanalysis. ▪ Science, Engineering, and Mathematics Knowledge: ▪ Mathematics: includes algorithms, logic-based mathematics. ▪ Science: • General Scientific Knowledge & Skills: includes understanding of scientific concepts and principles, scientific method. • Life Sciences: includes genetics and genomics. • Physical Sciences: includes physics (e.g., heat transfer), chemistry. o Miscellaneous Occupation-specific Skills: 144 ▪ Exposure to Field & Hands-on Experience: includes internship, research & lab experiences, training in profession. ▪ Professionalism: this skill was not expanded upon by participants and facilitated the use of a definition generated from literature sources to support categorization. Professionalism within this study is defined by how to conduct oneself within the workforce, dependent upon the context of the profession and work environment (Cao et al., 2023; Guraya et al., 2023; Manion, 2001; Whelan, 2008). 4. Problem-solving & Critical Thinking: • Creativity & Innovation: generating new ideas, being able to approach problems in different ways. • Analytical Skills: data analysis, working with data, assessing a problem and providing a solution based on the information provided (e.g., diagnosing a patient). 5. Prioritization & Time Management: • Scheduling & Preparation: includes planning skills, time dedicated for studying, courses, and research). • Efficiently Using Time • Multi-tasking • Meeting Deadlines • Organization Skills • Punctuality • Managing Priorities 6. Education & Learning Skills: • Education & Credentials: including acquiring knowledge (e.g., gaining general and/or basic knowledge & understanding that is not explicitly related back to discipline specific knowledge). • Learning Skills & Strategies: includes curiosity and willingness to learn, willingness to fail and learn from mistakes. 7. Other Skills: • Decision-making Skills: includes ability to make decisions, fast and quick thinking (e.g., thinking on your feet), good judgement. • Attention to Detail & Observation Skills • Navigating Across Boundaries • Working in Diverse Environments 145 A.IV.12. Skills Needed Perceived as Needed Without Prompting and Career Goal Capturing data with enough detail, while also maintaining a thematic framework was a delicate process when dealing with these data. While condensing a coding scheme can result in a loss of diverse responses, it was our aim to continue to capture more descriptive themes present within these data. To do this, skill sets were unpacked and displayed in Figures 4.6 - 4.7. Within these figures interpersonal, occupation-specific, and intrapersonal skills were disaggregated into further subcategories due to the prevalence of these categories being mentioned throughout interviews. Problem-solving & critical thinking, priority & time management, and other skill sets remained intact due to their subcategories only being mentioned by relatively small numbers of participants. Additionally, these figures show the skills that participants mentioned as relevant to their planned careers without prompting and are further disaggregated by career goal (health & medical professions, engineering & subspecialties, and other careers)(Figure 4.6) and prior experience (has/has not had internship/ shadowing/work opportunities)(Figure 4.7). Those who were not asked to elaborate on whether they had in-field experiences were not included in this analysis, resulting in n = 48 participants. Accompanying these figures are related tables displaying statistical outcomes of testing for comparisons of each individual skill (Table 4.9 and Table 4.10). All statistical methods employed are detailed in Chapter VI. Prevalent findings are discussed within the main chapter. All corresponding tables and figures can be found on pages 147 - 149. 146 ) % ( l a o G r e e r a C n i h t i w t n e c r e P 100 90 80 70 60 50 40 30 20 10 0 Health & Medical Professionals Engineering & Associated Subspecialties Other Careers Figure 4.6. Skills mentioned pre-prompting by career goal (n = 53)(%). Table 4.9. Chi-square test values and significance for skills mentioned pre-prompting by career goal (n = 53). Student-perceived Competency (SPC) Communication Skills Teamwork Skills Social & People Skills Miscellaneous Interpersonal Skills Personality & Character Traits Work Ethic Skills Problem-solving & Critical Thinking Prioritization & Time Management Field-specific Knowledge Technical Skills Miscellaneous Occupation-specific Skills Education & Learning Skills Other Skills Chi-square Test Value 0.333 3.135 10.242 2.945 1.617 0.806 2.900 0.497 1.157 3.787 2.541 1.008 1.864 Significance 1.000 0.684 0.105 0.684 0.750 1.000 0.684 1.000 0.964 0.661 0.684 1.000 0.686 147 r o i r P n h t i i w t n e c r e P ) % ( y r o g e t a C e c n e i r e p x E 100 90 80 70 60 50 40 30 20 10 0 Has Had Prior Experience In-field Has Had No Prior Experience In-field Figure 4.7. Student-perceived competencies mentioned pre-prompting by whether participant had hands-on prior experiences in field (%). 148 Table 4.10. Chi-square test values and significance for skills mentioned pre-prompting by prior experience (n = 48). Student-perceived Competency (SPC) Chi-square Test Value Significance Communication Skills Teamwork Skills Social & People Skills Miscellaneous Interpersonal Skills Personality & Character Traits Work Ethic Skills Problem-solving & Critical Thinking Prioritization & Time Management Field-specific Knowledge Technical Skills Miscellaneous Occupation-specific Skills Education & Learning Skills Other Skills 0.686 0.686 0.684 0.964 0.684 1.000 0.684 1.000 0.684 1.000 1.000 0.614 0.614 1.185 — 0.939 — 0.939 0.011 1.296 0.058 0.578 1.000 — — — * Test values denoted with an em dash (—) reference comparisons in which cells in the contingency table had values less than 5, facilitating the need to use the Fisher Exact Test resulting in no chi-square test values, only significance being reported. 149 A.IV.13. Ranking of six employer-desired competencies by Fa20 interview participants. a. ) n ( t n u o C c. ) n ( t n u o C e. ) n ( t n u o C 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 1 2 3 4 Ranking 5 6 1 2 3 4 Ranking 5 6 b. ) n ( t n u o C d. ) n ( t n u o C f. ) n ( t n u o C 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 1 2 3 4 Ranking 5 6 1 2 3 4 Ranking 5 6 1 2 3 4 Ranking 5 6 1 2 3 4 Ranking 5 6 Figure 4.8. Fa20 interview participant ranking of employer-desired competencies (n = 14)(n). Skills corresponding to rankings - a. Communication skills (median = 3), b. Teamwork skills (median = 2), c. Problem Solving & Critical Thinking Skills (median = 3), d. Priority & Time Management (median = 3), e. Self-motivation (median = 3), and f. Technical Skills (median = 5). One participant did not assign a ranking to self-motivation or technical skills, resulting in a total of n = 13 participants for these skills. Percentages regarding these skills are calculated using this value. 150 A.IV.14. Comparison of skills recognized as important for planned career with and without prompting. Table 4.11. Comparison of the top six SPCs as being recognized pre- or post-prompting (n). Student-perceived Competency (SPC) Communication Skills Teamwork Skills Work Ethic Skills Problem-solving & Critical Thinking Prioritization & Time Management Technical Skills Recognized Not-prompted 23 29 11 19 17 10 Additional Participants Who Recognized Prompted 21 17 35 29 25 26 Total 44 46 46 48 42 36 151 A.IV.15. Student perceptions of how skills are developed and why the course may not have contributed to perceived development. In addition to exploring instances in which the courses supported development, participants who definitively expressed that the courses lacked opportunities for development of the top six SPCs were analyzed to determine both explicit and implicit reasons behind why this could have been the case. As technical skills were explored in the main chapter, the skills investigated here will include communication skills, teamwork skills, work ethic, problem-solving & critical thinking, and prioritization & time management. Notable examples given by students were like those regarding technical skills – learning is seen as being context dependent. The context that students often spoke of included skill development being dependent upon the course and the individual. These themes, and others, will be explored below. When looking into how students spoke of skill development for those that were not related to the course, they often spoke of how these skills were only perceived to be developed in certain environments or when situations arose. One area that students spoke of was the importance of the type of course in which skills were developed seen in communication skills, teamwork skills, work ethic, problem-solving & critical thinking and prioritization & time management, with quotes from River and Aspen as examples. Ah, not really too much. No, it's kind of just been like going to classes have a lot of science classes, so you don't really have a lot of opportunities, so. (River, Communication Skills) [. . .] I'd say, one maybe not as much would be problem-solving and critical thinking, I'd say I'd learn more of that in lectures as opposed to the labs [. . .] (Aspen, Problem-solving & Critical Thinking) Another theme that relied on the situational and contextual component to learning SPCs was that development was dependent upon the individual and can be influenced by the presence of motivators or personal values. This was primarily seen for those speaking of work ethic skills but was also present in problem-solving & critical thinking and prioritization & time management skills. 152 I feel like self-motivation, not always getting the answers can motivate someone to, like not being given everything that they need can motivate them to want to figure it out. But no, I don't really think that they can be fully taught in a classroom. I feel like the student has to come in with some of those and they can grow on it in a classroom but not really be taught it. (Indy, Work Ethic) [. . .] So, when it comes down to time managing and priorities and mainly priorities because look, maybe on the top of my priority list may not be on top of the next person’s priority list. [. . .] So, yeah, I just feel like it just depends on the person.” (Oak, Prioritization & Time Management) An additional aspect as seen in Indy’s quote above, and implied in Kyles statement below, is that although these work ethic and problem-solving & critical thinking skills can be learned, they are not encouraged in a classroom environment. [. . .] So, yeah, most of my courses are like just math, computer, CSE. So, they're really just definite. There's no room for exploration, because they all have their certain formula, certain guidelines that you always have to follow [. . .] (Kyle, Problem-solving & Critical Thinking) In the case of problem-solving & critical thinking, one participant also spoke of how it was learned doing something new or challenging in addition to be dependent upon the course, and found that because they had already engaged in aspects found in the chemistry labs that there was no opportunities for growth. [. . .] So ,I studied chemistry [. . .] so, I do have some knowledge about all these chemistry things before. So, it's pretty obvious for me to, you know, to do those experiments, I did titrations in my 11th grade. [. . .] And, you know, I had that experience before and having that experience before, you know, I think it didn't really, it didn't help me that much. (Shiloh, Problem-solving & Critical Thinking) 153 While some found the context of learning to be important, others spoke of skill development being encouraged throughout their college career, for prioritization & time management and work ethic skills, and skill development not being specific to a certain course. I feel like I'm learning managing time and priorities, just as a college student, but not very specifically to one class. (Indy, Prioritization & Time Management) I think [. . .] just thinking in general about determination, I would say, and this is kind of sp-, not necessarily specific to those classes or courses. But I would say determined to finish a lab report or like, yeah. Like to finish a lab report or to complete your portion of some PowerPoint slides or something like that. That's like the only thing I can really think of as far as determination. (Reign, Work Ethic) For some participants who spoke of work ethic skills, these were perceived to be intrinsic traits or qualities inherent to a person. This was exemplified by Indy and Oak. I feel like self-motivation, empathy, and curiosity are good things to have going into the field, but I don't really know learn them from the lab or the class, just good personal attributes, I guess, but I don't feel like I learned them from the lab. [. . .] I feel like they're things that can be encouraged, but not really taught in a classroom setting. Maybe as much, like, I'm trying to think of an example. (Indy, Work Ethic) Self-motivation. I just think that's a natural occurring one. You have to, you have to be motivated to go to college anyway, like it's just something just a part of me, I guess you could say. (Oak, Work Ethic) Online was a very small component that was perceived to negatively impact skill development. One participant had related all skills they had mentioned not-prompted as being hindered due to the learning environment being online, with critical thinking being included in this. Additionally, n = 2 participants felt that the work ethic skill of self-motivation and drive was negatively impacted online. For Billie this was because their online experience and lack of 154 the ability to be in lab carrying out experiments in GCL2, contrasted with their GCL1 experience, was perceived to be less enjoyable and confusing, leading to a decrease in the work ethic skill of drive being applicable to their lab experience. Additionally, Corey felt the work ethic skill of self-motivation was harder online, and that in a course such as the lab it was harder to “visualize and teach myself.” Through the examples provided above it can be observed that there are a variety of perceptions that lead to how a student may view skill development as being supported or hindered in a learning environment and how important context is in building this perception. 155 A.IV.16. Course Element Codes When exploring how participants perceived development of SPCs within the course(s), codes were generated to capture participant experiences that were categorized further into themes representative of various course elements. The course elements explored in this section will pertain to those not expanded upon in the main chapter. Below are tables containing codes representative of the course elements and their occurrence (n) in skill development. The primary purpose of including these tables is to shed light on experiences that did were mentioned by participants but did not constitute an overarching theme and display patterns that were discussed in the main chapter as presented in Table 4.12, including codes for skill development that had a frequency of n = 1. Main codes not previously identified within the main chapter is that of real-world context and responsibility over lab drawer. Real-world context was showcased in participant responses speaking of how scenarios or the course environment are reminiscent of situations they would encounter in the working world (e.g., placing the scenario in the context of working for a company and being given a task by management or having to learn to work a group of people in which one is unfamiliar with that is symbolic of reality) or how the way in which the course was structured (e.g., providing less guidance) put students in the role of a scientist. Having responsibility over ones lab drawer was in reference to having to keep track and ensure everything was accounted for in ones assigned drawer (e.g., beakers, test tubes, etc). The course element identified most frequently by participants as contributing to SPC development was the Collaborative Environment. This involved interactions with either a participant's team, GTA, or in a rare mention their neighboring teams. The purpose of including Table 4.12 is to provide a tabulated general overview of the themes explored within the main chapter. Table 4.13 expands on the subcodes associated with the Collaborative Environment course element code and provides the frequencies of their association with development of specific skills by participants. Beyond the collaborative environment, participants mentioned many other aspects of the laboratory course(s) that contributed to development of SPCs. This is displayed in Table 4.14, that shows the frequencies of subcodes for several other frequently mentioned course element codes associated with SPC development: Open Inquiry Learning, Working with Data, Conceptual Learning and Application, and Using Course Resources. A subcode not touched 156 upon in the main chapter was that of Data Collection & Record Keeping under the main code of Working with Data. This code entailed engaging in the act of collecting data and maintaining a record of experimental steps and observations. Additionally, a breakdown of course activities and project scenarios that were reported as supporting development of SPCs can be located in Table 4.15 and 4.16 respectively. All corresponding tables can be found on pages 158 – 161. A small number of participants spoke of development in the course but offered no further commentary on their experience for communication (n = 2), teamwork (n = 3), prioritization & time management (n = 1), and work ethic (n = 1). Exploring the frequencies displayed below it is clear that many components of the course were beneficial in contributing to the perceived development of EDCs as mentioned through participant experiences. While some elements of the course were mentioned more often than others, the rich display of experiences offers an insight into how perceptions varied across SPCs. 157 Table 4.12. Course elements that contributed to perceived SPC development (expanded)(n). Skill Perceived as Developed in Course(s) Communication Skills (n = 36) 31 6 11 6 — Teamwork Skills (n = 37) 35 5 11 3 — Problem- solving & Critical Thinking (n = 40) 16 29 10 16 2 Prioritization & Time Management (n = 26) 14 2 12 2 18 8 — 1 1 1 — — — — — 1 1 — 1 1 — — — — — 3 2 13 10 9 8 6 — — 1 — — — 1 — — — 3 1 — Course Element that Contributed to Perceived Development Collaborative Environment Open Inquiry Learning Course Activities Course Projects Individual Task, Assignment, & Project Management Conveying Information to Others Real World Context Conceptual Learning & Application Working with Data Using Course Resources Troubleshooting Reasoning & Sensemaking Persevering Through Course Responsibility Over Lab Drawer Learned Lab Techniques Key Color Percentage (%) 81 - 100 61 - 80 41 - 60 21 - 40 0 - 20 158 Table 4.13. Frequency of subcodes within the Collaborative Learning Environment course element category and perceived association with supporting development of specific SPCs (n). Course Elements within the Collaborative Environment Communication & Collaboration with Team Communication & Collaboration with TA Subcodes Delegating & Coordinating Learning to Work with Others & Be a Team Player Listening & Sharing Ideas Recognizing Importance of Communication in Teamwork Asking Questions & For Help Building Relationships Solving Problems & Finding Solutions as a Team Learning from Others General Communication & Collaboration with Team Asking Questions & Receiving Guidance General Communication & Collaboration with TA TA Emphasizing/ Facilitating Teamwork Skill Perceived as Developed in Course(s) Communication Skills (n = 36) 22 Teamwork Skills (n = 37) 25 Problem- solving & Critical Thinking (n = 40) 3 Prioritization & Time Management (n = 26) 13 22 8 6 1 2 4 2 11 1 — 1 1 5 — 3 — 8 1 — 7 — — 6 2 1 1 — 1 — — 2 — — 14 9 13 4 3 2 2 5 5 1 — 159 Table 4.14. Frequency of subcodes within the Open Inquiry Learning, Working with Data, Conceptual Learning & Application, and Using Course Resources course element codes and their association with development of problem-solving & critical thinking skills (n). Course Element Open Inquiry Learning Subcodes Designing & Planning Experiments Less Guidance Provided in Course Investigating Driving Problem & Finding Resources Finding Solutions to Proposed Problem Working with Data Data Collection & Record Keeping Conceptual Learning & Application Using Course Resources Analyzing, Interpreting, & Displaying Data Learning New Concepts & Information Applying Knowledge Lab Manual Course Instrument & Tools Reading Materials Provided in Course (General) CLUE Textbook Software & Application Use (e.g., LoggerPro) Skill Perceived as Developed in Course(s) Problem-solving & Critical Thinking (n = 40) 22 16 10 3 1 10 4 11 4 3 2 1 1 Table 4.15. Course activities that contributed to skill development (n). Skill Perceived as Developed in Course(s) Activity Planning Document Laboratory Notebook Presentations (Oral/Poster) Reports (Informal/Formal) Quizzes Communication Skills 6 2 Teamwork Skills 5 1 4 4 — 5 2 — Problem- solving & Critical Thinking 6 1 2 2 — Prioritization & Time Management 2 2 4 9 1 160 Table 4.16. Project scenarios that contributed to skill development (n). Skill Perceived as Developed in Course(s) Project Course Scenario GCL1 Volume & Communication Skills 1 Teamwork Skills 1 Problem- solving & Critical Thinking 4 Prioritization & Time Management — Temperature of a Gas Unknown Ionic Compound Food Dye Spectroscopy Separation of Commercial Plastics Iodine Clock Reaction Soaps & Detergents Building a Calorimeter Artificial Kidney Stones GCL2 — 3 — — 2 — — — 1 — — 1 — — 2 5 4 1 4 2 — — — 1 — 1 — — 161 A.IV.17. Development of Self-motivation & Technical Skills Although the percentage of participants who reported development out of the overall percentage of participants who said a skill was needed was relatively low for work ethic (41%) and technical skills (37%) in comparison to the other top four SPCs reported in the main chapter, the instances or experiences in the course that led to development of these skills are further explored here. The same coding scheme investigating course elements that contributed to skill development outlined A.VI.16 was applied here. While no instances of development surrounding these skills were considered major themes, shared by 60% or greater of participants who spoke of development, minor themes shared by a handful of participants emerged (Table 4.17). These themes are explored below, with codes related to development shared by less than 20% or n = 1 participants not being investigated further. Development of work ethic skills encompassed four minor themes – 1) persevering through various course aspects (e.g., having to wake up and attend the course), 2) maintaining integrity in work, 3) independent management of tasks and assignments, and 4) collaborating with others. Technical skill development contained five minor themes – 1) using course resources (e.g., working with instruments & tools), 2) working with data (e.g., analyzing & interpreting data), 3) learning lab techniques, 4) working collaboratively, and 5) development sparked by doing something new. These themes and the number of participants associated with each are found in Tables 4.18 – 4.19. All corresponding tables can be located on pages 169 – 171. Of the n = 19 participants who mentioned that they developed work ethic skills in the general chemistry laboratory courses, n = 1 participant did not offer any specific examples of development. Similarly, of the n = 13 participants who spoke of technical skill development in the courses, n = 3 did not expand on how skill development related to the course. The perceived influence of online learning regarding development of these skills will be explored further in Chapter V. Minor themes associated with development of work ethic and technical skills are explored below. Work Ethic Skill Development One of the most prevalent themes that emerged when students spoke of work ethic skill development was having to persevere through a variety of general course aspects including - having to show up to class and endure a 3-hour course (Onyx), having to maintain focus while 162 taking a course in which they feel they do not excel at the subject and are not interested in (Onyx), having to maintain motivation in a course that is perceived to be hard (Codi), having to motivate oneself to participate in order to receive a good grade, and having to remain dedicated in order to do well. [. . .] But I definitely have to work extra hard in chemistry, I don't know, for what reason, I'm not good at it, I just never got interested by it, like everything else, so that makes focusing on it even harder. [Later in interview] But the being able to go through a three hour lab session where you're just sitting at your desk, that requires focus, especially when it's something that you're not particularly good at, that requires a lot of focus [. . .] like the course in general, just being able to sit, sit and learn about something or try and do something for three hours. (Onyx, Work Ethic) [. . .] also self-motivation, I do feel like because this is one of like, my, I would say, like, one of my harder classes, sometimes I'm just like, "Oh my gosh, I just don't want to do it." And kind of like, so I'm, I'm trying to keep at it, like, never give up type of things, so I think by the end of it I'll have learned more of, of self-motivation towards my own skills. (Codi, Work Ethic) Another minor theme that may be seen more as a general aspect attributed to taking college courses in general and touched upon by participants relating work ethic skill development to the course, was that these skills were developed through making sure they did not cheat through using internet resources in place of doing coursework themselves or copying another person’s work when completing assignments. Participant thoughts surrounding this particular theme was only related to the skill of performing with integrity that was provided on the skills list post-prompting. For one participant, they spoke of using their teaching assistant as a resource instead of engaging in unethical behavior surrounding performing with integrity. Jordan’s response below represents an example of this theme, also touching on how this was especially important when taking the course online. Here performing with integrity, also, because it is an online class. And you know, with an online class, you're exposed to internet all the time, it's kind of 163 important to perform with integrity, because, you know, you have all these resources available to you, but you know, that you're not supposed to be using other papers or documents for your lab. And because the purpose of the lab is to learn for yourself, not like, you know copy someone else's paper for a 9/10 grade, right. So, integrity is important here. Because it's good for you. Yeah, it's really just like, focusing on that, the, what's the word? Like the goal of the course. Yeah. So, we, especially with online chem lab, I think it's really important. (Jordan, Work Ethic) Other secondary themes surrounding components of the course that contributed to development of work ethic skills included individual task management, collaboratively working with others, and course activities. Having to meet deadlines and maintain motivation to stay on track, showing up to class prepared through reading relevant materials prior to coming to class, and having to focus completed the goals set for the day (Reign). Jordan spoke of how they had to maintain self-motivation in order to complete their assignments, a skill that was heightened in addition to taking the course in an online learning environment. As seen within Onyx’s quote below, this theme also corresponded to and was related with specific activities, in particular when generating reports. This included mention of formal reports and general lab reports (e.g., referring to no specific report in the course) as well. Chemistry lab, I think self-motivation stands out to me. Because, you know, we're kind of removed from the classroom setting. And with online classes, it's really easy to just kind of, you know, let it sit, and then we forget about it. Especially with the class that's not synchronous, like, this one is synchronous, which is makes it better. But even it's still online, and deadlines are, you know, not super strict. So having self-motivation to keep yourself on track and keep communicat[ing] with your teammates, and the TAs, is really, really important in being successful. (Jordan, Work Ethic) Yeah, I think even though back when, back when I was in [GCL1], there was a lot of focus going because I knew when I went to the lab, I knew what my lab 164 goal was for the day. So, like I, I kind of had something to focus on, while I was physically present in the lab. (Reign, Work Ethic) [. . .] focusing on meeting deadlines, the other two things I said, like that's all part of it. For example, tonight, I think we have the formal report, conclusion, or maybe its results, that we have to do, I think that's the same thing. But those have to be done and so that's meeting the deadlines and focusing [. . .] (Onyx, Work Ethic) An additional course element related to development of work ethic skills was collaboration with teammates within in the course. Students spoke of having a responsibility to their teammates and having to contribute as encouraging work ethic skills as seen in Bailey’s quote below. Bailey also references how the CATME surveys, used to generate student teams, aided skill development. Yeah, I think it's good. And even the CATME [. . .] surveys are, are good. Because if you're not good, in your teams, you will be marked down. So, you have to be good. And you have to take responsibility in the working team. (Bailey, Work Ethic) Additionally, having to delegate and coordinate through checking in with each other and dividing responsibilities between teammates helped development of work ethic skills. This can be observed in a continuation of Bailey’s quote above, where Bailey continues to elaborate below on how having responsibilities to their team included having to divide tasks for each project. For the planning documents, and the one, the notebook. It's like, we divide the work as like, every project, somebody handle something. Like [. . .] for the first part I was handling the first two questions or something? And yeah, as this and the following projects, I think, yeah, we're dividing the responsibility. And we are moving it across the group members. (Bailey, Work Ethic) Skylar spoke of how they initially struggled with the first project but through a culmination or problem-solving & critical thinking, teamwork, and self-motivation they were 165 able to overcome this. This led to application of the code solving problems & finding solutions as a team due to all aspects including teamwork coming together to help them through this initial project. I think, definitely the first assign- the first project wasn't that hard, and we understood it, but and we used like, I think we didn’t understand it at first and then we used problem solving and critical thinking to get it. And we also use our teamwork skills, and self-motivation. (Skylar, Work Ethic) Technical Skill Development When speaking of technical skills, the most mentioned course element that contributed to development of these skills was through use of course resources, in particular using or learning about course instruments and tools. Participants spoke of developing technical skills through learning about how to use instruments or equipment and for some students who were able to take the course in-person for a portion of the semester (Sp20 participants) this included being able to physically work with an instrument. The instrument most mentioned in regard to technical skill development was the spectrometer. Stevie offers evidence to support how use of spectrometers in the course aided development of these skills. I think getting to use this, this professional equipment like the spectrometers that was that was pretty cool. Just being able to see how that worked and just sort of getting a general idea of the kind of equipment that they use in the actual academic scientific world [. . .] (Stevie, Technical Skills) Additional course resources aided development of technical skills were software applications such as LoggerPro or Microsoft Excel. Students engage with these software when collecting and analyzing data. This can be observed through Blake’s statement below. Well, I do feel that I am going to need them if I'm going to work as resear- research assistant, or a lab assistant. Because the technical skills I learned here, it really helped me in a lot of things in how to use Excel Microsoft, which I believe will help in any future job I"ll have, or in daily life. So, I do believe that technical skills help. (Blake, Technical Skills) 166 Other experiences reported by students to aid technical skill development occurred when participants were collecting, analyzing, and interpreting data. Through instances in which students used mathematical equations, as well as learning to understand and draw conclusions from spectra, technical skills were developed. Interestingly, when one participant was asked how they developed professional/technical skills within the course when prompted with a skills list, they found that recording observations and taking concise notes within the course helped develop this skill. However, this participant also noted that technical skills are specific to the career in which they are used, stating that learning laboratory techniques and data analysis specific to the chemistry laboratory course are different than determining treatment of patient disorders. This finding may be due to the participant separating professional/technical skills into two individual skills and viewing record keeping as a “professional” skill they will need that was developed in the course versus technical skills such as lab techniques that are not as prevalent to their career. This is reflected in Arbor’s statement below. [. . .] I feel like clear communication and recording of stuff, I feel like labs require you extensively to know your observations, which I think it's gonna be really useful, at least in my career choice, like really observe what's going on, and take detailed, yet concise notes on them. [. . .] And I think developing professional and technical skills is specific to any career. In chemistry, I'm learning skills such as how to use all the tools, which are required, to like analyze chemicals, which might be slightly different in my career of choice, it's just like how to analyze people, or how to treat certain disorders and stuff like that. (Arbor, Technical Skills) Other students referred to technical skill development in a manner to be expected from a general chemistry laboratory course – learning of laboratory techniques. For most of the students who reported this, there was not a specific technique referenced, however, one participant spoke of learning how to perform a filtration. Indy expresses how the course allowed them to learn laboratory skills in general. [. . .] And then some of the lab skills I probably need again, like just the actual lab techniques of things that we work on in class. (Indy, Technical Skills) 167 Collaborating with others was not related to one group in particular (e.g., a participants immediate team or graduate teaching assistant) but embodied being willing to ask questions and listen when learning how to use instruments, receiving help from the instructor on how to use software such as Excel, and learning technical skills through a process that included working with teammates. Technical skills were also perceived to be learned through doing something new. By being introduced to new materials, instruments, techniques, and exploring new areas that had not yet been learned students were able to recognize technical skill development. Sam exemplifies how being introduced to something new led to development of technical skills. [. . .] being just in [GCL1], I was introduced to a lot of new materials, a lot of new equipment that I didn't know about. One of them specifically was the spectrometer, which I was like, "What is this?" and how to work it [. . .] (Sam, Technical Skills) 168 Table 4.17. Course elements that contributed to perceived work ethic and technical skill development (n). Skill Perceived as Developed in Course(s) Work Ethic (n = 19) 4 1 4 1 6 — 1 1 7 4 — — — Technical Skills (n = 13) 3 1 — — — 1 4 5 — — 4 3 1 Course Element that Contributed to Perceived Development Collaborative Environment Open Inquiry Learning Course Activities Course Projects Individual Task, Assignment, & Project Management Conceptual Learning & Application Working with Data Using Course Resources Persevering Through Course Making Sure Not to Cheat Learned Lab Techniques Doing Something New Ownership Over Work Key Color Percentage (%) 81 - 100 61 - 80 41 - 60 21 - 40 169 Table 4.18. Frequency of subcodes within the Collaborative Learning Environment course element category and perceived association with supporting development of work ethic and technical skills (n). Course Elements within the Collaborative Environment Communication & Collaboration with Team Communication & Collaboration with TA Additional Collaborative Elements Not Related to One Group in Specific Subcodes Delegating & Coordinating Learning to Work with Others & Be a Team Player Solving Problems & Finding Solutions as a Team Learning from Others Asking Questions & Receiving Guidance Being Willing to Listen to Others Asking Questions and Receiving Help Skill Perceived as Developed in Course(s) Technical Skills (n = 13) — Work Ethic (n = 19) 2 2 1 — 1 — 1 — — 1 1 1 1 Table 4.19. Frequency of subcodes within the Open Inquiry Learning, Working with Data, Conceptual Learning & Application, and Using Course Resources course element codes and their association with development of work ethic and technical skills (n). Subcodes Course Element Open Inquiry Learning Less Guidance Provided in Course Working with Data Data Collection & Record Keeping Analyzing, Interpreting, & Displaying Data Conceptual Learning & Application Using Course Resources Learning New Concepts & Information Lab Manual Course Instrument & Tools Reading Materials Provided in Course (General) Software & Application Use (e.g., LoggerPro) Skill Perceived as Developed in Course(s) Work Ethic (n = 19) 1 Technical Skills (n = 13) 1 1 — — 1 — 1 — 1 3 1 — 4 — 2 170 Table 4.20. Course activities that contributed to work ethic skill development (n). Activity Planning Document Presentations (Oral/Poster) Reports (Informal/Formal) CATME Survey Skill Perceived as Developed in Course(s) Work Ethic 1 1 3 1 171 A.IV.18. Student perceived value of skills beyond their planned career goal Table 4.21. Value of skills beyond career goal (n). Places in Which Skill is Perceived to be Valuable Any Job/Career College Career & Campus Life General Life Skill Communication Skills (n = 44) 19 13 Student-perceived Competency Problem-solving & Critical Thinking (n = 48) 12 6 Teamwork Skills (n = 46) 19 10 Prioritization & Time Management (n = 42) 17 21 8 8 10 14 172 A.IV.19. Skill Desired in Course Table 4.22 displays the numerical values concerning the number of participants who wished to have a skill present within the general chemistry laboratory courses but felt that they were not given an opportunity to work on. Table 4.22. Skills participants desired but were not given an opportunity to develop in the project-based general chemistry laboratory courses (n). Skill Lacking in Course that Students Desired Development Of Occupation-specific Skills No Skills Listed Problem-Solving & Critical Thinking Intrapersonal Skills Education & Learning Skills Other Skills Interpersonal Skills # of Participants 10 8 5 3 2 2 2 173 REFERENCES Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society, 57(1), 289–300. Bridgeman, B., Burton, N., & Pollack, J. (2008). Predicting Grades in College Courses: A Comparison of Multiple Regression and Percent Succeeding Approaches. Journal of College Admission, 19–25. Cao, H., Song, Y., Wu, Y., Du, Y., He, X., Chen, Y., Wang, Q., & Yang, H. (2023). What is Nursing Professionalism? A Concept Analysis. BMC Nursing, 22(1). https://doi.org/10.1186/s12912-022-01161-0 Ghasemi, A., & Zahediasl, S. (2012). Normality Tests for Statistical Analysis: A Guide for Non- statisticians. International Journal of Endocrinology and Metabolism, 10(2), 486–489. https://doi.org/10.5812/ijem.3505 Guraya, S. S., Guraya, S. Y., Doubell, F. R., Mathew, B., Clarke, E., Ryan, Á., Fredericks, S., Smyth, M., Hand, S., Al-Qallaf, A., Kelly, H., & Harkin, D. W. (2023). Understanding Medical Professionalism Using Express Team-based Learning: A Qualitative Case-based Study. Medical Education Online, 28(1). https://doi.org/10.1080/10872981.2023.2235793 Liu, R. (2022). Statistical Guideline #7 Adjust Type 1 Error in Multiple Testing. International Journal of Behavioral Medicine, 29(2), 137–140. https://doi.org/10.1007/s12529-022- 10070-0 Lumley, T., Diehr, P., Emerson, S., & Chen, L. (2002). The Importance of the Normality Assumption in Large Public Health Data Sets. The Annual Review of Public Health, 23, 151–169. https://doi.org/10.1146/annurev.publheath.23.100901.140546 Manion, M. (2001). Engineering Professionalism and the Imperative of Sustainable Development. International Symposium on Technology and Society Technology. Telfor, B. (2017). 16 Essential Health and Science Skill Sets. https://natsci.msu.edu/sites/_natsci/assets/File/Current%20Students/College%20Career%20 Services/Option%201%20Skill%20Sets%20with%20Definitions%20for%20Website- CNS%20Articulation%20Exercise%20Website%20Posting%20BT_ST.pdf Westrick, P. A., Marini, J. P., Young, L., Ng, H., Shmueli, D., & Shaw, E. J. (2019). Validity of the SAT for Predicting First-Year Grades and Retention to the Second Year. https://satsuite.collegeboard.org/media/pdf/national-sat-validity-study.pdf Whelan, C. J. (2008). The Paradox of Professionalism: Global Law Practice Means Business. Penn State International Law Review, 27(2), 465–494. 174 CHAPTER V: ONLINE & OUT OF THE LAB: EXPLORING THE IMPACT OF VIRTUAL LEARNING ON 21st-CENTURY SKILL DEVELOPMENT IN PROJECT- BASED GENERAL CHEMISTRY LABORATORY COURSES Introduction As the COVID-19 pandemic swept the globe in the early months of 2020, college instructors were forced to rapidly adapt their courses to accommodate government-mandated quarantines and social distancing measures (Centers for Disease Control and Prevention, 2023; Kathy Katella, 2021; OECD, 2020; U.S. Department of Defense, n.d.). Overnight, in-person instruction, facilitated through face-to-face interactions, needed to be transformed to instruction taking place entirely online. Instructors had to quickly adapt their methods of instruction to a remote modality that was unfamiliar to both them and their students (Means et al., 2020). As a result, instructors had to reflect on barriers presented by online learning when determining the best pedagogical practices to implement online. Instructors were tasked with determining how best to keep students active and engaged (Castelli & Sarvary, 2021), while also being cognizant of internal and external stressors (such as distracting home environment, familial/work obligations, and differing time zones)(Ramachandran & Rodriguez, 2020). At the same time, students reported a lack of self-motivation (Means et al., 2020) and difficulties accessing the technology necessary to engage in remote courses (Bharadwaj et al., 2023; Castelli & Sarvary, 2021; Morgan, 2022) as hindering their ability to learn. Reliance on unfamiliar technology to disseminate course materials, with little time to learn how to use these resources also presented challenges for instructors (Díez-Pascual & Jurado-Sánchez, 2022). Further, many instructors had to determine how to best transfer in-person pedagogical practices to a remote learning environment (Sansom, 2020), including whether to conduct their courses synchronously or asynchronously (Castelli & Sarvary, 2021). While college administrators and educators worked diligently to adapt to these circumstances and ensure that students could effectively continue their education remotely, the job market also experienced a catastrophic blow that was predicted to result in long-term changes (Ice et al., 2021). Amid record high layoffs and subsequent unemployment rates (Centers for Disease Control and Prevention, 2023), a majority of the workforce had to become accustomed to working remotely (Parker et al., 2022). While stay-at-home orders and quarantine measures have since been lifted, the way people work post-COVID parallels the shifts in course modalities found in higher education, where hybrid (mix of in-person and online) and online (or remote) 175 modalities became more prevalent following the pandemic. Even though remote work existed before the pandemic closures, a report released by PEW Research found a shift in preferences towards working from home, citing benefits such as better work/life balance and ease of work completion despite drawbacks such as less connection to co-workers (Parker et al., 2022). This preference aligns with college students’ desire for the integration of more hybrid, blended, and remote courses, along with incorporation of digitally accessible materials (Bharadwaj et al., 2023). These trends show that there is a desire for the inclusion of remote practices, emphasizing the need for higher education to embrace remote learning technologies, while also better preparing students for careers that may include remote work. This study aims to take a deeper look into an intersection of online learning and career preparation, in particular the influence that remote learning had on students’ perceived ability to develop valuable 21st-century skills in project-based general chemistry laboratory courses. Chemistry Laboratory Learning Online Transitioning laboratory courses to an online format, in response to the pandemic, posed a particularly challenging problem because hands-on experimentation is typically a focus of the curriculum (Bruck et al., 2010; Hofstein & Lunetta, 2004). A variety of creative approaches were employed to facilitate remote learning in chemistry laboratory courses, which included use of a) at-home laboratory kits (Sansom, 2020), b) kitchen chemistry activities (Schultz et al., 2020) c) recorded videos, tutorials, and associated data (Sansom, 2020; Wild et al., 2020), d) online simulations (Accettone, 2022; Youssef et al., 2020), d) virtual reality (Williams et al., 2022) and e) online databases (Youssef et al., 2020). Many instructors questioned how they could continue to give students an immersive hands-on learning experience if only engaging with and observing chemical phenomenon at a distance. The answer for some was at-home laboratory kits purchased from a third-party distributor and delivered to students (Sansom, 2020) or kitchen chemistry activities using readily available materials typically present in anyone’s kitchen (Schultz et al., 2020). While these were deemed useful methods for continuing experimentation outside of the college laboratory, integration of at-home laboratory kits, and even kitchen chemistry activities, into the curriculum imposed additional costs for students, raising accessibility issues. Safety was also a concern because students were no longer under direct supervision. 176 The use of video recordings or simulations presented different challenges. Many students believed that watching videos of recorded experiments and engaging with virtual simulations were less valuable than in-person experiences and hindered their ability to learn (Accettone, 2022; Jeffery & Bauer, 2020). However, another study investigating an analytical techniques laboratory course reported better student performance, measured through higher grades and pass rates, for a remote laboratory course relative to the corresponding in-person course (Díez-Pascual & Jurado-Sánchez, 2022). Students in this study viewed video recordings as a valuable resource that supported their interest and engagement. Considering the costs and risks associated with traditional chemistry laboratory courses, combined with limited evidence about their contributions to learning, raises questions about their value (Bretz, 2019). Assumptions about the value and goals of laboratories, in general, are often tied to the perceived importance of practical hands-on experiences (Bretz et al., 2013; Bruck et al., 2010). Even before the COVID pandemic forced laboratory courses online, virtual laboratories in which students engage in computer-based experiences that simulate performing hands-on laboratory tasks had begun to appear in chemistry curricula (Ali & Ullah, 2020). These types of courses are just one of many resources used for remote learning that could potentially allow students to engage with a wider variety of experimental scenarios and techniques, provide a cost-effective alternative to maintaining an in-person laboratory, support online learning, and eliminate the risk associated with in-person laboratories (Ali & Ullah, 2020; Brinson, 2015). In a synthesis of empirical research in 56 studies, across STEM disciplines, that compared learning outcomes for remote/online and virtual laboratories to traditional in-person laboratories, Brinson reported that the majority of the 56 studies analyzed showed evidence of greater (65%) or equal (24%) learning in the remote and virtual laboratories (Brinson, 2015). However, a limitation in this work was that nearly all (n = 53) of these studies assessed content knowledge while only 4 assessed inquiry skills. While many chemistry instructors believe learning in the laboratory occurs primarily through hands-on engagement to build laboratory skills and application of conceptual knowledge learned in lecture to activities in laboratory, instructors also acknowledge that development of skills such as teamwork and critical thinking are learning outcomes of these environments (Bretz et al., 2013; Bruck et., 2010). These skills are among those identified by employers as valuable career competencies, indicating there may be more to be learned from general chemistry 177 laboratory courses than technical skills, which could contribute to preparation for the working world. Online Learning & Career Readiness While preparation for future careers is becoming a measure of student success in higher education, gaps remain in how well academic institutions are meeting employer expectations (Carlson, 2022). Employers consistently identify the following skills as important to success in today’s workforce: communication, teamwork, and problem-solving & critical thinking (Finley, 2023; Kondo & Fair, 2017; National Association of Colleges and Employers, 2023; National Research Council et al., 2011). Unfortunately, many employers also report deficiencies in these employer-desired competencies (EDCs) among recent graduates (Finley, 2023; The Economist Intelligence Unit Limited, 2014). Considering that preparation for career and workplace success is perceived by many as an important goal of a college education (Fermin & Pope, 2003; Gallup Inc & Strada Education Network, 2018; Lee, 2019), and enrollment in online courses is continually increasing (Allen & Seaman, 2013; Garrett et al., 2023; National Center for Education Statistics, 2023; Seaman et al., 2018), understanding how online learning may contribute to building EDCs is important. Introductory STEM courses reach large numbers of students from a range of majors with varied career goals and can have a significant impact on students’ college experience (President’s Council of Advisors on Science and Technology, 2012; Seymour & Hunter, 2019). Most STEM majors enroll in general chemistry laboratory courses early in their undergraduate studies, and we posit such courses could be used to kickstart EDC development that could be built upon in subsequent courses. Notably, in pre- and post-surveys of undergraduate students enrolled in chemistry laboratory courses at two Midwestern universities, a statistically significant number identified “To prepare for the career I want to pursue” as a most important learning goal (Santos-Díaz et al., 2019). This study, however, did not explore what preparation for career meant to the students surveyed. Hansen & Overton found that chemistry graduates considered generic skills of teamwork, problem-solving, and time management as more useful on the job than knowledge of chemical terminology or skills/experience with chemical instrumentation (Hanson & Overton, 2011). Continuing this work, Galloway observed that students anticipating going into careers that were perceived to use little or no chemistry recognized value in these generic skills significantly higher than chemistry-specific skills (Galloway, 2017). This 178 contrasted with students pursuing occupations believed to have moderate to high involvement of chemistry who found both generic and chemistry-specific skills to be of high value with no statistical significance between the two sets of skills. Another study conducted in Singapore found that both employers and students seeking chemistry degrees ranked communication skills, work experience, and teamwork skills highly, showing that students understand the importance of these skills for finding employment (Yasin & Yueying, 2017). While the majority of students in this study indicated that they developed practical skills (91%) and technical skills (76%) in their degree program, less than half indicated that they had developed teamwork (38%) and communication (27%) skills. These findings support the necessity of assessing how the chemistry curriculum, particularly laboratory courses, is preparing students for careers. The transition to remote learning in March 2020, due to the COVID pandemic, occurred prior to the second round of data collection for a qualitative study exploring student perceptions of development of skills they reported to be of value to their future career (referred to as student- perceived competencies or SPCs) in project-based general chemistry laboratory courses. This provided an opportunity to expand the study to explore student perceptions of how online learning affected the development of these skills. Perspectives were provided by students who experienced the transition from in-person to emergency remote learning in Spring 2020 and those who experienced fully online laboratory learning in Summer 2020 and Fall 2020. This chapter explores explicit connections made by interview participants between the online learning environment and development of specific SPCs, both positive and negative. It should be noted that this study did not assess student proficiency in the skills needed for future careers. This study will answer the following research question: Research Question 1. (RQ1) What impact did the online learning environment have on students’ perceived ability to develop student-perceived competencies (SPCs) in project-based general chemistry laboratories? Theoretical Framework The social constructivist theory of learning, which posits that knowledge is actively constructed through social interactions and cultural influences, provides a theoretical basis for this study. Vygotsky proposed that cognitive development was the product of active engagement by the learner (Vygotsky, 1978), a position shared by Jean Piaget (Wadsworth, 1996). While both 179 scholars believed in the central role of the learner in crafting their knowledge, Vygotsky challenged Piaget’s belief that learning was independent of the social and cultural world in which the learner was situated and proposed that it is precisely these social influences and experiences that shape one’s knowledge and learning (Vasileva & Balyasnikova, 2019). Vygotsky proposed that there is a gap between what the learner knows and can do independently and what the learner cannot yet do even with assistance, which he called the zone of proximal development (ZPD) (Ferguson, 2007). Learning within the ZPD can occur by receiving guidance from a more knowledgeable instructor or peer. The project-based learning environment investigated within this study placed a strong emphasis on collaboration within the course’s learning community, facilitating the use of a social constructivist approach (Krajcik & Blumenfeld, 2006). The transition to online teaching and learning during the pandemic created challenges to fostering these social interactions between students and their peers or instructors (Baldock et al., 2021), that are central tenets to project- based laboratories and supporting learning within the ZPD. Although maintaining a social constructivist approach online posed a potential problem, the act of carefully curating collaborative activities in which students could engage with one another can still promote such a learning environment that continues to aid in 21st-century skills (Agopian, 2022). When administering psychology courses online, Agopian implemented team-based activities, oral presentations, and online discussion forums to maintain a learning environment and community reflective of social constructivism. Darby & Lang further emphasized the importance of social interactions and community building in online courses and the influence it can have on the construction of knowledge, stating: [. . .] a well-designed course will also provide opportunities (and incentives) for learners to interact with one another, both to help each other learn and to build that sense of community. When these two forms of presence have been established, the learners in the course are more likely to engage in the kinds of active, collaborative processes that help them construct new knowledge through their cognitive presence (Darby & Lang, 2019). Some of the ways that Darby & Lang proposed fostering this social community mirrored that of Agopian’s teaching method of incorporating team projects, exchange of thoughts and 180 ideas through small group discourse, and use of discussion forums. A few of these elements can be observed in the curriculum of the project-based general chemistry laboratory courses in this study, in which students engaged in team-based projects, were encouraged to communicate and collaborate with teammates during and outside of class hours using a variety online resources (e.g., google documents, group chats), and were asked to convey information collected and synthesized as a team to the rest of the class. Beyond the aim of promoting peer interactions, cultivating relationships between students and their instructors is another important aspect to building an online learning community (Archambault et al., 2022). These relationships are integral to social constructivism and project-based learning, where instructors are meant to guide students through the learning process. These roles were established by the graduate student teaching assistants (GTAs) that facilitated the general chemistry laboratory courses in this study. The transition to online posed additional challenges in supporting project-based laboratory learning where “doing” is considered an important part of laboratory learning. This concept of “doing science” is defined by engaging students in a process of inquiry using practices in which they can apply knowledge and skills in a way that will result in a deeper understanding of scientific knowledge and be reflective of what scientists do (National Research Council & Committee on a Conceptual Framework for New K-12 Science Education Standards, 2012). To get students engaged in these practices, the eight scientific and engineering practices were defined by the National Research Council in A Framework for K – 12 Science Education. These practices include: 1. Asking questions (science) and defining problems (engineering) 2. Developing and using models 3. Planning and carrying out investigations 4. Analyzing and interpreting data 5. Using mathematics and computational thinking 6. Constructing explanations (science) and designing solutions (engineering) 7. Engaging in argument from evidence 8. Obtaining, evaluating, and communicating information Although these practices were intended for use in K – 12 education, they are of value to higher education as well. Each of these practices have been identified, by Carmel et al., as components of the various project’s students are given throughout the project-based general 181 chemistry laboratory courses in this study (Carmel et al., 2019). The most prevalent scientific practices that emerged in every project included asking students to design or plan experiments, analysis and interpretation of data, and construction of explanations and engaging in argumentation from evidence. While maintaining opportunities for students to engage in scientific practices online presented potential challenges, activities continued to be structured in a way to encourage use of these practices. Methodology This study aims to derive meaning from student experiences in their general chemistry laboratory courses developing pertinent 21st-century skills and the influence online learning had on supporting or hindering development of these skills. Framing skill development in those of which students ascribe importance to their future career goal, this study seeks to show the importance of in-person learning on student experiences and how these experiences are seen as advantageous and beneficial for students’ future careers. To investigate the influence of online learning on students’ perception of skill development in project-based general chemistry laboratory courses, a mixed-methods approach was followed by using both qualitative and quantitative methods of data collection and analysis (Plano Clark & Ivankova, 2018). Although a mixed-methods approach was used, this study focuses heavily on the qualitative methods employed, with quantitative methods used as supplemental analysis to further investigate comparisons of interest. The qualitative methodology used in this study centered around a transcendental phenomenological approach. Use of this qualitative methodological approach involves five primary steps: 1) defining the phenomenon and sample of interest, 2) collecting data using qualitative methods (e.g., interviews), 3) identifying key pieces of data (e.g., excerpts) and generating themes, 4) engaging in epoché throughout the study to confront researcher biases, and 5) presenting results through use of participant voices, focusing on descriptive explanations of experiences. The way in which this framework was applied to this study is explored below. Defining the Phenomenon & Sample of Interest Phenomenological studies aim to explore human experiences with the purpose of gaining a better understanding surrounding a particular phenomenon (Creswell, 2013). The phenomenon under investigation in this study was student experiences developing SPCs in online project-based general chemistry laboratory courses. Investigating this phenomenon was not the primary goal at 182 the inception of this study. This study initially explored students’ perceptions of development of SPCs in in-person project-based general chemistry laboratory courses. During preparation for the first round of interviews, emergence of the COVID-19 pandemic and the subsequent switch to online learning presented a new area of exploration - the influence of online learning on SPC development. To investigate both SPC development and the effect of online learning, the sample selected included students enrolled in large-enrollment general chemistry 1 (GCL1) and 2 (GCL2) laboratory courses for STEM majors at a large Midwestern university from Spring 2020 through Fall 2020. The majority of students in the sample population were freshmen and sophomores, and under 4% had a declared major in chemistry. Progression from GCL1 to GCL2 is not required for most majors. Concurrent enrollment with the corresponding general chemistry lecture course is not required as the courses operate independently. To provide additional context for the phenomenon investigated, the course structure and adaptations made for the online modality are described below. Course Structure & Adaptation for Online Learning The general chemistry laboratory courses in this study follow a cooperative project-based learning curriculum (Carmel et al., 2017, 2019; Cooper, 1994). Each project is framed by a scenario (e.g., propose a method for dissolving kidney stones or recreate the color profile of a name-brand beverage), and students are tasked with working collaboratively in teams of 3 - 4 students to plan investigations to answer questions or design solutions. The four multi-week projects, in each course, require students to work in teams of 3 or 4 to plan and carry out experiments, make observations and collect data, analyze and interpret data, and argue from evidence or use data to design solutions. Students communicate their methods and findings for the four projects in a variety of formats including informal and formal written reports and oral and poster presentations. During this study, the learning environment evolved from in person to fully online in response to the COVID-19 pandemic. Classification of these different learning environments, adopted for this study from Hodges et al., were emergency remote learning (ERL) and online learning (OL)(Hodges et al., 2020). Emergency remote learning corresponds to the rapid mid- semester transition from in-person to remote learning during Spring 2020. Subsequently, the laboratory courses were restructured to support fully online learning during Summer 2020 and Fall 2020. 183 Maintaining as many of the essential elements of the project-based curriculum as possible was a primary goal in adapting the general chemistry laboratory courses for online learning. Figure 5.1 shows a comparison of key student activities and the instructor role between the in- person curriculum and the online adaptation. The main difference in OL was that students could no longer carry out their planned procedures and engage in hands-on experimentation. In its place, students were provided with videos of experiments to allow them to make observations, simulations that generated measurement data with error based on student input, and Excel workbooks containing previously collected data. Students in both the ERL and OL environments continued to collaborate in teams through Zoom breakout rooms to design and plan experiments, engage in data analysis and interpretation, and complete each course project with a report or presentation. There were two key differences between student experiences in the ERL and OL semesters. Students in the ERL semester worked with their teammates in-person and performed the experiments that they planned with their team for just over half the semester before the transition to remote learning and remained with their original team as they finished the semester remotely. Results from GCL1 and GCL2 were combined and presented together because the two courses have the same overarching structure. Figure 5.1. Comparison of student activities and instructor role in the from in-person and online learning modalities. Semi-structured interviews provided the data for this study. Over 3 semesters, the 3,772 students enrolled in GCL1 and GCL2 were invited to participate via email with extra credit offered as an Data Collection 184 incentive. Students who either did not volunteer or were not selected for an interview could complete a survey for extra credit. Institutional review board (IRB) approval was obtained for the interview protocol and survey. Study participants provided consent after being informed of their rights. All participants were 18 years of age and older. Interviews were conducted in three consecutive semesters (Spring 2020, Summer 2020, and Fall 2020). From the 781 students who volunteered for interviews, purposeful sampling based on declared major, and in a later semester class standing (Fall 2020), was used to select participants. Interviews took place remotely via Zoom and were initially transcribed verbatim using a third-party transcription software (OTTER.ai) and then cleaned by two of the authors. All participants were assigned unique identification numbers and non-binary pseudonyms to protect their identities. A total of n = 54 interviews, which lasted from 16 to 63 minutes, were conducted. During preliminary analyses, one participant was removed from the sample due to their college education being unrelated to career advancement. Additionally, poor internet connection caused interruptions during questioning and made transcription difficult. Removal of this participant resulted in a final sample of n = 53 participants. Independent samples t-tests found no statistically significant differences between interview participants and the course population based on SAT scores. Participant demographics were similar for the ERL (n = 23) and OL (n = 30) samples (A.V.1). Nearly all participants (50 of the 52 with age data) were 18-20 years old, and 75% were freshmen or sophomores by credit, as expected for an introductory class. More participants identified as female (57%) than male (43%)(only binary choices were permitted in institutional data). At the time of this study 53% of undergraduate students at the institution identified as female. Race and ethnicity of participants were Asian (n = 1, 1.9%), Black or African American (n = 7, 13.2%), Hispanic/Latinx (n = 6, 11.3%), International (n = 3, 5.7%), Two or More Races (n = 1, 1.9%), White (n = 28, 52.8%), and not reported (n = 3, 5.7%). The distribution of course enrollments for the ERL and OL samples differed; 61% of the participants in the ERL sample were enrolled in GCL2, while 60% of the students in the OL sample were enrolled in GCL1. Across the study, 27 students from GCL1 and 26 students from GCL2 were interviewed. Since both courses followed the same project-based format, enrollment in a particular course was not expected to impact the findings of this study, which did not explore 185 differences between the courses. In the sample, the most common planned careers were in the health & medical professions (e.g., doctor or nurse) (n = 33, 62%). The remaining participants had planned careers in engineering & related subspecialties (e.g., mechanical engineering or computer science) (n = 10, 19%) or other careers (e.g., forensic scientist or medical laboratory scientist) (n = 10, 19%) categories. Interview Protocol Interviews were conducted following a semi-structured format that allowed interviewers to ask follow-up questions to prompt elaboration and/or clarification of responses (Herrington & Daubenmire, 2014). Participants were first asked about their career goal, prior experiences and exposure to their intended career, and perceptions of the skills needed to be successful in their planned career. During the next segment, participants were asked to relate experiences in their general chemistry laboratory course as well as outside the course that helped them develop skills that they think are important for their career. Next, participants were shown a list of skills desired by employers of STEM graduates (Telfor, 2017) and given a chance to talk about any skills from the list not mentioned previously and opportunities, if any, to develop these skills within the course. Interviews concluded with explicitly prompting participants to share their thoughts on how remote learning may have impacted skill development in their laboratory courses, although many participants had already discussed online learning and its impact on skill development without prompting earlier in the interviews. Questions were refined iteratively over the three semesters (Sp20, Su20, Fa20) to improve clarity, but the general line of questioning outlined above was unchanged. The most significant change was refinement of the skills list provided in each semester based on participant responses in the prior semester. Identifying Key Pieces of Information & Generating Themes Interviews were coded using thematic analysis to identify emerging themes (Nowell et al., 2017). Initial analysis included identifying excerpts relevant to the research question, namely where participant responses related skill development to the online learning environment in GCL1 or GCL2. From these excerpts, emerging ideas from participants’ responses were summarized to produce an initial set of codes (e.g., inability to perform data collection). Related codes were then grouped to create representative parent codes (e.g., inability to perform data collection and less control over experimentation were organized under the parent code lack of hands-on). 186 Inter-rater reliability (IRR) was used to assess the validity and reliability of the coding scheme (Nowell et al., 2017). Two independent coders applied codes to interview segments and compared the results to determine the percent agreement and Brenan-Prediger’s Kappa (𝜅BP), which is a measure of researcher agreement in applying a code versus chance application of a code by the two coders. Brennan-Prediger’s Kappa is deemed to be sufficient or near perfect with values of 0.80 or greater (Brennan & Prediger, 1981; Gwet, 2016; Landis & Koch, 1977), with these parameters adopted for this study. Prior to reaching IRR, coding disagreements prompted combining, reorganizing, renaming, or addition of codes to better reflect findings in student responses. Once sufficient IRR was reached, one independent coder coded the remaining responses. After all interviews were coded, co-occurring or overlapping codes were analyzed to identify patterns or themes in the dataset (e.g., many participants were found to express a lack of hands-on resulting in hindered development of technical skills). Engaging in Epoché & Confronting Researcher Biases Along with finding meaning through human experiences, transcendental phenomenology seeks to observe the data with a “fresh pair of eyes” through what is known as epoché, where the researcher identifies their experiences, thoughts, feelings, and ideas surrounding the phenomenon under study with the aim of not only making them clear but also setting them aside when assessing the data (Creswell, 2013; Moerer-Urdahl et al., 2004; Moustakas, 1994). To provide transparency, the primary author’s (BE) experiences with the courses in this study and how they addressed potential bias are outlined below. BE was a graduate teaching assistant (GTA) in either GCL1 and GCL2 for five semesters. Following data collection, BE taught GCL1 in an online modality in Spring 2021 and served as an undergraduate learning assistant (ULA) coordinator for GCL1 for an in-person course in Spring 2022. As a ULA coordinator, BE worked with the course coordinators to recruit and hire ULA’s, manage training resources for GTAs and ULAs, and administer surveys to assess student satisfaction and gather information on how course experiences could be improved. Based on the experience of teaching GCL1 online and its contrast with in-person experiences in the general chemistry laboratory courses before and after, BE recognized the presence of personal notions about online laboratory learning including a lack of connection between the members of student teams as well as with the instructor and an overall lack of engagement from students. 187 Throughout this research, BE aimed to set aside these biases and approach the data with an objective lens to focus on what was present within the data. While the early stages of this study did not explicitly incorporate epoché, BE always endeavored to let the data speak for itself by documenting and distancing themselves from what they thought participants were saying to focusing on what the participants were saying. Several techniques were used by the research team to ensure epoché occurred during analysis and interpretation. Memos were used to document thoughts and uncertainties about the interpretation of participant statements, and the researchers engaged in frequent discussions to ensure interpretations were justified by the data. In presenting the results of this study, we aim to provide full transparency on the methodology, rich descriptions of the coding scheme, and a robust discussion of findings. Statistical Analysis for Comparison of Skills Perceived as Valuable Career Competencies Comparison of the skills reported as valuable to a student’s career goal and course modality were explored using chi-square test of association (𝜒2) for each individual skill. Student responses were coded using dichotomous variables to represent a student either 1 = reporting or 0 = not reporting a skill, throughout the entirety of the interview, as needed for their anticipated career goal. To control for multiple comparisons, the Benjamini-Hochberg method of adjustment was used (Benjamini & Hochberg, 1995) and any p-values that maintained significance after adjustments were followed using Phi (φ) to report effect size (Cohen, 1988; Reid, 2022). Chi- square analyses was carried out using SPSS 27.0 (IBM Corp, 2020) and p-value adjustments were performed using SAS 9.4 (SAS Institute Inc., 2023). Results & Discussion Continuing with the methodological framework of transcendental phenomenology, the results focus on the descriptions of main themes that emerged from the phenomenon of participant experiences in developing valuable career competencies in an online laboratory learning environment. We begin by reviewing key findings from a related study that investigated student perception of skills needed for their planned career and explore any related statistical findings associated with changes in course modality investigated in the study herein. We then explore themes, presented in this current study, associated with the impact of online learning on perceived skill development using student voices captured through interview excerpts and vignettes to support findings. In presenting interview excerpts, “um”, “like”, and pauses have been excluded for clarity. 188 In the quotations presented as evidence, participants are identified using non-binary pseudonyms and the learning environment (ERL or OL) they were sampled from to provide context. Development is a key term that is referenced throughout the results and is used to describe instances in which a skill was learned, applied, or improved upon. Prior to delving into the results, it is important to acknowledge that participants mentioned many other experiences related to online learning and skill development outside of GCL1 and GCL2 (e.g., other classes, college in general, or their personal lives), which will not be explored in this study. Skills Needed for Career Goal & Perceived Development in the Lab Courses In a related study, interview participants identified a list of key skills necessary for their future career endeavors, referred to as student-perceived competencies (SPCs). Seven main skill sets were identified – interpersonal skills, intrapersonal skills, occupation-specific skills, problem- solving & critical thinking, prioritization & time management, education & learning skills, and other. Categorization of specific skills into each of these skill sets is outlined in the Appendix of Chapter IV. Of the skills identified in the prior study, the most prevalent SPCs participants recognized was communication and teamwork skills from the interpersonal skill set, work ethic from the intrapersonal skill set, problem-solving & critical thinking, prioritization & time management, and technical skills from the occupation-specific skill set, as being needed for their planned careers. These skills were found to align with skills highly sought by employers in the 21st-century workforce (Kondo & Fair, 2017; National Association of Colleges and Employers, 2022, 2023; National Research Council et al., 2011; Organisation for Economic Co-operation and Development, 2019; World Economic Forum & in collaboration with The Boston Consulting Group, 2015). Figure 5.2 shows the six SPCs most frequently identified by participants as needed for their planned career disaggregated by learning environment, with ERL to represent Sp20 and OL representative of Su20 and Fa20. Although OL participants appear to have more frequently recognized all six EDCs, no statistically significant differences were detected between the course modalities (A.V.2). While online learning did not appear to influence student recognition of the skills needed for their career goal, thematic analysis, presented in the study herein, captured the variety of experiences students had developing skills these skills while engaging in the remote project-based general chemistry laboratory courses. Our exploration of how online learning 189 impacted SPC development focuses on the six skills most frequently identified by participants as needed for their future career (as represented in Figure 5.2). n i h t i w s t n a p i c i t r a P f o t n e c r e P ) % ( t n e m n o r i v n E g n i n r a e L 100 90 80 70 60 50 40 30 20 10 0 Emergency Remote Learning Online Learning Figure 5.2. Student-perceived competencies selected as needed for planned career goals by interview participants. The Influence of Online Learning on SPC Development Figure 5.3 shows the number of interview participants who discussed online learning as supporting, hindering, and/or having no change on development of the six SPCs. Participants may not appear in Figure 5.3 for primarily one reason –they did not discuss a skill in the context of online learning. Two primary themes emerged. When participants discussed the relationship between the online environment and technical skill development, they were unanimous that online learning hindered technical skill development, largely due to the lack of hands-on opportunities (Theme 1). For the other five SPCs: communication, teamwork, problem-solving & critical thinking, prioritization & time management, and work ethic skills, participants were relatively split as to whether the online environment hindered, supported, or had no influence on development (Theme 2). Participants can be represented in Figure 5.3 as believing the online modality to 190 both support and hinder development of SPCs. However, this was only observed when students were talking about the influence of online learning on problem-solving & critical thinking skill development and will be further explored when discussing Theme 2. Additionally, one participant found that teamwork was mostly unaffected in the online modality, however, they found also found this learning environment to support development of these skills as well, as explored further in Theme 2. This led to this participant being represented in Figure 5.3 as both believing no change to skill development occurred as a result of being online and citing instances in which online supported development of teamwork skills. These themes will be discussed further below, with a numerical breakdown of themes present in Appendix (A.V.3). Communication Skills Teamwork Skills Work Ethic 4 3 3 3 5 3 2 1 5 Problem-solving & Critical Thinking 6 4 4 Priotization & Time Management 1 3 3 Technical Skills 14 Hindered Development No Change Supported Development Figure 5.3. Number of the 53 participants interviewed who associated online learning with supporting, hindering, or having no change on development of SPCs (n). Theme 1: Online Learning Negatively Impacted Technical Skill Development Traditionally, general chemistry laboratory curricula have prioritized helping students build technical skills through gaining experience with techniques (e.g., dilutions), equipment (e.g., beakers, graduated cylinders), and instrumentation (e.g., UV-visible spectroscopy)( Bretz et al., 2013; Bruck et al., 2010; Hofstein & Lunetta, 1982; Reid & Shah, 2007) used in chemical experimentation. Technical skills are not isolated to the chemistry laboratory, however, and is used to describe a plethora of skills that are specific to different disciplines or industries (Lamri 191 & Lubart, 2023). This is exemplified in participant responses that encompasses and expand beyond laboratory skills and techniques to include technological or computer skills (e.g., coding software & programs) and other hands-on skills (e.g., hand-eye coordination working with tools of the trade). A large number of interview participants identified these skills as key career competencies (66%, n = 35). Although a sizeable number of participants reported these skills to be valuable, not all found the course to be beneficial in providing opportunities for technical skill development and many of the reasons behind this surrounded the online learning environment in which the course was taking place. While many participants said that technical skills were required for their career, 14 participants said that they didn’t have opportunities to develop technical skills in GCL1 or GCL2, with a majority of participants (n = 9, 64%) identifying the lack of hands-on experiences as the cause. An additional 4 participants said that while they had some opportunities to develop technical skills while taking the course (e.g., working with laboratory equipment prior to the transition or working with software when taking the course online), they felt that being online limited development of these skills due the loss of hands-on experiences. Participants spoke of no longer being able to carry out planned procedures and engage in data collection or having access to laboratory equipment and instruments as obstacles to developing of technical skills. In place of conducting their own experiments, students were given videos of experiments, predefined datasets collected at an earlier time by laboratory assistants, and access to web-based simulations. This resulted in students no longer being given control over the data that was collected or the methods that were used. Several subthemes emerged as participants elaborated on the negative impact that online learning had on technical skills development. These themes include lack of connection and agency in the experimental process, concerns about inadequate preparation for future courses and career, and unfulfilled expectations for a hands-on experience. The first subtheme that emerged in some participant comments was being distanced from the experimental process and a lack of ownership over their work, as exemplified in Taylor and Grey’s comments below. [. . .] building skills wise, you didn’t get to perform anything by hand anymore, so that definitely impacted the skills you can learn. I mean you watch the experiment, but you didn’t get to do it yourself. (Taylor, Technical Skills, ERL) 192 [. . .] in all reality, we’re come up with general, kind of, procedures, and then we just have to work with the data we’re given and like the procedure they went with. So, a lot of the times the procedure that we wrote in like the preparation document is different than the one that was actually carried out. [. . .] (Grey, Technical Skills, OL) Further, Taylor felt that no longer being physically present to carry out experiments, following the transition to ERL in Sp20, meant they could no longer engage in the process of trial and error, an experience Taylor described as an enjoyable and beneficial learning experience. When asked what watching versus performing the experiment themselves did to their ability to acquire technical skills, Taylor responded: I mean it takes away all the physical stuff. Also, all the problems you are gonna incur. They’re not gonna happen. [. . .] A lot of trial and error just goes away. [. . .] I think anytime you screw up in life [. . .] that’s just one more way that you’re gonna remember not to do it again and so I do enjoy it sometimes when you mess up, as long as you’re in an okay setting because then you’ll never do it again. (Taylor, Technical Skills, ERL) It was this ability to manipulate physical components in the lab environment, accompanied by the possibility of making mistakes, that Taylor felt would offer them opportunities to learn, correct, and avoid mistakes in the future. The second subtheme that came to light within participant responses was concerns about being inadequately prepared in learning laboratory techniques needed for future lab courses because of the lack of ability to be in the laboratory environment, expressed by Indy below. Um this one might not always apply, but we didn’t get to like, do all of the labs [. . .] hands on, so if I had another lab class, I’m [. . .] not gonna know how to do some of the lab techniques and stuff [. . .](Indy, Technical Skills, ERL) Blaine saw a fully online lab experience as a detrimental to preparing for their career and felt that having direct experience with laboratory skills and techniques would have been beneficial in preparing for a career in research & development. 193 But if I’m looking more holistically, like, you kind of need that, a lab experiences. [. . .] I am considering going into like research & development [. . .] so it’d be nice if I got like more hands-on experience [. . .] (Blaine, Technical Skills, OL) That last subtheme seen within student responses, discussing a lack of hands-on being detrimental to technical skill development, was that some had expressed desires or expectations of being physically in the laboratory and performing experiments that were unmet in the online environment. For example, Morgan identified specific experiments that she would have liked to perform and expressed the general sentiment of wanting an in-person experience. Grey saw remote learning as losing the feeling of being in a lab setting, where hands-on work was viewed as an integral component to that work. [. . .] performing the acid-base neutralizations or the salt dissolutions [. . .] that would have been great to do. And I kind of said this earlier [. . .] it’s sort of a course that I would like to have in-person. (Morgan, Technical Skills, OL) [. . . ] for labs specifically [. . .] it definitely feels like an element is lost being online. [. . .] just ‘cause lab, like especially the chemistry ones [. . .] are so based on actually doing [. . .] the work, the procedures, and stuff like that. And so just not doing that [. . .] you just lose like the feeling of a lab [. . .]. (Grey, Technical Skills, OL) Stevie offered another perspective by commenting on missing out on the opportunity to work with instruments like the infrared spectrometer, which is something that “actual chemists” do. The IR, infrared spectrometer, we didn’t get to use it this semester. But I think that would have been helpful being able to see how that worked, because [. . .] it’s a very expensive piece of equipment that [. . .] actual chemists, actual [. . .] scientists use for their research. (Stevie, Technical Skills, ERL) 194 When talking about technical skill development, Indy, who experienced the transition from in-person to emergency remote learning, compared the impact on lecture to those of the lab course. [. . .] I feel like the labs are like the thing I’m missing out most on, because lectures [. . .] you watch them in person or you watch them at home and you take notes, but you’re [. . .] basically doing the same thing. [. . .] But like the labs [. . .], especially chem over physics lab, are [. . .] more hands on. [. . .] I feel like we missed out more on like some of the lab techniques. (Indy, Technical Skills, ERL) Although technical skill development is not the primary focus of the project-based curricula in GCL1 and GCL2 courses, students have opportunities to develop technical skills as they carry out planned investigations when these labs are offered in person. Comments from participants in this study, who didn’t have these opportunities in the online environment, speak volumes to the importance they attribute to gaining these skills through hands-on experiences. These findings are reflective of the literature in which both instructors and students believe laboratory skills and techniques are an outcome of laboratory learning (Bretz et al., 2013; Bruck et al., 2010; DeKorver & Towns, 2015; Petillion & McNeil, 2021). Only one participant offered suggestions for how to improve technical skill development in online labs. When asked how the course could better incorporate these skills remotely, Jamie described that akin to virtual simulation, in which they can effectively carry out their procedure, as being beneficial. Maybe they can develop [. . .] a software that we can just click on the thing and click on the water, and we can design how much water we can put [. . .]. (Jamie, Technical Skills, OL) Other studies have explored the use of apps to simulate the laboratory experience virtually by allowing students to manipulate quantities of reagents and virtually interact with laboratory equipment to provide a more immersive remote laboratory learning experience and enhance practical lab skills (Hawkins & Phelps, 2013; Williams et al., 2022). Although the loss of experience handling laboratory materials is of concern in remote courses, Williams et al. 195 found positive gains between students pre-lab expectations and post-lab experiences, when measuring meaningful learning across cognitive and affective domains, in an organic chemistry laboratory course using virtual reality (VR)(Williams et al., 2022). Students who anticipated being confused about how laboratory instruments worked and having a feeling of nervousness when handling chemicals prior to engaging in the course, reported a decrease in these feelings after their VR experience. Further, Hawkins and Phelps found that there was no statistically significant difference in general chemistry 2 student scores measuring conceptual knowledge, or their ability to apply laboratory techniques to construction of a voltaic cell battery, when comparing students who were either enrolled in a traditional hands-on laboratory course or one that employed the use of virtual simulations. While simulations may not completely replace the experience of being in a laboratory, they can enhance the virtual experience for distance learners (Ali & Ullah, 2020) and other situations where access to in-person laboratories is limited. Providing virtual learning experiences that allow students to manipulate lab equipment and chemicals may alleviate student feelings of being disconnected from the data provided and increase their perceived autonomy. However, enhanced virtual experiences alone cannot address students’ desire to physically be in the laboratory. Despite students talking about the online environment hindering the development of technical skills, particularly the physical activities typically associated with performing experiments, some students reported the ability to learn technical skills in their courses. These student responses did not indicate a relationship to online learning as a supportive structure to building these skills, but that it was within student engagement with course materials in which these skills were developed. A clear pattern emerged in the types of technical skills perceived to be developed and how they were developed when comparing the responses from the ERL and OL learning environments (A.V.4). Students in the fully online courses were more apt to report technical skill development through using software (e.g., excel, LoggerPro) and building a knowledge base of technical skills (e.g., learning about techniques or instruments) as represented by Blake’s quote. Technical skills, I feel like having background about like, the equipment we're going to use or like being able to imagine what equipment we'll use or like 196 how to use, like whether it's Logger Pro, or Excel in Microsoft that really helps in the course [. . ]. (Blake, Technical Skills, OL) Whereas those in ERL were able to connect development to physically using instruments (e.g., working with spectrometers) and applying lab techniques in the first half of the course prior to switching to the online format as seen in Sam’s comment below. [. . .] Um, one of them specifically was like the spectrometer, which I was like, "What is this?" and like how to work it, but in my lab group like I would always be working it because I got the hang of it really fast [. . .] (Sam, Technical Skills, ERL) Since most students in these general chemistry laboratory courses do not plan to pursue a career in chemistry, becoming proficient in traditional laboratory techniques may not be as beneficial to students’ career pursuits as having a chance to analyze data using different software (e.g., Microsoft Excel or LoggerPro). Interestingly, despite most participants having career goals that did not involve chemistry related career goals, many wanted to be in the lab and build these skills. In some instances, this may be related to concerns about preparation for future courses as expressed by Indy. These findings are reflective of the literature in which students who engaged in their remote laboratory courses reported a lack of preparation due to missing out on opportunities to learn hands-on laboratory techniques and the inability to learn through trial and error when watching pre- recorded videos (Petillion & McNeil, 2021). Another possible reason that students underscored the importance of hands-on experiences and the loss of technical skills is that students find physically participating in laboratory activities to be fun and more engaging as will be discussed later on. Regardless, across ERL and OL learning environments, technical skill development was negatively impacted by the remote environment. Theme 2: Contrasting Perceptions of EDC Development Perceptions of how online learning affected development of the five other SPCs, were divided between supported, hindered, or no change to development (Figure 5.3). As will be explored below, how participants perceived the impact of the online learning environment on SPC 197 development depended on whether they saw challenges associated with adapting to a remote lab course as beneficial or detrimental to the learning experience. Communication Skills In discussing the impact of remote learning on the development of communication skills, participants talked about how they had to learn to convey information in a clear and concise manner. They spoke of having to adapt how they communicated to ensure that their ideas could be understood by teammates when they couldn’t necessarily show what they were talking about. [. . .] I feel like, essentially, being online, not being able to [. . .] physically show what you’re talking about, you must be kind of clear and efficient in a way, about you’re talking. So, you can help [. . .] others understand your perspective easier and just working together [. . .]. (Arbor, Communication Skills, OL) [. . .] I think [online] definitely helps with your communication, because you have to learn how to explain things when the person can’t actually see it. Like you can’t do the experiment in front of them to show you how it’s done. (Rori, Communication Skills, ERL) Navigating team dynamics online, from getting acquainted to checking in to make sure the work is being completed, supported development of communication skills. [. . .] definitely, for this semester, it’s way different than the past years, because we’re all on Zoom and working online. But I think gradually, we, I guess, warmed up to each other, me and my group and were able to talk about things easier and point things out and don’t feel like we are stepping on each other’s toes. [. . .]. (Corey, Communication Skills, OL) [. . .] communication skills, especially, like, everything online. It’s been important to reach out and make sure everyone is doing stuff. (Jasper, Communication Skills, OL) 198 Riley talked about adapting their method of communication in a different way, where they had to situate themselves in an alternative perspective to learn how to write about a reaction that they were unable to carry out themselves. “[. . .] it’s just a lot harder to write about a reaction if you haven’t done it yourself, even though it’s a good skill to have actually [. . .]” (Riley, Communication Skills, ERL) While these participants noted that the courses provided opportunities to build communication skills, others focused on the challenges. The online modality created barriers to collaboration and communication with teammates. For example, Blaine explained how easy it was for a teammate to disengage through a muted mic and blank screen. This was a frustrating experience, in which Blaine spoke of not only having to pick up additional work to compensate for this teammate but also feeling that questions could easily go unanswered. A key part of the course was encouraging students to ask questions and rely on teammates to navigate through problems or difficulties. For Blaine, these important and potentially beneficial components were lost online. [. . .] in my chem lab [. . .] one of the girls in there sits in the zoom for three hours with no mic, no video, I didn’t even think she was there [. . . ] until our TA came in one day and like directly asked her something [. . .]. She never writes in the document. [. . .] honestly, its just really hard in the online environment because I can ask the question and people can literally just not answer it, because I am not going to see them directly. (Blaine, Communication Skills, OL) Lev, who had the experience of working with teammates for half the semester before the transition to ERL, also felt the collaborative component was lost online because the environment was more independent and less team-based. [. . .] you’re not in the teamwork aspect, of course, because you’re just, the new online perspective is like we watched a video and took our own notes. But 199 beforehand [. . .] we would do the experiment together. (Lev, Communication Skills, ERL) Online learning also impacted the ability to build communications skills through asking for and receiving help. Kai compared their in-person experience with communication skill development in GCL1 with their online experience in GCL2. When Kai asked their teammates for help in GCL1, being in the same room permitted teammates to supplement verbal explanations by showing what they meant, which Kai found helpful. Kai noted that the situation was different in GCL2 where communication with teammates was somewhat difficult. [Interviewer asking how they developed communication skills in the course [. . .] for GCL1, [definitely], because we were actually in class I was able to ask them for help on certain things I was confused on [. . .] and I could receive help that way, because they could show me and also speak at the same time. (Kai, Communication Skills, OL) Communicating project results in different formats, including oral and poster presentations, was another important feature of the two courses. Bailey commented on the lack of an audience presence during presentations because no one turned their cameras on, indirectly pointing to important an aspect of building communication skills that was absent in the online environment - audience feedback on how well information was being communicated. [. . .] I have done one presentation [. . .]. It was good. But it was like using zoom and nobody even opened the camera. So like it was, if you, if you want to eat [. . .] nobody will notice [. . .] because there's no cameras. So I think for the current circumstances, that's the best that TA can do for the presentation, but it's not as good as we want it, I think. (Bailey, Communication Skills, OL) This split perception of the impact online learning has on communication skills is shared in the literature (Kinsky et al., 2021), in which Kinsky et al. observed that students either felt that adapting their methods of communication was a beneficial learning experience or that the limited opportunities for face-to-face interactions in remote courses resulted in a decrease of communication skill development. While the exact pedagogies of the communications courses 200 investigated in Kinsky et al.’s study is unknown, these differing results continue to be reflective in this study, where course pedagogy focuses on social interactions through collaborative efforts and communication of results in various formats. Teamwork Skills For the participants who mentioned that online learning supported the development of teamwork skills, adapting the way in which they worked with teammates and utilizing technology to aid collaboration was key. Jordan and Jesse found that having to learn how to operate within the constraints of working online provided a valuable experience. Um, I think I’ve kind of learned how to connect with people more in a like a video-based setting. And in the future that might be important, because we’re going to be stuck inside for a while. So, learning how to collaborate here in a virtual setting has been really useful for me. (Jordan, Teamwork Skills, OL) And you do learn to work around obstacles of, especially as people move off campus, um, how to get together and make things work and use technology to your advantage and make teamwork work in a non-traditional way of sitting down together and getting something done. [. . .] Like Zoom or like Google Hangouts or even just like having a shared document [. . .] (Jesse, Teamwork Skills, ERL) Zuri, an ERL participant, talked about using Zoom meetings or FaceTime to maintain a collaborative working environment with their teammates. Um, well since we've been at that like this stay at home thing [. . .] we use our, the lab time that we're given, we all hopped on zoom and used our time wisely and got our slideshow done and then one other day, we all contacted each other and got on FaceTime and went through the entire presentation once. (Zuri, Teamwork Skills, ERL) Working remotely necessitated the use of video conferencing, group chats, and online shared documents. With remote and hybrid work becoming more common in the workplace, learning how to work collaboratively in remote environments can provide valuable career 201 preparation. Some students, however, perceived that online learning impeded building teamwork skills. For Stevie, the transition from in-person to ERL in Sp20 resulted in a breakdown of communication within their team, which hindered work on group assignments. Um, well, specifically in lab it's, it's obviously made it a lot more difficult because the group environment has, has essentially disappeared. I haven't really been communicating all that much with my group. Because a lot of the assignments have been... well, there, there was one assignment that was meant to be a group assignment that was just turned into an individual assignment. Two labs we weren't able to do. [. . .] But yeah, I think, I think sort of the, the development of those skills have definitely been dampened by the transition. (Stevie, Teamwork Skills, ERL) Another ERL participant, Riley felt that working with their team became challenging after transitioning online, and lamented the loss of in-person collaboration. I think the working as a team has been a little harder just cuz of what's on the world right now, a lot of its like virtual and I feel like there's no replacement for like being in lab with your team and like that human aspect of it. (Riley, Teamwork Skills, ERL) Despite the negative impact that remote learning had on teamwork skills discussed by some participants, these same participants provided examples of how their general chemistry laboratory course supported development. Online learning can act as a disruption to social interactions and norms that students are often accustomed to when entering traditional classrooms (Darby & Lang, 2019) and because of this care must be taken to create a learning environment that facilitates these interactions. However, determining how to best encourage these social interactions can be challenging, with many additional variables, such as lack of engagement, general dislike for group work, or student preferences for cameras to be turned off, to consider (Castelli & Sarvary, 2021; Donelan & Kear, 2023). Allowing students to engage in these opportunities is important for growth of not only teamwork skills but all SPCs explored in this chapter, as emphasized by the social constructivist approach taken in this study. 202 In an expansive literature review, by Donelan & Kear, of the issues online learning presents in group work and the strategies to better facilitate these interactions, they found that some of the potential ways to overcome problems that emerged were through outlining clear expectations, having a purposeful design of projects to facilitate collaboration, providing ample guidance throughout the process, and emphasizing the potential gains and benefits of engaging in work with others (Donelan & Kear, 2023). These strategies were used in the laboratories explored in the study herein, by having projects that require input and participation from all team members to complete, implementing team contracts, defining and allocating team roles, having GTAs monitor team activity throughout synchronous class times and end-of-project surveys, and stressing the importance of collaboration in being successful in the course. However, even with these strategies in place, issues still occurred as displayed by commentary provided by interview participants. Like the findings in this study, there are variable experiences of engaging in teamwork online in the literature. Some studies reported that students had positive experiences engaging in teamwork online (Díez-Pascual & Jurado-Sánchez, 2022; Vergara-Castañeda et al., 2021), while others found online learning was not as conducive to such tasks (Jeffery & Bauer, 2020). This shows the need to investigate and employ different strategies to facilitate interpersonal skill development in remote laboratory courses. Work Ethic Work ethic skills, such as self-motivation and focus, were tested online, but for some participants these obstacles proved to be a learning experience that promoted skill development. Participants felt that the online lab environment was demotivating and distracting; however, several discussed how they had to motivate themselves to get the work done. [. . .] its been hard for me [. . . ], I’m at home and I’m like “I don’t want to do it” kind of thing, but my self motivation has been down. But at the same time, I think in [GCL1] [. . .] I have to push to get it done, so [. . .] I have to force myself to get my motivation up. (Codi, Work Ethic, OL) I don’t have anyone to motivate me right now. I just need to motivate myself and [. . .] try to do good in the class right now [. . .] So, to keep up with the class, I need to motivate myself and [. . .] get myself to work on it, even if I 203 am not invested in doing it. [. . .] and I really improved my [. . .] self- motivating skills because of that. (Shiloh, Work Ethic, OL) But I think one big thing that comes out of this is self-motivation. Because you don't really have your team to [. . .] encourage you to like get going on stuff. It's kind of like, “Oh, I'll work on that tomorrow.” And like, you're not concerned about like getting something out to your team earlier. So I think that I've learned a lot about motivating myself and telling "Get it done early. Just start doing it now. (Rowan, Work Ethic, ERL) Being at home impacted focus because participants felt they could push off academic tasks more easily, had greater access to distractions (e.g., TV, cellphone, getting on social media), and could engage in other tasks (e.g., making food). Several noted that it was easier to be attentive when in person and working at staying focused in the online environment helped them to build this skill in the presence of distractions. [. . .] definitely during 162 for being online, its so easy to just “Okay, we're not in person, let me just get on my phone, we're not really doing anything, let me just be distracted.” Or like, especially being at my house, I have to get out of my house, because, you know, there's a big TV in front of me, it's like, “Okay, let me just watch some TV or I can have it on in the background, I'll, I'll be focused”, and then it turns into I'm not actually paying attention [. . .]. The online learning especially uh 162 definitely, but online in general, like, really test your focus, too, because there's so many distractions at your own house, it's, it's tough sometimes. [. . .] especially if we were in an in-person lab, because you know, you can have your phone out and I, I personally leave it like, I don't even have it in my pocket, I would just leave it in my backpack. And I don't know, being in the lab setting, you just feel like, okay, I'm here to do an experiment, whereas if I'm sitting on my couch in my living room doing 162 virtual lab, it's like, I don't know, I don't really feel like I'm in a lab like I am, but I'm not, you know, I don’t know, it's just a different vibe. (Morgan, Work Ethic, OL) 204 Limited interactions with teammates impacted Onyx’s ability to focus and stay on task. “Oh, I'd say, I mean, actually, great example is every, every single lab session [. . .] usually, I'll be not working on something for an hour or two, that's when I'll do like my math quiz or whatever, I'll take a nap, sometimes I'm really tired. Um, but... like, sometimes during that lab session, when nobody's talking, we're all thinking or writing or whatever, I'll like [. . .] snap out of it, [. . .] or I'll like zone off. Um, because just nobody's talking in the Zoom, I literally just, like stop writing, stop thinking about it. And that's an example of when I don't focus. [. . .] But [. . .] being able to go through a three-hour lab session where you're just sitting at your desk, that requires focus, especially when it's something that you're not particularly good at, um, that requires a lot of focus. [. . .] the most I've been away from my computer is probably like, 15 minutes, grabbing food, making food, maybe and then bringing it to the computer, stuff like that. But um, the, the labs require a lot of focus. The three hour, three hours a day of chemistry, of actually being in the lab that requires focus for me, like, that's a lot of focus. And, and I and I still can't even like usually do it all. Like, I have to take that 10 minute break or whatever. (Onyx, Work Ethic, OL) Billie discussed how development of the work ethic subskills drive and self-motivation suffered online because they found the online course less enjoyable and sometimes confusing. In addition, there were many differences from their previous positive in-person experience in GCL1. Billie also stated that when they were in-person they had more of a drive to attend and do well in the course. “Honestly, I, I'm going to say 161 [participant utilized drive more], honestly, a little bit more, because I liked it a little bit better. I like just being in the lab experience and everything. It was honestly a lot of fun. Like, I really had, like a good time, like going to class. Like it wasn't a drag. And I also had like a good group too. So it was just like a win win in that class. I ended up having a really, like good time. But, um, I feel like when I enjoyed the class more, I had 205 more of a drive to go and more of a drive to like, do well in it, if that makes sense [. . .] So, [online] made it a little bit more confusing for sure. Just because there was just a lot of changes. I mean, it was completely different from you know, I mean, performing everything in a lab setting and being able to be there in person. It's just not the same experience. And it really, I think, put sort of a damper on the excitement. Honestly, like, I wasn't, I was just kind of like, "Oh, great, like, you know, chem, ahhh." Like, you know, it's just like, it's another three-hour meeting. I mean, there were times where I definitely like, didn't mind being there, or I wasn't like, you know, like, "This sucks," but I just um, I definitely think it didn't make it as exciting, if that makes sense.” (Billie, Work Ethic, OL) Studies have shown the effect external factors, such as the online learning environment, can have on student motivation and engagement (Aguilera-Hermida et al., 2021; Kinsky et al., 2021; Means et al., 2020). An investigation by Aguilera-Hermida et al. into the motivation of students in behavioral and health science fields, across four different countries, saw a decrease in motivation after the shift to remote learning (Aguilera-Hermida et al., 2021). In another study conducted by Digital Promise & Langer Research Associates, capturing the perceptions of undergraduate students across the nation during the pandemic, it was observed that students felt a lack of motivation and decreased interest in their courses (Means et al., 2020). Taken together, with the results of this study, it can be observed that a lack of motivation and engagement due to online learning is not isolated to one discipline, affecting university students throughout higher education. This current study shows that these hardships can be framed in either a positive or negative light and depending upon student perception this can result in believing online to support or hinder work ethic skill development. Problem-solving & Critical Thinking Skills Many participants cited the lack of hands-on experiences as the reason the online environment hindered development of technical skills. In contrast, some participants discussed how not being able to carry out experiments forced them to think in different ways about how to explain experiments, which supported development of problem-solving & critical thinking skills. 206 [. . .] I feel like a lot of creativity is needed when it comes to writing any type of lab report or anything, because you have to look at a video of someone doing something and try and picture yourself in their place. [. . .] So, I feel like it takes an element of creativity to figure out the details of it and to figure out how to still learn something from the lab. [Later in interview] it has just really helped me be able to think from different points of views, not only the whole idea that is online, and it’s not me doing the lab, I have to pretend to be someone else doing it [. . .]. (Ash, Problem-solving & Critical Thinking, OL) [. . .] critical thinking, since we are not actually doing the lab, kind of just putting yourself in the headspace that you are doing the lab. (Corey, Problem- solving & Critical Thinking, OL) [. . .] in this lab, since its online, its kind of, it not the critical thinking where I am necessarily solving a problem, but it’s the critical thinking of trying to be able to do and articulate and the planning docs, the notebook pages, how something would go based on what we have available to us online. [. . . ]. This is like critical thinking in terms of “How do I, how do I try to make it seem like I’m actually going through these motions and doing the experiment? (Onyx, Problem-solving & Critical Thinking, OL) Interestingly, while Ash discussed on how the lack of hands-on experiences in the online labs supported development of the problem-solving & critical thinking subskill of creativity by asking students to write about experiments from a different or distanced perspective, they also felt that development of the subskill of analytical thinking was hindered. Because student teams were given the same or similar data, they spoke of no longer having to rely on their own intellect or reasoning to support why they got the results they did. [. . .] when you’re given data and values for every trial, it also doesn’t help when you’re going to be getting the same data values. [. . .] So, even though we all do it individually [. . .] and the fact that everyone is within a certain range, you don’t really have to rely on your own trust and intellect, because everyone else is doing it already. [. . .] you’re not having to rely on that trust 207 and help explain to yourself why you’re right. (Ash, Problem-solving & Critical Thinking, OL) Alex was another participant who offered differing views on the impact of remote learning. They perceived that absence of opportunities to make mistakes while performing experiments, which was seen by another participant as a barrier to technical skill development, as hindering development of problem-solving & critical thinking skills as well as their understanding. [. . . ] because we’re doing everything online and we’re given the exact data [. . .] there’s no chance of human error. Like usually when we do it in a chemistry lab [. . .] there’s a chance that we may have, do a mistake and to fix it we use our presence of mind. [Later in interview] [. . .] labs are harder when it’s just visual [. . . ] you don’t get to understand the depth [. . .] (Alex, Problem- solving & Critical Thinking, OL) However, by engaging in a new experience and having to focus more, the online learning environment provided an opportunity for Alex to apply their problem-solving & critical thinking skills. I would say it helped me, [. . .] because it's something different, I never did this before. [Later in interview] And [online] allowed me to think, think more critically, 'cause since like you have to focus more because it's not in, in person. I would say these two. (Alex, Problem-solving & Critical Thinking, OL) Although Alex felt an aspect of problem solving & critical thinking was hindered online, having the instructor there, following instructions, and watching the provided experimental videos multiple times helped them with these challenges. Similar to Alex, Jordan saw the lack of hands-on experiences as a barrier to problem- solving & critical thinking skill development due to an inability to engage in data collection and learn through trial and error. 208 [. . .] I would like to [. . .] perform some of these experiments in person, just so I could see [. . .] if things went wrong in real life, how would I be able to handle them. Because everything’s so controlled here [. . .] you click a button and it gives you an answer. I think there is not really an opportunity to develop problem solving as well here. (Jordan, Problem-solving & Critical Thinking, OL) Finally, Reign reflected on the lack of agency in controlling how an investigation was carried out and associated it with impeding development of problem-solving and critical thinking skills. [. . .] cause everything is given to you, you don’t really have much, you don’t have the space to question anything because that’s all you have to work with in general. [. . .] I think in-person, it was like a, I guess I would call in-person freedom of being you get to choose how much of something you would use, like how much of a solution you would use or like what type of beaker or system you would put it on or like the process it through. Whereas [. . .] during the online labs, its kind of like, you have to use this kind of system and like these are these measured out things that are given to you that you have to work with. (Reign, Problem-solving & Critical Thinking, OL) The method of course delivery having an impact on student perceived ability to engage in problem-solving & critical thinking skills has been explored in the literature (Petillion & McNeil, 2021). When comparing the differences in students enrolled in a laboratory section that employed either the use of livestreaming or pre-recorded experiments and their ability to engage in problem-solving & critical thinking, Petillion & McNeil found that students who were given pre-recorded videos were more apt to report negative or neutral experiences over the overwhelmingly positive experiences of their livestreaming counterparts. As noted, the issue with investigating problem-solving & critical thinking skills is how conceptually nebulous it is, lacking a clear definition (Bowen, 2022; Cooper, 2016). The findings of the study herein highlight the nuanced nature of problem-solving & critical thinking and how one participant can view different aspects of these skills in a positive or negative light. 209 Prioritization & Time Management Several students discussed prioritizing tasks and time management in the context of online laboratory learning. For two participants, the online laboratory courses supported development of prioritization & time management skills because for them it required a greater focus on deadlines. [. . .] in GCL2 [. . .] having it be online, I definitely need to manage these due dates, maybe more than I would have, as opposed to like a traditional in-person lab. So, that’s definitely something I’m, you know, learning and working on. (Aspen, Prioritization & Time Management, OL) And meeting deadlines, like if, where in person you do the notebook, you turn the notebook in like at the end of lab. Here, I have to do the notebook and then still like, like, there’s not that physical, like, necessity, where “Oh I have to turn it in.” Its like, oh, since its online, its like not as important to the mind, like you put it, you can, your mind puts it off more easily. Um, so it definitely helps with those skills for sure. (Onyx, Prioritization & Time Management, OL) Onyx viewed being physically present in the lab as an aide to remembering to submit assignments that was lost online. This experience may be supported by the literature that posits that physical spaces can be associated with tasks that need to be completed or memory recall (Lawrence & Peterson, 2016; Pettijohn & Radvansky, 2018; Radvansky & Copeland, 2006). Coordination of tasks online with teammates aided Morgan in developing time management skills. Morgan initially spoke of time management in the context of getting up early for their in-person GCL1 course but then pivoted to speak about how the switch to remote learning supported development of these skills. Operating in a different time zone from their teammates, Morgan spoke of having to navigate working on team assignments outside of scheduled lab meetings and coordinating tasks for team assignments (e.g., poster presentation). And then, you know, totally redoing our poster to now we got to make a PowerPoint and present on a zoom and coordinating with different people, the people in my group, who all live in different places, and “Okay, we're all 210 gonna like get on a call at this time and work on the thing.” You know, keeping track of all that, all the moving parts [. . .]. (Morgan, Prioritization & Time Management, OL) Kai offered a contrasting perspective on the impact of online learning on time management skills, showcasing how online learning placed a heavier reliance on technology. For Kai, issues with their laptop prevented them from working efficiently and interfered with time management. [. . .] for time management, because again, I have a terrible laptop. [. . .] its been hard to work on some things and get it done quickly, because I can’t for the most part, I can only go as fast as my laptop will allow me. (Kai, Prioritization & Time Management, OL) Access to sufficient technology has been reported in the literature as a struggle for some students participating in the online learning environment (Bharadwaj et al., 2023; Means et al., 2020). This highlights a salient equity issue during the pandemic, where not all students were able to access reliable technology to complete tasks, assignments, or participate in synchronous lectures or courses. Additionally, although Kai felt that virtual learning created obstacles for prioritization & time management skill development, they were able to relate coordinating team tasks in the course to supporting growth of this skill. Time management skills are recognized as an important skill for success in the online learning environment (Darby & Lang, 2019), in addition to being a pertinent 21st-century skill (National Academies, 2011). Participant responses presented above capture how students recognized the importance of time management skills when engaging in their online laboratory courses and the subsequent development that came from this. No Change or Impact to EDC Development Compared to In-person Learning A few participants thought that the online GCL1 and GCL2 offered the same opportunities to build SPCs as the corresponding in-person courses. For some participants the remote laboratory courses were believed to provide the same number of opportunities for development online as it would in-person. This was seen for almost all of the SPCs, including communication (n = 3), teamwork (n = 3), problem-solving & critical thinking (n = 1), priority & time management (n = 211 4), and work ethic (n = 3). Interestingly the participants who expressed these thoughts all came from OL learning environments and were primarily from the Fa20 data collection period. The findings may be reflective of students becoming accustomed to the online learning environment, either coming from remote learning in high school or continuing their college courses remote. Grey and Skylar are great representations of how the modality was perceived as having little to no influence on SPC development. [. . .] these like six skills [. . .] don't really change that much, just being like in this lab right now, because like, I think there's ample amount of room for teamwork and communication skills to improve just because you still are working with a group that doesn't change, like, just the fact, [. . .] in my opinion that you're in person and stuff, like wouldn't really change that that much. And then, like managing time and priorities, I mean, all the assignments are still like due, it's not like we're getting like less, or at least to my knowledge, it's not like we're getting like a ton of less assignments or like stuff more spaced out, so yeah. (Grey, OL) Um, I don't really think it's like affecting skill development being online, just because you're still doing teamwork, you still have to communicate, you still have to problem solve, you still have to self-motivate yourself to do the work. You still have to manage your time properly. (Skylar, OL) One participant was in both camps of feeling that teamwork skills were on one hand not affected taking the course online, but on the other also believing virtual learning gave them opportunities for further development. Jordan felt teamwork did not differ greatly in the remote learning environment early in the interview, but as the interview progressed, they also spoke of how beneficial it was to learn how to work remotely in a team. This participant may show that being able to develop the skills of working with others remotely was just an added benefit to something that would otherwise remain unaffected. Implications Framing development within the context of a motivator (e.g., career goal) can support learning (Cavanaugh, 2016; National Academies of Sciences Engineering and Medicine, Division of 212 Behavioral and Social Sciences and Education, et al., 2018), and by placing a lens on the skills students perceive as valuable we can further focus on how these courses can contribute to development of these skills as a guide for future courses. Student experiences provided within this study show the pros and cons of an online learning environment and SPC development. With some students believing that engaging in the course virtually provided support for development of SPCs, it can be observed that online learning can provide beneficial experiences for students. We believe that these experiences can reach larger audiences through instructors framing the learning environment in a beneficial way that can aid students in recognizing development, regardless of the modality. Online learning was a new experience for many students and instructors alike during the pandemic and although campuses have since returned to in-person learning there has been a culture shift both in higher education and the workforce towards incorporating more hybrid experiences. Educators play an integral part in shaping student perceptions of the learning environment and the potential gains that can be had from a course, and through recognizing and leveraging opportunities that can have a positive influences on skill development, students can become aware of and engage in potentially beneficial experiences. Especially seeing as instances such as collaboration on global levels is a necessary component of many modern careers, being able to learn valuable skills such as communication and collaboration virtually can assist students in being successful in today’s workforce. Reflecting upon instances in which students experienced struggles online offers additional opportunities for growth. Pursuing different avenues such as virtual reality may offer beneficial opportunities for students to continue to get acquainted with laboratory skills and techniques (Williams et al., 2022) that could help students maintain a sense of technical skill development when operating remote. Other areas that could improve include providing more resources and guidance that may help students navigate the difficulties of teamwork online (Donelan & Kear, 2023), resulting in greater support for communication and teamwork skill development. Integration of online learning strategies can be seen as favorable due to being able to accommodate more students in response to a shortage of in-person seats (Enneking et al., 2019), provide a cost-effective solution to mounting costs of maintaining traditional labs (Ma & Nickerson, 2006), and answer an increased demand for distance-learning courses (Allen & 213 Seaman, 2013). This push for more remote laboratory learning experiences further emphasizes the need for curricula reform that can deliver student learning goals while also providing opportunities to develop valuable career competencies in preparation for future careers. Conclusions & Future Directions Despite online learning, participants were still able to cite multiple instances of developing valuable 21st-century skills. This mode of learning offered students’ ways to develop employer- desired competencies through learning in unexpected circumstances but provided many challenges and difficulties to learning these important skills as well. Technical skills were the most heavily impacted, with many students reporting a lack of development due to absence of hands-on in-person experiences. For the remaining skills of communication, teamwork, problem- solving & critical thinking, prioritization & time management, and work ethic skills participants were split between three perceptions of development: online supporting, hindering, or not changing opportunities to develop these skills. Considering the variety of perspectives participants shared on remote learning in the laboratory and skill development, it is apparent that online learning affected participants differently with no clear pattern beyond the noticeable deterrent of technical skills. This is possibly because hard skills, such as technical skills, are concrete and tangible (Rockwood, 2021), leading to a shared perception of the impact online learning had on development. Whereas soft skills, such as teamwork and communication, are harder to define and more subjective, resulting in more contrasting and differing opinions. Another aspect captured in student responses was that how a participant viewed adapting to the remote learning environment (e.g., as either a positive learning experience, negative learning experience, or indifferent) seemed to influence how development was perceived (supported, hindered, or no change). The ability to adapt to these changing circumstances has been proposed to be a potentially beneficial attribute for workforce success (Kinsky et al., 2021). In-person and online classrooms would benefit from instructors pulling students attention to the potential gains each course modality can offer and ways in which students can expect to engage in EDC development. While measuring skill development or application was not a goal of this study, developing an instrument or method to more accurately determine whether development is occurring would be advantageous. Much work has been done prior in this area (National Research Council et al., 2011), however, more work in this field is needed. Although education 214 is continually transforming, using streamlined methods to measure skill development would greatly aid educators in determining the best pedagogical practices and course modalities to use to ensure adequate career preparation for the 21st-century. Limitations Technological issues plagued interview transcripts at inconsistent intervals, causing portions of interviews to be cut or incoherent. Because of this there were issues transcribing the interviews and ultimately portions of excerpts were lost during analysis. Due to these problems, some thoughts reflected by participants may not have been accurately captured. With a reliance on technology to participate in this study, access to proper WiFi, internet, or technology could have been a problem that occurred when recruiting students. Limited access may have been a determining factor in whether a participant felt they could participate in a remote interview, causing some students to not be included in the sample. We also acknowledge that the graduate teaching assistants responsible for the online courses could have had an impact on student experiences. Some teaching assistants could have taken a more relaxed approach to teaching online by interacting little with their students and providing baseline information, whereas others may have had the goal to interact with students as much as possible in the zoom rooms and provide rich resources to make sure students excelled in the course. As we did not control for or investigate these variables, we cannot speak on how the teaching assistants affected the learning environment. However, being that the instructor can set the tone of the classroom, this could have been a key component to why students reported some of these experiences, even if they did not explicitly mention it. Even though no statistically significant differences were found in SAT scores between interview participants and those who did not participate, interview participants still may not be representative of the general student population in the course. Additionally, using methods of data collection based on volunteer participants can introduce bias, generating inaccurate or conflated results (Remler & Van Ryzin, 2015). Many of the results above represent the viewpoints of a small number of students in comparison to the course, however, as seen above these participants are representative of diverse viewpoints. All these limitations were taken into consideration when presenting the results of this study. 215 REFERENCES Accettone, S. L. W. (2022). Student Perceptions of Remote Chemistry Laboratory Delivery Models. Journal of Chemical Education, 99(2), 654–668. https://doi.org/10.1021/acs.jchemed.1c00757 Agopian, T. (2022). Online Instruction During the Covid-19 Pandemic: Creating a 21st-Century Community of Learners Through Social Constructivism. The Clearing House: A Journal of Educational Strategies, Issues, and Ideas, 95(2), 85–89. https://doi.org/10.1080/00098655.2021.2014774 Aguilera-Hermida, A. P., Quiroga-Garza, A., Gómez-Mendoza, S., Del Río Villanueva, C. A., Avolio Alecchi, B., & Avci, D. (2021). Comparison of Students’ Use and Acceptance of Emergency Online Learning Due to COVID-19 in the USA, Mexico, Peru, and Turkey. Education and Information Technologies, 26(6), 6823–6845. https://doi.org/10.1007/s10639-021-10473-8 Ali, N., & Ullah, S. (2020). Review to Analyze and Compare Virtual Chemistry Laboratories for Their Use in Education. Journal of Chemical Education, 97(10), 3563–3574. https://doi.org/10.1021/acs.jchemed.0c00185 Allen, I. E., & Seaman, J. (2013). Changing Course: Ten Years of Tracking Online Education in the United States. https://files.eric.ed.gov/fulltext/ED541571.pdf Archambault, L., Leary, H., & Rice, K. (2022). Pillars of Online Pedagogy: A Framework for Teaching in Online Learning Environments. Educational Psychologist, 57(3), 178–191. https://doi.org/10.1080/00461520.2022.2051513 Baldock, B. L., Fernandez, A. L., Franco, J., Provencher, B. A., & McCoy, M. R. (2021). Overcoming the Challenges of Remote Instruction: Using Mobile Technology to Promote Active Learning. Journal of Chemical Education, 98(3), 833–842. https://doi.org/10.1021/acs.jchemed.0c00992 Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society, 57(1), 289–300. Bharadwaj, P., Shaw, C., NeJame, L., Martin, S., Janson, N., & Fox, K. (2023). Time for Class 2023: Bridging Student and Faculty Perspectives on Digital Learning. https://tytonpartners.com/time-for-class-2023-bridging-student-and-faculty-perspectives-on- digital-learning/ Bowen, R. S. (2022). Student Perceptions of “Critical Thinking”: Insights into Clarifying an Amorphous Construct. Chemistry Education Research and Practice, 23(3), 725–741. https://doi.org/10.1039/d2rp00097k Brennan, R. L., & Prediger, D. J. (1981). Coefficient Kappa: Some Uses, Misuses, and 216 Alternatives. Educational and Psychological Measurement, 41, 687–699. Bretz, S. L. (2019). Evidence for the Importance of Laboratory Courses. Journal of Chemical Education, 96(2), 193–195. https://doi.org/10.1021/acs.jchemed.8b00874 Bretz, S. L., Fay, M., Bruck, L. B., & Towns, M. H. (2013). What Faculty Interviews Reveal About Meaningful Learning in the Undergraduate Chemistry Laboratory. Journal of Chemical Education, 90(3), 281–288. https://doi.org/10.1021/ed300384r Brinson, J. R. (2015). Learning Outcome Achievement in Non-traditional (Virtual and Remote) versus Traditional (Hands-on) Laboratories: A Review of the Empirical Research. Computers and Education, 87, 218–237. https://doi.org/10.1016/j.compedu.2015.07.003 Bruck, L. B., Towns, M., & Bretz, S. L. (2010a). Faculty Perspectives of Undergraduate Chemistry Laboratory: Goals and Obstacles to Success. Journal of Chemical Education, 87(12), 1416–1424. https://doi.org/10.1021/ed900002d Carlson, S. (2022). Building Tomorrow’s Work Force. In The Chronicle of Higher Education. The Chronicle of Higher Education. Carmel, J. H., Herrington, D. G., Posey, L. A., Ward, J. S., Pollock, A. M., & Cooper, M. M. (2019). Helping Students to “Do Science”: Characterizing Scientific Practices in General Chemistry Laboratory Curricula. Journal of Chemical Education, 96(3), 423–434. https://doi.org/10.1021/acs.jchemed.8b00912 Carmel, J. H., Ward, J. S., & Cooper, M. M. (2017). A Glowing Recommendation: a Project- based Cooperative Laboratory Activity to Promote Use of the Scientific and Engineering Practices. Journal of Chemical Education, 94(5), 626–631. https://doi.org/10.1021/acs.jchemed.6b00628 Castelli, F. R., & Sarvary, M. A. (2021). Why Students Do Not Turn on Their Video Cameras During Online Classes and an Equitable and Inclusive Plan to Encourage Them to Do So. Ecology and Evolution, 11(8), 3565–3576. https://doi.org/10.1002/ece3.7123 Cavanaugh, S. R. (2016). The spark of learning: energizing the college classroom with the science of emotion (Lang, James M). West Virginia University Press. Centers for Disease Control and Prevention. (2023). CDC Museum COVID-19 Timeline. https://www.cdc.gov/museum/timeline/covid19.html Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences Second Edition. Lawrence Erlbaum Associates. Cooper, M. M. (1994). Cooperative Chemistry Laboratories. Journal of Chemical Education, 71(4), 307. https://pubs.acs.org/sharingguidelines 217 Cooper, M. M. (2016). It Is Time to Say What We Mean. Journal of Chemical Education, 93(5), 799–800. https://doi.org/10.1021/acs.jchemed.6b00227 Creswell, J. W. (2013). Qualitative Inquiry and Research Design: Choosing Among Five Approaches (3rd ed.). Sage Publications, Inc. Darby, F., & Lang, J. M. (2019). Small Teaching Online: Applying Learning Science in Online Classes. Jossey-Bass. DeKorver, B. K., & Towns, M. H. (2015). General Chemistry Students’ Goals for Chemistry Laboratory Coursework. Journal of Chemical Education, 92(12), 2031–2037. https://doi.org/10.1021/acs.jchemed.5b00463 Díez-Pascual, A. M., & Jurado-Sánchez, B. (2022). Remote Teaching of Chemistry Laboratory Courses during COVID-19. Journal of Chemical Education, 99(5), 1913–1922. https://doi.org/10.1021/acs.jchemed.2c00022 Donelan, H., & Kear, K. (2023). Online Group Projects in Higher Education: Persistent Challenges and Implications for Practice. Journal of Computing in Higher Education. https://doi.org/10.1007/s12528-023-09360-7 Enneking, K. M., Breitenstein, G. R., Coleman, A. F., Reeves, J. H., Wang, Y., & Grove, N. P. (2019). The Evaluation of a Hybrid, General Chemistry Laboratory Curriculum: Impact on Students’ Cognitive, Affective, and Psychomotor Learning. Journal of Chemical Education, 96(6), 1058–1067. https://doi.org/10.1021/acs.jchemed.8b00637 Ferguson, R. L. (2007). Constructivism and Social Constructivism. In G. Bodner & M. Orgill (Eds.), Theoretical Frameworks for Research in Chemistry/Science Education (pp. 27–47). Fermin, B., & Pope, M. L. (2003). The Perceptions of College Students Regarding the Factors Most Influential in Their Decision to Attend Postsecondary Education. College and University, 78(4), 19–25. Finley, A. P. (2023). The Career-Ready Graduate: What Employers Say About the Difference College Makes. American Association of Colleges & Universities . https://dgmg81phhvh63.cloudfront.net/content/user-photos/Research/PDFs/AACU-2023- Employer-Report.pdf Galloway, K. W. (2017). Undergraduate Perceptions of Value: Degree Skills and Career Skills. Chemistry Education Research and Practice, 18(3), 435–440. https://doi.org/10.1039/c7rp00011a Gallup Inc, & Strada Education Network. (2018). Why Higher Ed? Top Reasons U.S Consumers Choose Their Educational Pathways. https://news.gallup.com/reports/226457/why-higher- ed.aspx 218 Garrett, R., Simunich, B., Legon, R., & Fredericksen, E. E. (2023). CHLOE 8: Student Demand Moves Higher Ed Toward a Multi-modal Future. https://qualitymatters.org/sites/default/files/research-docs-pdfs/QM-Eduventures-CHLOE- 8-Report-2023.pdf?token=hhzhE8rpfxTazsb6oNnt6aaRf8rZC7cQaLRsXn2blv4 Gwet, K. L. (2016). Testing the Difference of Correlated Agreement Coefficients for Statistical Significance. Educational and Psychological Measurement, 76(4), 609–637. https://doi.org/10.1177/0013164415596420 Hanson, S., & Overton, T. (2011). Skills Required by New Chemistry Graduates and Their Development in Degree Programmes. https://edu.rsc.org/download?ac=509473 Hawkins, I., & Phelps, A. J. (2013). Virtual Laboratory vs. Traditional Laboratory: Which is More Effective for Teaching Electrochemistry? Chemistry Education Research and Practice, 14(4), 516–523. https://doi.org/10.1039/c3rp00070b Herrington, D. G., & Daubenmire, P. L. (2014). Using Interviews in CER Projects: Options, Considerations, and Limitations. In Tools of Chemistry Education Research (Vol. 166, pp. 31–59). Oxford University Press. https://pubs.acs.org/sharingguidelines Hodges, C., Moore, S., Lockee, B., Trust, T., & Bond, A. (2020). The Difference Between Emergency Remote Teaching and Online Learning. Educause. https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remote-teaching- and-online-learning Hofstein, A., & Lunetta, V. N. (1982). The Role of the Laboratory in Science Teaching: Neglected Aspects of Research. Review of Educational Research Summer, 52(2), 201–217. Hofstein, A., & Lunetta, V. N. (2004). The Laboratory in Science Education: Foundations for the Twenty-First Century. Science Education, 88(1), 28–54. https://doi.org/10.1002/sce.10106 IBM Corp. (2020). SPSS (27.0.1.0). Ice, L., Reiley, M. J., & Rinde, S. (2021). Employment Projections in a Pandemic Environment . In U.S. Bureau of Labor Statistics. https://doi.org/https://doi.org/10.21916/mlr.2021.3 Jeffery, K. A., & Bauer, C. F. (2020). Students’ Responses to Emergency Remote Online Teaching Reveal Critical Factors for All Teaching. Journal of Chemical Education, 97(9), 2472–2485. https://doi.org/10.1021/acs.jchemed.0c00736 Kathy Katella. (2021). Our Pandemic Year—A COVID-19 Timeline. https://www.yalemedicine.org/news/covid-timeline Kinsky, E. S., Merle, P. F., & Freberg, K. (2021). Zooming Through a Pandemic: An Examination of Marketable Skills Gained by University Students During the COVID-19 Crisis. Howard Journal of Communications, 32(5), 507–529. 219 https://doi.org/10.1080/10646175.2021.1965927 Kondo, A. E., & Fair, J. D. (2017). Insight into the Chemistry Skills Gap: The Duality Between Expected and Desired Skills. Journal of Chemical Education, 94(3), 304–310. https://doi.org/10.1021/acs.jchemed.6b00566 Krajcik, J. S., & Blumenfeld, P. C. (2006). Project-Based Learning. In The Cambridge Handbook of the Learning Sciences (pp. 317–334). Cambridge University Press. Lamri, J., & Lubart, T. (2023). Reconciling Hard Skills and Soft Skills in a Common Framework: The Generic Skills Component Approach. Journal of Intelligence, 11(6). https://doi.org/10.3390/jintelligence11060107 Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159–174. https://about.jstor.org/terms Lawrence, Z., & Peterson, D. (2016). Mentally Walking Through Doorways Causes Forgetting: The Location Updating Effect and Imagination. Memory, 24(1), 12–20. https://doi.org/10.1080/09658211.2014.980429 Lee, Y. A. (2019). Understanding Parents’ View on US Higher Education Marketing Communication. Educational Research for Policy and Practice, 18(3), 279–299. https://doi.org/10.1007/s10671-019-09249-2 Ma, J., & Nickerson, J. V. (2006). Hands-on, Simulated, and Remote Laboratories: A Comparative Literature Review. ACM Computing Surveys, 38(3), 1. https://doi.org/10.1145/1132960.1132961 Means, B., Neisler, J., & with Langer Research Associates. (2020). Suddenly Online: A National Survey of Undergraduates During the COVID-19 Pandemic. https://files.eric.ed.gov/fulltext/ED610781.pdf Moerer-Urdahl, T., Creswell, J., & Creswell, J. W. (2004). Using Transcendental Phenomenology to Explore the “Ripple Effect” in a Leadership Mentoring Program. International Journal of Qualitative Methods, 3(2). https://doi.org/10.1177/160940690400300202 Morgan, H. (2022). Alleviating the Challenges with Remote Learning during a Pandemic. Education Sciences, 12(2). https://doi.org/10.3390/educsci12020109 Moustakas, C. (1994). Phenomenological Research Methods. Sage Publications. National Academies. (2011). Assessing 21st Century Skills. In Assessing 21st Century Skills. National Academies Press. https://doi.org/10.17226/13215 National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social 220 Sciences and Education, Board on Science Education, Cognitive Board on Behavioral, & Committee on How People Learn II: The Science and Practice of Learning. (2018). How People Learn II: Learners, Contexts, and Cultures. In How People Learn II: Learners, Contexts, and Cultures. National Academies Press. https://doi.org/10.17226/24783 National Association of Colleges and Employers. (2022). Job Outlook 2022. www.naceweb.org|610.868.1421 National Association of Colleges and Employers. (2023). Job Outlook 2023. https://www.naceweb.org/store/2022/job-outlook-2023 National Center for Education Statistics. (2023). Undergraduate enrollment. Condition of Education (should be italicized - overwrite citation). https://nces.ed.gov/programs/coe/indicator/cha National Research Council, & Committee on a Conceptual Framework for New K-12 Science Education Standards. (2012). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. In A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. National Academies Press. https://doi.org/10.17226/13165 National Research Council, Committee on the Assessment of 21st Century Skills, Center for Education, Board on Testing and Assessment, & Koenig, J. A. (2011). Assessing 21st Century Skills: Summary of a Workshop. In Assessing 21st Century Skills. National Academies Press. https://doi.org/10.17226/13215 Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic Analysis: Striving to Meet the Trustworthiness Criteria. International Journal of Qualitative Methods, 16(1). https://doi.org/10.1177/1609406917733847 OECD. (2020). The Potential of Online Learning for Adults: Early Lessons from the COVID-19 Crisis. https://www.oecd.org/coronavirus/policy-responses/the-potential-of-online-learning- for-adults-early-lessons-from-the-covid-19-crisis-ee040002/ Organisation for Economic Co-operation and Development. (2019). Future of Education and Skills 2030: Conceptual Learning Framework. https://www.oecd.org/education/2030- project/teaching-and-learning/learning/skills/Skills_for_2030.pdf Parker, K., Horowitz, J., Minkin, R., & Arditi, T. (2022). COVID-19 Pandemic Continues to Reshape Work in America (Vol. 16). https://www.pewresearch.org/social- trends/2022/02/16/covid-19-pandemic-continues-to-reshape-work-in-america/ Petillion, R. J., & McNeil, W. S. (2021). Student Satisfaction with Synchronous Online Organic Chemistry Laboratories: Prerecorded Video vs Livestream. Journal of Chemical Education, 98(9), 2861–2869. https://doi.org/10.1021/acs.jchemed.1c00549 221 Pettijohn, K. A., & Radvansky, G. A. (2018). Walking Through Doorways Causes Forgetting: Recall. Memory, 26(10), 1430–1435. https://doi.org/10.1080/09658211.2018.1489555 Plano Clark, V. L., & Ivankova, N. V. (2018). Mixed Methods Research: A Guide to the Field. In Mixed Methods Research: A Guide to the Field. Sage Publications, Inc. President’s Council of Advisors on Science and Technology. (2012). Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and Mathematics. https://files.eric.ed.gov/fulltext/ED541511.pdf Radvansky, G. A., & Copeland, D. E. (2006). Walking Through Doorways Causes Forgetting: Situation Models and Experienced Space. Memory & Cognition, 34(5), 1150–1156. Ramachandran, R., & Rodriguez, M. C. (2020). Student Perspectives on Remote Learning in a Large Organic Chemistry Lecture Course. Journal of Chemical Education, 97(9), 2565– 2572. https://doi.org/10.1021/acs.jchemed.0c00572 Reid, H. M. (2022). Finding Differences With Nominal Data—II : The Chi-Square Test of Independence. In Introduction to Statistics: Fundamental Concepts and Procedures of Data Analysis. SAGE Publications, Inc. https://doi.org/10.4135/9781071878927 Reid, N., & Shah, I. (2007). The Role of Laboratory Work in University Chemistry. Chemistry Education Research and Practice, 8(2), 172–185. Remler, D., & Van Ryzin, G. G. (2015). Research Methods in Practice: Strategies for Description and Causation (2nd ed.). Sage Publications, Inc. Rockwood, K. (2021). The Hard Facts About Soft Skills: Why You Should Teach Employees to Be More Resilient, Communicative, and Creative. Society for Human Resource Management. https://www.shrm.org/hr-today/news/hr-magazine/summer2021/pages/why- soft-skills-are-important.aspx Sansom, R. L. (2020). Pressure from the Pandemic: Pedagogical Dissatisfaction Reveals Faculty Beliefs. Journal of Chemical Education, 97(9), 2378–2382. https://doi.org/10.1021/acs.jchemed.0c00657 Santos-DÍaz, S., Hensiek, S., Owings, T., & Towns, M. H. (2019). Survey of Undergraduate Students’ Goals and Achievement Strategies for Laboratory Coursework. Journal of Chemical Education, 96(5), 850–856. https://doi.org/10.1021/acs.jchemed.8b00984 SAS Institute Inc. (2023). SAS Version 9.4. Schultz, M., Callahan, D. L., & Miltiadous, A. (2020). Development and Use of Kitchen Chemistry Home Practical Activities during Unanticipated Campus Closures. Journal of Chemical Education, 97(9), 2678–2684. https://doi.org/10.1021/acs.jchemed.0c00620 222 Seaman, J. E., Elaine Allen, I., & Seaman, J. (2018). Grade Increase: Tracking Distance Education in the United States. https://files.eric.ed.gov/fulltext/ED580852.pdf Seymour, E., & Hunter, A. B. (2019). Talking about Leaving Revisited Persistence, Relocation, and Loss in Undergraduate STEM Education. Springer International Publishing : Imprint: Springer. Telfor, B. (2017). 16 Essential Health and Science Skill Sets. https://natsci.msu.edu/sites/_natsci/assets/File/Current%20Students/College%20Career%20 Services/Option%201%20Skill%20Sets%20with%20Definitions%20for%20Website- CNS%20Articulation%20Exercise%20Website%20Posting%20BT_ST.pdf The Economist Intelligence Unit Limited. (2014). Closing the Skills Gap: Companies and Colleges Collaborating for Change. https://www.luminafoundation.org/files/publications/Closing_the_skills_gap.pdf U.S. Department of Defense. (n.d.). Coronavirus: Timeline. Retrieved October 23, 2023, from https://www.defense.gov/Spotlights/Coronavirus-DOD-Response/Timeline/ Vasileva, O., & Balyasnikova, N. (2019). (Re)Introducing Vygotsky’s Thought: From Historical Overview to Contemporary Psychology. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.01515 Vergara-Castañeda, A., Chávez-Miyauchi, T. E., Benítez-Rico, A., & Ogando-Justo, A. B. (2021). Implementing Project-Based Learning as an Effective Alternative Approach for Chemistry Practical Courses Online. Journal of Chemical Education, 98(11), 3502–3508. https://doi.org/10.1021/acs.jchemed.1c00379 Vygotsky, L. S. (1978). Mind in Society: Development of Higher Psychological Processes. Harvard University Press. Wadsworth, B. J. (1996). Piaget’s Theory of Cognitive and Affective Development (5th ed.). Longman. Wild, D. A., Yeung, A., Loedolff, M., & Spagnoli, D. (2020). Lessons Learned by Converting a First-Year Physical Chemistry Unit into an Online Course in 2 Weeks. Journal of Chemical Education, 97(9), 2389–2392. https://doi.org/10.1021/acs.jchemed.0c00678 Williams, N. D., Gallardo-Williams, M. T., Griffith, E. H., & Bretz, S. L. (2022). Investigating Meaningful Learning in Virtual Reality Organic Chemistry Laboratories. Journal of Chemical Education, 99(2), 1100–1105. https://doi.org/10.1021/acs.jchemed.1c00476 World Economic Forum, & in collaboration with The Boston Consulting Group. (2015). New Vision for Education: Unlocking the Potential of Technology. https://www3.weforum.org/docs/WEFUSA_NewVisionforEducation_Report2015.pdf Yasin, N. Y. B. M., & Yueying, O. (2017). Evaluating the Relevance of the Chemistry 223 Curriculum to the Workplace: Keeping Tertiary Education Relevant. Journal of Chemical Education, 94(10), 1443–1449. https://doi.org/10.1021/acs.jchemed.7b00296 Youssef, M., McKinstry, E. L., Dunne, A., Bitton, A., Brady, A. G., & Jordan, T. (2020). Developing Engaging Remote Laboratory Activities for a Nonmajors Chemistry Course During Covid-19. Journal of Chemical Education, 97(9), 3048–3054. https://doi.org/10.1021/acs.jchemed.0c00792 224 APPENDIX A.V.1. Participant Demographics Table 5.1. summarizes demographic data for participants. Gender reported within this study may not be reflective of a participant’s identity. This is acknowledged and is used with the intent to provide contextual information to build a complete picture of the case. Table 5.1. Demographics of interview participants; number of participants, n (%). Demographic Course Legal Sex GCL 1 GCL 2 Female Male Class Standing Lower Classman Upper Classman Age Ranges 18-20 21-23 Mean Career Goal Health & Medical Professions Engineering & Subspecialties Other Careers Emergency Remote Learning Transition Online Learning Total Learning Environment 9 (39) 14 (61) 13 (56) 10 (44) 19 (83) 4 (17) 22 (96) 1 (4) 18.8 16 (70) 2 (9) 5 (22) 18 (60) 12 (40) 17 (57)* 12 (40)* 20 (67)* 9 (30)* 28 (93)* 1 (3)* 18.9* 27 (51) 26 (49) 30 (57)* 22 (42)* 39 (75)* 13 (25)* 50 (96)* 2 (4)* 18.9* 17 (57) 33 (62) 8 (27) 5 (17) 10 (19) 10 (19) *Denotes one US20 participant who had no registrar data on file. This is reflected in the annotated numbers above. †Class standing is based on credits completed, with lower classmen completing <56 credits (considered Freshman & Sophomore level) and upper classmen ≥56 credits (considered Junior and Senior level). §Percentages may not be equal to 100 due to rounding. 225 A.V.2. Chi-square Test Results Comparing Course Modality to Skills Recognized as Needed for Career Goal The outcomes of chi-square test of association for each individual SPC referenced in the main chapter are presented below. Statistical methods used in this study and the guidelines employed are fully detailed in Chapter VI. Table 5.2. Chi-square analysis of association between whether a student recognized a skill as prevalent to their career goal and course modality. Student-perceived Chi-square Test Competency Value Significance Communication Skills Teamwork Skills Work Ethic Skills Problem-solving & Critical Thinking Prioritization & Time Management — — — — — Technical Skills 1.588 0.640 0.686 0.397 0.686 0.105 0.684 *Test values denoted with an em dash (—) reference comparisons in which cells in the contingency table had values less than 5, facilitating the need to use the Fisher Exact Test resulting in no chi-square test values, only significance being reported. 226 A.V.3. Influence of Online Learning on SPC Development When speaking of how online learning affected SPC development, students shared a variety of experiences that contributed to believing virtual learning supported (Table 5.3) or hindered development (Table 5.4). Themes explored below are not mutually exclusive and can co-occur within a participant’s response. Additionally, all themes observed below have been described and discussed within the main chapter. This section is for those who wish to observe the frequency of each code occurrence. Table 5.3. Perception of how virtual learning in the laboratory supported SPC development (n). Student-perceived Competency (SPC) Reason Online Learning Supported Development Learned to Adapt/Be Flexible Lack of Hands-on/ In-person Learning Harder to Communication/ Collaborate Lack of Communication/ Collaboration Lack of Motivation Distracting Home Environment Lack of Focus More Difficult/Challenging Lack of Interest, Enjoyment, & Engagement Lack of Learning Confusion Having to Focus More Communication Skills (n = 5) 3 Teamwork Skills (n = 3) 3 Work Ethic (n = 5) — Problem- solving & Critical Thinking (n = 4) 3 Prioritization & Time Management (n =3) 2 1 — 1 3 2 1 1 1 — — — 3 — — — — — — — 1 1 1 — — — — — — — — — — — 1 2 — — — — — — — — — — 1 — — — — — — — — — 227 Table 5.3 (cont’d) Importance of Communication Online 1 — — — — 228 Table 5.4. Perception of how virtual learning in the laboratory hindered SPC development (n). Student-perceived Competency (SPC) Reason Online Learning Hindered Development Harder to Communicate/ Collaborate Lack of Hands-on/In- person Learning Lack of Communication/ Collaboration Presentations Were Challenging & Difficult More Individual Learning Experience Lack of Motivation Confusion Lack of Enjoyment, Interest, & Engagement Empathetic Towards Why Hardships Occurred Lack of Learning Lack of Understanding Instructor & Videos Helped Overcome Challenges Technical Issues Communication Skills (n = 4) 1 Teamwork Skills (n = 3) 1 Work Ethic (n = 2) — Problem- solving & Critical Thinking (n = 6) — Prioritization & Time Management (n = 1) — — 1 1 1 — — — 2 — — — — — — — — — — — — — — — 1 1 1 — 5 — — — — — — — 1 1 1 — — 1 — — 1 — — — 4 1 1 — 1 — — — — — — — — 229 A.V.4. Technical Skill Development Comparison Based on Modality Below are the excerpts used to determine the pattern between technical skill development reported by students in emergency remote learning transition (ERL) or online (OL) learning environments. Emergency Remote Learning Quotes Used Laboratory Techniques & Instruments in Class • [. . .] Just like being able to collaborate with others. And then some of the lab skills I probably need again, like just like the actual lab techniques of like, things that we work on in class.” (Indy, worked on physical lab techniques in course) • So far, yes. Um, being just in Chem 161, like I was introduced to a lot of new materials, like a lot of new equipment that I didn't know about. Um, one of them specifically was like the spectrometer, which I was like, "What is this?" and like how to work it, but in my lab group like I would always be working it because I got the hang of it really fast. So I would definitely say it's very helpful. Um, you just have to like, be listening, willing to pick up quickly, and like ask questions when needed. (Sam) • I think getting to use this, this professional equipment like the spectrometers that was that was pretty cool. Just being able to see how that worked and, ah... Just sort of getting a general idea of the kind of equipment that they use in the actual academic scientific world. And sort of being able to make, draw conclusions from the data that we collect. Rather than just seeing it and looking at a bunch of spiky lines, we actually understand intuitively, like, what the meaning of the data is [. . .] (Stevie) Did Not Expand on Type of Technical Skills Developed • I do think so yeah, especially technical skills, cuz there's a lot of new things that now I know how to do that I didn't originally know how to do. (Taylor) • I think that, um, developing professional and technical skills is very important because that's, you know, more of the work you're doing. Um, making sure that's correct. And I think that when we're working in 161, doing everything that you're doing correctly is a big part of it, because those are your results that you're going to be using. And like, obviously, you want to get it right. (Rowan) 230 Online Learning Environment Quotes Building Knowledge of Techniques & Instruments • And that applies to chem lab too. So for example, I learned how to do a spectroscopy. So next time, you know, I'm tasked with doing a spectroscopy, I don't have to start from scratch. And I kind of have a knowledge base of where to go from there. So that's also, um really useful. (Jordan) • Okay, so yeah, I would definitely say that that's going to help with future, for sure. And, um, and now, I mean, if I didn't know how to use, like, if I wasn't educating myself on ah, filtration, how to do filtrations and things like that I, you know, I have to definitely know how to do it before performing it and stuff. So having that skill is super important. (Billie) • No, I feel like these things, it comes like from excessive reading, having a background about things, I feel like these comes like you learn them in the course, you learn how to do these things in the course. You've like, after, like a few labs, you realize that you developed these critical thinking and problem solving. I don't feel like someone comes up, “oh, I have, like, I know how to critically solve this," or, like "I have these techniques," I feel like these are things you learn in the course lab, whether you learn them in the course, or like from previous times you need to, like you develop these things.[Asked by interviewer if they will ever stop learning these skills]. Um, no. I feel like I'm still learning, especially in the CEM 161. Like, there's a lot of technical skills I really don't know. (Blake) • I do feel like technical skills. Mainly because, like, for some of the things like experiments and stuff we've done, like, I had no idea like, what some of the like, areas of science or whatever we, wait, what like I had never learned some of that stuff before, so then going into I was like, "Oh my god, like I don't even know what this is. And I'm about to do an experiment on it" ,kind of thing, but I think like going through the process and working with others, like I learned so much like and I can like yeah, I feel like I was able to learn those technical skills and like equations and how to solve like, some math or just like the thinking skills that you might mean or stuff like that, like basic facts I definitely learned throughout the whole process of like the experiment when at the beginning, I might not have known. (Codi) 231 • Technical skills, I feel like having background about like, the equipment we're going to use or like being able to imagine what equipment we'll use or like how to use, like whether it's Logger Pro, or Excel in Microsoft that really helps in the course, 161, and like how to finish things. (Blake) Used Software • Yeah. So basically, the other day, I had a friend she was she was like, she had a lab. She's basically taking CEM 161. And like, she had a lab where after class, she had, like, the graphs you need to do in Logger Pro. And she had, she was like, really confused in how to do the Logger Pro. So based on the technical skills I learned in CEM 161, I was able to fix the problem she had and show her like, how to do it. So it really, like it really helped helped her like to solve these things. And especially in Excel where, like, I live like with three roommates, my other roommate like she she didn't know like how to use Excel and like, based on how the TA showed us like how to use it. Like, I feel like it really helped, helped, like helped me and help my like, the girls, I helped like how to solve them. So yeah, there's like, and like in, to really, to really like to me, like technical skills have helped me. Like basically, I didn't know how to use Excel, like, on a personal level, I was really confused on how to use it. So I went to one of the office hours, and I learned a lot like on how to use and how like to guide it. And like how to, like, it's like how, like how to use it in general, because I didn't have like a lot of background information about it. So it really helped me in my daily life. (Blake) • Well, I do feel that I am going to need them like if I'm going to work as resear- research assistant, or like a lab assistant. Because the technical skills I learned here, it really helped me in a lot of things in how to use Excel Microsoft, which I believe will help in any future job I"ll have, or in like daily life. So I do believe that technical skills help. (Blake) • And, um, technical skills. Yeah, I mean, like, if I didn't have that, I guess I wouldn't be able to do things like, I don't know, I think of software when I think of technical. I think of all the software that we use for it, if I didn't know how to use it, then I wouldn't be thriving, you know, I guess. (Billie) 232 Recording Observations • I feel like clear communication and recording of stuff, I feel like labs require you extensively to know your observations, which I think it's gonna be really useful, at least in my career choice, like really observe what's going on, and take detailed, yet concise notes on them. (Arbor) Did Not Expand on Type of Technical Skills Developed • Um, and then I would say a little bit of technical skills, this is more about mindset. (Casey) • And I mean, yeah, technical skills (Parker) • Um, and, yeah, all those skills can be found, I think, in some small way, at least for the ones that aren't super applicable. But they all come up, for sure. (Onyx) • Definitely, developing professional and technical skills [. . .](Arbor) 233 CHAPTER VI: SURVEY RESULTS & FINDINGS: COMPARISON TO INTERVIEWS & EXPLORING NEW LINES OF INQUIRY Introduction While interviews allowed for in-depth investigation of the skills students believe are necessary for their career goal and developed within the general chemistry laboratory courses, the mixed- methods analysis of survey data reported in this chapter will provide a more general overview of student perceptions from a broader sample of students. The quantitative analysis of closed questions and qualitative analysis of open-ended questions presented in this chapter will supplement the findings from interviews reported in Chapters IV and V. Survey administration also allowed for exploration of additional lines of questioning outside the scope of the interviews. The results reported in this chapter have the primary goal of contributing to a more complete picture of student ideas about the skills needed for their career goals and their perceptions of opportunities for development within two project-based general chemistry laboratory courses. Research Questions Research questions explored in Chapters IV and V, accompanied by additional areas of inquiry not previously explored, guided the investigation in this chapter: 1. (RQ1) What skills do students believe are needed for their career goal, and how do these skills align with employer-desired competencies (EDCs)? 2. (RQ2) What relationships, if any, existed between student perceptions of valuable career competencies and student characteristics, such as career goal, prior experience and exposure to career goal, first-generation status, and class standing? 3. (RQ3) How do students relate course components and activities in a project-based general chemistry laboratory learning environment to skills perceived as necessary for their planned career? 4. (RQ4) What relationships, if any, existed between student perceptions of development of valuable career competencies within the general chemistry laboratory courses and student characteristics, such as career goal, prior experience & exposure to career goal, first- generation status, and class standing? 5. (RQ5): What impact did the online learning environment have on students’ perception of skills considered valuable for their future career and their perceived ability to develop 234 these student-perceived competencies (SPCs) in project-based general chemistry laboratories? 6. (RQ6): How did introduction of EDCs in course materials contribute to building student awareness of EDCs? Methods The following section will explore the methods used to analyze survey data in two parts. Part 1 will outline the methods used prior to analyses of survey responses. This includes outlining survey administration (e.g., when the surveys were administered and eligibility criteria), demographics (e.g., overall sample distribution, determining if there is a significant difference between demographics of survey samples, and comparison to interview participant demographics), and survey protocol (e.g., lines of inquiry followed in survey questions and connection to interview protocol). Part 2 then explores the outcomes of survey responses. This section will include the statistical methods used in analyses of 1) multi-response testing (e.g., selection of skills compared against student characteristics), 2) multiple choice testing (e.g., choice response compared against student characteristics), and 3) rank testing (e.g., how a skill was ranked compared against student characteristics). Application of qualitative methods to analyze open- ended survey responses will conclude this section. SPSS 27.0 software was used for most quantitative data management and statistics reported in this study (IBM Corp, 2020), while MAXQDA was used for qualitative analysis of all open-answer responses (VERBI Software, 2021). Part 1. Data Collection & Demographics Survey Administration & Sample Selection Survey questionnaires were generated and administered through Qualtrics for Fall 2019 (Fa19), Spring 2020 (Sp20), Summer 2020 (Su20), Fall 2020 (Fa20), and Fall 2021 (Fa21) semesters (Qualtrics, 2023) and Desire2Learn (D2L) for the Sp21 semester (Desire2Learn, 2023). In all semesters except Fa21, students enrolled in General Chemistry Laboratory 1 and 2 courses (GCL1 and GCL2) were invited to participate via email near the end of each semester and were given a week to complete the survey (~7 days). The Fa21 survey administration consisted of two parts, an Initial Reflective Assignment (IRA) near the beginning of the semester and a Final Reflective Assignment (FRA) near the end. Reminders were sent to students approximately 235 halfway through the survey administration window to notify those who had not yet opened or completed a survey that the opportunity to participate was still available. All data collection was conducted under approval by the Institutional Review Board (IRB) and determined to be exempt. Age of consent to participate in surveys was 18 years old and older. Participant ages were verified using registrar information, and participants under the age of 18 or with no registrar information on file during the time surveys were administered were removed from the sample. All students had been sent an email outlining their rights as a participant and the option to opt- out of the study at the beginning of each semester. They were also provided a brief overview of the study at the start of every survey. Students who opted out of the study or were ineligible because of age were still eligible to receive extra credit upon survey completion, but their data were removed from any analyses and subsequent findings associated with this study. The response rate for this study (A.VI.1) was relatively high for most semesters (> 55%), compared to an average online survey response rate of 44.1% (Wu et al., 2022). This was not the case for all semesters, however, as Sp21 returned a low response rate of 22%, possibly due to the use of a different platform and mode of solicitation that semester. During Sp21, the platform used to administer the survey changed from Qualtrics to D2L. Students were also no longer notified or reminded of the opportunity to participate through email solicitation from the research team. These factors could have caused a barrier to survey visibility, resulting in a decreased response rate in comparison to other semesters. Although low response rates may be thought to be associated with inadequate representation of the population, they have not been found to be detrimental to research studies and have been reported to result in similar findings to studies with high response rates (Wu et al., 2022). Prior to conducting statistical analyses to investigate student perceptions of skill development within the general chemistry laboratory courses and the relevance of these skills to their future career goals, the data collected underwent a thorough cleaning process. This was done to ensure that participants who opted out of the study or were under the age of consent as well as duplicate participants and those who returned incomplete surveys were removed from data analyses. Complete information pertaining to the method of participant removal can be found in appendix (A.VI.2). After data were cleaned based on the guidelines outlined in appendix (A.VI.2), pattern analysis was run on each semesters’ dataset to determine the general trends of incomplete or missing data for the 1) overall survey and 2) individual questions. 236 Missing values patterns varied per semester and ranged from expected to random patterns. In all semesters two predicted response extremes were seen: 1) responding to all questions or 2) leaving all questions blank. The primary pattern observed for all survey administrations were those in which all survey response items were complete (excluding optional responses). A synopses of these general trends and associated figures can be found in Appendix A.VI.3. Once missing values were identified, case-wise deletion was performed in which all participants who had one or more missing responses were removed from the sample (Carpenter, 2020). Participants were removed in such a manner due to the intimate relationship between each successive question in these surveys and the necessity of having all data present to formulate a clear and concise picture of participants’ beliefs and experiences. Multiple imputation, in which different methods are used to fill in missing values, was not conducted to retain the individuality of participant responses that underlies the primary goal of this research. After identifying and removing participants that fit within each of the categories outlined above, the final sample was obtained and used for further analyses (A.VI.2). To ensure that those who were represented in survey findings did not differ from the remaining student body who did not participate, SAT composite scores of each group were compared. No significant differences were found (Appendix A.VI.4). Demographics of Survey Participants All demographic information was obtained from registrar records. A total of 5,134 survey participants were included in this study. The Fa21 sample contains both Fa21 IRA and FRA participants combined, with students who took both surveys during this semester only being represented once. In contrast with the interview population, the survey sample across semesters contained a majority of participants from GCL1 (n = 3,848, 75%) and was more representative of the ratio of the GCL1 to GCL2 student population, in which GCL1 is a much larger course. Approximately equal representation from each course was sought in the sampling used for selection of interview participants. In addition, the availability of volunteers to complete interviews influenced the final sample. Class standing, legal sex, and first-generation status of the survey sample were similar to interview participants, being primarily lower classmen or those classified as having freshman or sophomore status (n = 4,036, 79%), a majority female (n = 3,087, 60%), and mostly continuing generation students (n = 4,035, 79%). The mean age of survey participants was 19.2 years, which 237 was marginally higher than interview participants (18.9). The distribution of race and ethnicities across the six semesters was Asian (n = 438, 8%), Black or African American (n = 344, 7%), Hispanic/Latinx (n =283, 6%), International (n = 253, 5 %), White (n = 3,505, 68%), and Other (n = 311, 6%). The small number of participants identifying as American Indian/Alaskan Native, Hawaiian/Pacific Islander, two or more races, or not reported prompted the creation of an “Other” category to encompass these ethnicities. This was done for ease of reporting and statistical testing and does not indicate that students of these race/ethnicities should not be celebrated and recognized as distinct cultures. A complete breakdown of interview and survey participant demographics is located in Appendix A.IV.5. Chi-square test of association (𝜒2) was used to determine if there was a significant difference between the distribution of gender, first-generation status, class standing, and ethnicity/race across semesters. This method of testing detects differences in the distribution of categorical variables by comparing the frequency of observed and expected counts (Frost, 2020; Reid, 2022). To maintain the null hypothesis (H0) that no relationship is present between variables, the frequencies of expected and observed counts would not differ significantly. In contrast, rejection of the null hypothesis would support the alternative hypothesis (H1) to be true, signifying that observed values differ significantly from expected counts and a relationship is present between the two variables. The outcomes of these tests were reported using Yate’s Correction for Continuity for traditional 2 x 2 contingency tables or Pearson Chi-square values for tables greater than 2 x 2. Further assumptions needed to satisfy use of this test are provided in A.VI.5. Prior to interpreting significant values, adjustment of p-values was necessary, as multiple comparisons were tested on these data. Multiple comparisons are known to increase the likelihood of Type I errors, in which the null hypothesis is falsely rejected, or false positives are detected (Liu, 2022). The prevalence of Type I errors is dependent on the number of hypothesis tests being run on a data set, with the chance of false positives increasing with an abundance of testing due to chance variability in the data and not the presence of true significance in relationships between variables. To control for the occurrence of Type I errors, the Benjamini-Hochberg control for false discovery rate (BH-FDR) was used to adjust p-values of all primary hypothesis test values included in this study, and the Bonferroni family-wise error rate (B-FWER) method was used to further adjust for post hoc comparisons. Post hoc comparisons within this study included 238 multiple comparisons within ANOVA hypothesis tests and adjusted residuals that were the outcome of chi-square tests of association. The exception to applying a p-value adjustment was when p-values were obtained to satisfy assumptions for the use of a hypothesis test (e.g., Levene’s Test for Homogeneity). Since testing assumptions is a prerequisite to justify the use of a statistical test or reported test value, rather than a primary result of a hypothesis test, p-value adjustment was not necessary. Each test mentioned above will be explored in greater detail within the section in which it was applied. A brief explanation regarding the methods behind Benjamini-Hochberg’s FDR and Bonferroni’s FWER is found in A.VI.6. Any values reported in this study are the result of these adjustments. Risks of adjusting p-values for multiple comparisons can include over-correction or under-correction (Porter, 2018). The BH-FDR method is known to be more forgiving in allowing the emergence of false positives, while the B-FWER method has a higher likelihood of producing more false negatives, or incorrectly identifying significant findings as insignificant. Additionally, by reducing the chance of Type I errors, we acknowledge that the probability of Type II errors increased. This occurs as a result of controlling for one error increasing the prevalence of another. Type II errors falsely maintain the null hypothesis when true significance is present. After adjustments were made to p-values, results that maintained significance facilitated the use of further testing to determine how small or large the difference was using effect sizes. When conducting quantitative tests, it is not sufficient to rely on significance of p-values alone. P-values only detect if differences are present between variables, but do not tell us the size of the difference (Sullivan & Feinn, 2012). Relying solely on the significance of p-values can pose a threat to research because a result can be significant but if the difference between the two variables is very small due to aspects such as random variances in large sample sizes, then the significance no longer has value. Sample size has a direct effect on significance, in which larger samples are more likely to result in detection of statistical significance (Kline, 2004; Sullivan & Feinn, 2012). Although effect size is independent of sample size, an association has been found by Slavin & Smith, where they reported small samples tended to result in large effects and larger samples are more likely to produce minimal or small effects (Slavin & Smith, 2009). Effect sizes are used to determine the magnitude of difference for significant p-values and can give weight to whether 239 research findings have practical value or are limited in application (Verma & Verma, 2020). Each statistical test relies on different methods and guidelines for computing and interpreting effect size. The guidelines outlined in Appendix (A.VI.7) were applied to analyses and interpretation for all effect sizes reported in this study. This table (Table 6.18 in A.VI.7) contains the tests in which specific effect sizes are applicable, along with ranges that guide interpretation of the strength of the relationship. For all effect sizes the following strength of association is related to the magnitude: small effect = weak association, medium effect = moderate association, and large effect = strong association. As stated by Cohen, the magnitude of the effect size must be interpreted based on the field or context in which the research is applied. Guidelines provided for effect size must be used with caution, “as a general frame of reference for [effect size] and not to take them too literally” (Cohen, 1988, p. 224). Effect size as used within the confines of this study was not aimed at stating the value of an intervention that has practical application, where medium to large effect sizes are of high importance. This exploratory study used large samples, so small effect sizes were investigated. Results are not meant to be used to compare curricula or establish correlation between curricula and measurable development of EDCs. Future studies could incorporate hypothesis testing to aid in decisions on what learning environments would best support EDC development in the classroom. If significance was found, accompanied by a sufficient effect size based on the parameters of this research (small, medium, or large), post hoc tests using adjusted standardized residuals (referred to as adjusted residuals) were used to determine which variables contributed to significance. Adjusted residuals of chi-square tests account for the difference between expected and observed values for each cell of the contingency table, and values above or below the critical value are contributing factors (Agresti, 2013). The widely accepted critical value of adjusted residuals is ±2; however, based on adjustments made to critical values to account for multiple comparisons, these values will vary throughout this study. Positive values exceeding the Bonferroni adjusted critical value indicate the observed cell count is statistically greater than expected, while negative values that fall below the critical value signify that the observed cell count is statistically lower than expected. The Bonferroni method was used to adjust the threshold for considering cell counts to be of significance. This method is further outlined in A.VI.6. The outcomes of these adjustments are reported with the results. Use of adjusted 240 residuals was only applied to tables greater than 2 x 2, where determining the factors contributing to significance was more difficult than traditional 2 x 2 contingency tables. SPSS 27.0 software was used for data management and calculating most quantitative values reported in this study (IBM Corp, 2020). Applying the methods outlined above to participant demographics, there were no significant differences found across semesters for gender (𝜒2 = 9.928, p = 0.213, df = 5) or first- generation student status (𝜒2 = 8.449, p = 0.319, df = 5). However, significant differences were found for class standing (𝜒2 = 90.000, p = 0.006, df = 5) and ethnicity (𝜒2 = 52.276, p = 0.006, df =25). The effect size for ethnicity was determined to be negligible (V = 0.045) and most likely attributed to large sample sizes. Class standing was found to have a small effect size (V = 0.133) indicating a weak association between semesters. Adjusted residuals attributed the difference to samples from Su20 and Fa20 semesters. Su20 had smaller frequencies of lower classmen (or those classified as freshman and sophomore status based on credits) and higher frequencies of upper classmen (or those classified as having junior and senior status based on credits) compared against all other semesters (A.VI.5). This same trend was found to a lower magnitude when comparing Fa20 to Fa19, Sp20, Sp21, and Fa21 samples (A.VI.5). Due to Su20 being a summer course taken out of the normal progression of traditional fall and spring semesters, this finding was expected. Generally, the course is mostly comprised of lower classmen, but students often take summer courses to catch up on previously missed prerequisites or to get a head start on completing credits. This can result in an anticipated change in the usual makeup of the course and subsequent sample demographics. An additional contributing factor may be that there was a consistent shortage of available seats in GCL1 causing a backlog of students who were unable to enroll in this course during their first year. This in combination with the increasing size of incoming classes may have resulted in larger amounts of those classified as upper classmen being represented in the Fa20 semester survey sample. To test for differences in mean age across semesters, one-way analysis of variance (ANOVA) was used. Following the same assumptions as the independent samples t-test, one- way ANOVA detects if there is a statistically significant difference between the means of a dependent continuous variable (e.g., age) against an independent categorical variable with more than three groups (e.g., semester) (Evans, 2022; Frost, 2020; Gravetter & Wallnau, 2017). Use of ANOVA requires the assumption of normality. Traditional tests for normality (e.g., Shapiro- 241 Wilk and observation of Q-Q plots) were not applicable within this study due to our sample sizes being large enough to follow the central limit theorem (n >30) and normality was assumed. Additionally, since GCL1 and GCL2 are intended as first- and second-year introductory courses, the distribution of ages followed a predictable pattern of being heavily concentrated in the late teens and early twenties age range. The second assumption to be met was the Levene’s test for homogeneity of variances. This assumption states that the distributions of ages are equal or similar to one another. Results found this assumption was violated (p = 0.001), prompting the use of Welch’s statistic, accompanied by Games-Howell post-hoc analysis to determine which groups were responsible for the difference (Gray & Kinnear, 2012; Huizingh, 2012; Lee & Lee, 2018). Although there was a statistically significant difference between the mean age across semesters (F = 15.835, p = 0.006, df = 5, 728.49), the difference was small (𝜂2 = 0.027). Post hoc analysis found the differences to be the product of Su20 and Fa20 semesters where the mean age was slightly higher than the Fa19, Sp20, Sp21, and Fa21 semesters (A.VI.5). This could be related to the higher distributions of upper classmen seen in these semesters, resulting in a higher mean age. In summary, the differences found across survey sample semesters were not stark, and the samples were relatively similar. The statistically significant differences that were found were small and expected (e.g., Su20 semester). Additionally, beyond the differences seen in the distribution of interview and survey participants being sampled from GCL1 and GCL2, the overall distribution of demographic variables across the methods used in this study remained relatively consistent. Survey Protocol Survey questioning followed the same general outline as interview questioning (Chapter IV) by probing students’ anticipated career goals, skills needed for their anticipated career, skills developed in the general chemistry laboratory courses, and experiences surrounding how these skills were developed. Survey questions with the corresponding response type can be found in Appendix (A.VI.8 - A.VI.13) for the six semesters over which surveys were administered. In each semester following the initial pilot administration, questions were added or modified iteratively based on preliminary outcomes from interviews, changes to instruction necessitated by the COVID-19 pandemic, and research group discussions (e.g., exploring impact of online learning on SPC development or ranking of skills by importance). For analysis, related questions 242 from different semesters were grouped according to common themes (e.g., investigating SPCs needed for career goal, development of SPCs in course, etc.) as outlined in Appendix A.VI.14. Part 2. Analysis of Survey Responses Quantitative analysis was used for survey analyses to not only discern patterns and trends in the data but also to determine if there were any associations between survey responses and student characteristics. The four main student characteristics used in analyses were a) career goal b) class standing, c) first-generation student status, and d) prior experience. Methods used depended on the type of response (e.g., multiple choice, multi-response, or ranking), categorization of the dependent variable (e.g, nominal or ordinal), and the number of independent variables used in testing (e.g., 2 groups). The statistical methods applied are introduced below, categorized by response type and question asked. Additionally, Figure 6.1 can provide a point of reference for how decisions were made concerning the statistical methods employed in this study. The three primary response types explored are multi-response (nominal), multiple choice (nominal), and ranking (ordinal). Figure 6.1. Statistical methods used to analyze survey responses based on the categorical dependent variable. (*Fisher Exact Test value was used for contingency tables will cell counts below 5.) 243 Career Goal Codes Participants were asked to state their career goals for Fa19, Sp20, Su20, Fa20, and Fa21 (both IRA and FRA) semester surveys. Participant career goals were exported and qualitatively coded in Excel using the same protocol used for interview responses in Chapter IV. Once coded and uploaded into SPSS, career goals were compared across semesters to determine if there were any significant differences in the distribution. This was done in three parts due to recurring participants in Fa21 IRA and FRA surveys. First, Fa19, Sp20, Su20, and Fa20 were compared using chi-square test of association, with significant results prompting the use of Cramer’s V to determine the effect size. If a sufficient effect size was found, adjusted residuals were used to determine the contributing factors. Following this, two subtests comparing Fa19 - Fa20 semesters against Fa21 IRA and FRA were conducted independently using the same methods. Fa21 IRA and FRA were not compared due to a high recurrence of participants (n = 815), which violated the assumption of independent samples. Multi-Response Analysis Example Question: Which of these skills are you developing in your general chemistry lab course? Identify skills that you will need for your future career. Check the boxes that apply. (Fa19) Investigation into student perceptions of EDCs was initially explored using multi-response prompts. This form of questioning used “select all that apply” checkboxes, where participants could select all, some, or no survey response items to answer the question. Within these prompts, participants in Fa19, Sp20, and Su20 semesters were asked to select skills needed for their future career goal, as well as those they are developing within the laboratory courses. Problems could occur with this style of survey questioning; participants may not have read through the response items thoroughly or employed low-effort strategies, such as selecting response items closer to the top of the list (Smyth et al., 2006, 2008). To ensure that participants were reading survey response items carefully, a validation prompt was purposefully integrated into the survey design at the conclusion of each multi-response question. Validation items prompted students to select a specific response for a particular survey item (e.g., for survey validation please select “Gen Chem Lab”). Participants who did not pass validation remained in the overall survey sample for other analyses but were not included in analysis of multi-response questions. The number of participants removed from multi-response analyses can be found in A.VI.15. Most participants 244 passed validation. Although validation was incorporated into the survey design, it remains possible that participants used low-effort answering strategies while also passing the validation prompt. Analysis of data from multi-response prompts had two primary aims. The first aim was to investigate participant selection choices (selected/did not select) of EDCs as valuable competencies for their planned career, referred to as student-perceived competencies or SPCs. From these results, selection choices were tested for association with student characteristics to determine if student characteristics had a relationship to skill selection. The second aim explored if participants who selected a skill as a valuable career competency also perceived this skill to be developed within the course(s) and whether perceived development (skill was selected/not selected as developed) was related to student characteristics. To explore these aims, chi-square test of association (𝜒2) was used with the corresponding Phi (φ) or Cramer’s V (V) effect size reported if significance was found. When an adequate effect size was present, adjusted residuals were used to determine the groups that contributed to significance. To test for differences, each skill (e.g., communication, teamwork, etc.) and the corresponding selection choice (e.g., valuable to career or developed in course) were treated as separate dichotomous variables (e.g., did/did not select communication skills as valuable for career goal). These dichotomous variables were used in testing for association with career goal, class standing, first-generation student status, and prior experience. Testing for relationships between a participant’s prior experience and skill selection was only explored for the Su20 semester, which was the only semester in which that student characteristic was collected. It was not possible to test for differences between skills (e.g., investigating significant differences between communication and teamwork) or selection choices (e.g., exploring significant differences between a skill being chosen as developed in the course and needed for future career goal) because the interdependency of choices violated the chi-square assumption that samples are independent (e.g., a participant can select and be represented more than once in this form of testing, resulting in interdependent samples). It is important to recognize that Fa19 participants had two less skills to choose from in their multi-response prompts than Sp20 and Su20 participants. The presence or absence of choices may have influenced how an overall prompt was perceived and the way in which a participant responded. Additionally, when a participant does not select a survey response item, it 245 cannot be directly interpreted as, “no” to a prompt. Participants may have passed over or felt indifferent about a response item, in addition to those who viewed a choice as not applicable (Smyth et al., 2006). These are acknowledged as limitations of both the multi-response prompt used and the subsequent analyses. Multiple Choice Analysis Example Question: Have you had any opportunities to intern/shadow in the field of your future career goal? (Su20) As the study progressed, a question was added to explore students’ opportunities to engage with their career goal through shadowing or internship experiences. This question was included in Su20 and Fa20 surveys as a multiple-choice (yes/no) question (A.VI.14). Chi-square test of association (𝜒2) was used to determine if there was a relationship between the nominal response about prior experiences and the student characteristics of career goal, semester, class standing, and first-generation student status. If significance was detected, Phi (φ) or Cramer’s V (V) was used to determine effect size. For sufficient effect sizes, adjusted residuals were used to determine the groups responsible for the difference in tables larger than 2 x 2. Example Question: Do you believe these skills are valuable for success in your future career? & Do you believe that these skills are generally valuable in a variety of different careers? The Fa20 survey (A.VI.11) did not include the multi-response prompt that allowed students to select skills needed for their career goal but instead presented them with a list of six EDCs and asked them if these skills would be valuable to their career and careers beyond their own using multiple choice (yes/no) responses. It was determined that testing for differences based on student characteristics was not applicable because responses were so heavily skewed towards selecting one response for both questions. Consequently, only the data frequencies were reported. Example Questions: Have you learned about skills sought by employers at MSU or in high school? & Would you like to have more opportunities in your college courses to learn about the skills needed for your planned career goal and/or additional schooling? Based on interview and survey data showing that students had a wealth of knowledge about EDCs prior to being introduced to a list, two questions were introduced in Fa21 surveys (A.VI.13.2). These questions aimed to determine instances in which students had heard about EDCs outside of the general chemistry laboratory courses and if they would like more 246 opportunities to learn about EDCs during their undergraduate career. Using the chi-square test of association (𝜒2), the nominal responses to these questions were tested for relationships with career goal, class standing, and first-generation student status. Effect size using Phi (φ) or Cramer’s V (V) was reported if significance was found. Based on the results of this testing, use of adjusted residuals was not necessary. Example Question: Do you think that the opportunities to develop transferable skills in your general chemistry laboratory course are valuable in helping to prepare for your future career? Sp21 surveys contained a question asking if the opportunities to develop EDCs in the GCL1 and GCL2 courses were valuable for career preparation (A.VI.12.2). To test if there was an association present between the nominal participant responses and class standing or first- generation student status, chi-square test of association (𝜒2) was used and associated effect size of Phi (φ) or Cramer’s V (V) was reported if significance was found. No significance was found, making use of adjusted residuals unnecessary. Ranking Analysis Example Question: Rank these skills from most important to least important, with 1 being the most important (can have duplicate numbers if skills are equally important). During the Fa20 survey administration, an additional prompt was included that asked students to rank a set of six EDCs based on their perceived value (A.VI.11). This prompt was added after a Fa20 interview participant ranked skills from most to least important, inspiring the question of how both interview and survey participants would perceive the value of these skills. Within this prompt, students were asked to assign importance to EDCs, with 1 representing the highest value and 2 – 6 decreasing in value. Duplicate values were allowed where participants felt skills had the same value or ranking. Some duplicate rankings went beyond what was considered applicable to this prompt (e.g., rankings of 6, 6, 6, 6, 6, 6). However, due to the large number of participants included in analysis (n = 1115) it was not feasible to assess all participant responses to determine if rankings followed a more applicable pattern (e.g., rankings of 1, 1, 2, 3, 4, 5). Therefore, any participants who ranked skills within the range of 1 – 6 in any pattern were maintained in the sample. Participants who ranked any skills greater than 6 (e.g., 10) were excluded from analyses. With only six skills given, values above 6 were considered invalid. To test the relationships between student characteristics and assigned rankings, the Kruskal-Wallis test was used. This test is a non-parametric alternative to independent samples t- 247 test used to compare the medians of two or more independent samples for ordinal-dependent variables (Gravetter & Wallnau, 2017; Huizingh, 2012; Kruskal & Wallis, 1952). If significance was found, the effect size is reported through epsilon-squared (𝜀2). Because SPSS statistics has no formal way of calculating the effect size of 𝜀2 for Kruskal-Wallis tests, these values were calculated using one-way ANOVA tests. Based on available student characteristics, the categorical variables used in this testing included career goal, class standing, first-generation student status, and prior experience. Example Question: Rank the following transferable skills in order from most important (value of 1) to least important for your planned career, values can be repeated if skills are equally important (For example: Teamwork and Communication Skills could both be ranked as 1). Space is provided for you to add any skills that you think are important for success in the workforce and/or graduate and professional schools, but are missing from the list. The Sp21 reflective assignment contained a prompt asking participants to rank four EDCs, accompanied by four blank spaces to rank additional skills students found relevant to their career goals (A.VI.12.2). Most participants (n = 183, 86%) did not add any skills to their ranking. Because of the option to include additional skills in their responses, ranking the four EDCs provided did not always fall within a range of 1 – 4 (e.g., communication skills could have a ranking of 5 because of an additional skill listed). This was taken into consideration when assessing the outcomes of this prompt, and affected how the raw data was processed prior to analysis. Participant responses that fell outside of the range of 1 – 4, in absence of having additional skills listed, were not considered a valid response and were removed from statistical analysis of skill rankings. Although these participants were removed from rankings analyses, they were maintained in the sample to test further survey responses. The Kruskal-Wallis test was used to compare the median rank by the student characteristics of class standing and first- generation student status. Effect size was reported through epsilon-squared (𝜀2) if significance was detected. Application of Qualitative Methods to Analysis of Open-answer Survey Responses The qualitative coding scheme outlined for interviews (Chapter IV and V) was adopted for open-answer responses collected through surveys. In addition to the multi-response prompt, Fa19, Sp20, and Su20 surveys asked students to select a skill from a multiple-choice list and then elaborate on their experiences building the selected skill within the laboratory courses. Fa19 and 248 Sp20 participants were asked to select and expand on two skills, while Su20 participants were only asked to elaborate on one (A.VI.14). A small subset of participants selected duplicate skills in Fa19 (n = 9) and Sp20 (n = 1) semesters (A.VI.16). These students were removed from analyses of these questions since the prompt asking participants to select an additional skill and not duplicate skills for both questions. The approach used in analyzing responses to these questions was similar to used when assessing development of SPCs in interview data (Chapter IV and V). Students who selected a skill as being both valuable for their career and developed within the course in the multi- response prompt were identified in Fa19, Sp20, and Su20 semesters. After filtering for these participants, skills with the highest selection frequencies were targeted for qualitative analysis of the corresponding open answers. This involved random sampling of n = 60 participants from each of the top selected skills in each semester. Small sample sizes in Su20 precluded reaching the target of n = 60 participants. As a result, all responses that met the selection criteria were analyzed. Further questioning in Su20 prompted students to list skills they felt they would like to continue working on within the laboratory courses but have not yet had a chance to. All Su20 participants (n = 101) were included in this qualitative analysis. Selection of Fa20 participants for qualitative analysis of open answers differed from those in Fa19, Sp20, and Su20 semesters. First a randomized group of n = 200 participants was selected from the overall survey sample and the entirety of these survey responses were analyzed. Questioning concerning development within these surveys was aimed at determining if and how perceived development of the six EDCs, provided in the open answer survey prompt, were related to specific experiences participants had within the course. Although participants were asked to list the skills they believe are valuable to their future career goal earlier in the survey, no further filtering was done based on these responses during this semesters analysis. This occurred for one primary reason – difficulty interpreting alignment between the specific and often expansive set of skills students reported as needed for their career goal to the six EDCs provided. To exemplify this a participant may have reported that they needed “creativity” to be successful in their future career, while also recognizing “problem-solving & critical thinking” on the EDC skills list as being developed within the laboratory course. Although the coding scheme used categorizes these two skills under the same broad category, it may be possible that the 249 participant would not make this connection causing analysis of development to be independent of skills recognized as valuable to future career goals. Unique to Sp21, students were introduced to EDCs over the course of the semester, with 4 EDCs (teamwork, problem solving & critical thinking, communication, and time management) defined in the scenario documents that students engaged with at the beginning of each project. At the conclusion of the semester, surveys were administered containing three open-ended questions (A.VI.12.2). Participants were first asked about how the introduction to transferable skills throughout the course helped them develop awareness of the skills needed in today’s workforce. After being asked if they think that opportunities to develop transferable skills in the general chemistry laboratory course are valuable in helping to prepare for their career, they were prompted to elaborate on their response to this question. Finally, a prompt asked for additional comments or suggestions about transferable skill development in the course. This survey administration had relatively low engagement (22%) compared to other semesters, resulting in a total of n = 214 participants. All participant surveys were used in qualitative analysis. Survey administration in Fa21 represented another idiosyncratic method of data collection because data were collected through both an initial and final survey during this semester. The initial reflective assignment was an open-answer survey asking students what skills they believe are needed for their career, what skills they feel they are proficient in, and what skills they would like to continue working on throughout their college career (A.VI.13.1). The final reflective assignment contained a mix of multiple-choice and open-answer questions, with the open-answer prompts asking students again what skills they believe they needed to be successful in their future career, how they were able to develop those skills within the course, and for suggestions regarding ways in which skill development could be better integrated within the course (A.VI.13.2). Selection of participants to include in analysis was determined by first identifying participants who had taken both the Fa21 IRA and FRA (n = 815). Once these participants were isolated, n = 200 surveys participants from Fa21 IRA were selected at random. These same n = 200 participants were then used for Fa21 FRA analyses. Once a participant was selected, their open-ended responses for both IRA and FRA were coded and analyzed. 250 Results & Discussion Undergraduate Career Goals Inductive coding of participant career goals produced an expansive coding scheme, with the guidelines found in Appendix (A.VI.17). The three mutually exclusive categories that emerged from interview results - Health & Medical Professions, Engineering & Subspecialties, and Other Career Goals – were also observed in survey responses (Figure 6.2). ) % ( r e t s e m e S n i h t i w t n e c r e P 100 90 80 70 60 50 40 30 20 10 0 Fa19 Sp20 Su20 Fa20 Fa21 IRA Fa21 FRA Health & Medical Professions Engineering & Subspecialties Other Career Goals Figure 6.2. Survey participant career goal aspiration disaggregated semester represented by percent of participants (%). The career goal distribution remained relatively consistent across the five semesters in which students were asked to report their occupational aspirations. Each semester was predominately made up of participants who planned to pursue a career in Health & Medical Professions (~50%). This was followed by approximately a third of participants with Other Career Goals (e.g., research-centered careers, education, business) and under a fifth of participants desiring a career in Engineering. Further testing using chi-square test of association for comparison of career goal distributions across semesters revealed no differences. Although a significant difference was detected when comparing Fa19, Sp20, Su20, and Fa20 career goals (𝜒2 = 22.480, p = 0.006, df = 6), interpretation of effect size revealed that the size of this difference was negligible (V = 0.057) and most likely due to large sample sizes. Subsample testing of Fa19 - Fa20 semesters against Fa21 IRA and Fa21 FRA using Bonferroni adjusted p- values also found that although significance was detected (𝜒2 = 44.977, p = 0.002, df = 8, V = 251 0.069 and 𝜒2 = 34.870, p = 0.002, df = 8, V = 0.062, respectively) the effect sizes were extremely small indicating no true differences across semesters. Examination of the stated career goals of recurring participants (n = 815) from Fa21 IRA and FRA surveys revealed an experience common to many undergraduates, shifting career aspirations. Within the Fa21 sample of participants who took both the Fa21 IRA and FRA, 15% (n = 124) reported different career goals based on the coding scheme utilized in this study (A.VI.18) at the two time points. Although this coding scheme does not capture changes within career goal categories (e.g., changing from an aspiring doctor to a nurse), these findings highlight the importance for students to develop EDCs that are applicable across many different careers. Additionally, n = 33 interview participants noted uncertainty in their career plans due to a) being unsure about a specialty or area of work (n = 29), b) wanting to keep their options open (n = 3), or c) planning for alternatives for if their career goal falls through (n = 1). As students move into their careers following college career, it is unlikely that they will remain in one job or with one employer (Bureau of Labor Statistics, 2023). Therefore, building skill sets that are broadly applicable regardless of industry or specialty is critical to workplace success. The primary occupation area planned by both interview and survey participants was Health & Medical Professions, followed by Engineering and Other careers. Although the percentage of participants planning careers in Engineering careers was slightly lower for surveys than interviews, this can be attributed to the purposeful sampling method used to select interview participants that targeted engineering majors as a subsample. The three main career goal categories serve as both a variable to provide a contextual lens and to be used in statistical testing when investigating student perceptions of valuable career competencies and development within the general chemistry laboratory courses. Prior Exposure & Experiences Surveys also explored students’ prior experiences related to their planned career choice. Hands-on experiences, such as internship or shadowing opportunities, often provide valuable insights into possible career paths and are reported to be beneficial additions to a student’s career preparation (Gault et al., 2000; Jones, 2006; Kim et al., 2012; Kusnoor & Stelljes, 2016; Mader et al., 2017.; Oswald et al., 2017; Silva et al., 2016; Wart et al., 2020; Wolinsky-Nahmias & Auerbach, 2022). Some of these benefits include becoming familiar with potential career paths, engaging in real-world experiences, application of knowledge and skills learned in coursework in 252 the workplace, networking and building connections, heightened confidence when entering the workforce, and increased potential for future employment. Investigating students’ prior experiences may provide insight into how a participant perceives their field and the skills that were chosen as needed for a future career goal or developed within the course(s). Although the quantitative association between prior experience and skill selection is not explored until later sections, statistical relationships between planned career goal, class standing, first- generation status, and semester are examined within this section to determine if these variables had a relationship to whether or not a student reported hands-on experiences. Yes, has had internship/ shadowing experience 31% No, has not had shadowing/internship experience 69% Figure 6.3. Percent of Su20 and Fa20 participants who did/did not have prior experiences in field of career goal (n = 1,216)(%). In the aggregated data from Su20 and Fa20, the majority of survey participants (n = 842, 69%) had no hands-on experience related to their planned career, while less than a third (n = 374, 31%) had prior internship or shadowing experience in their field (Figure 6.3). No significant differences were found between semesters (𝜒2 = 2.803, p = 0.244, df = 1) or between first- generation and other students (𝜒2 = 0.238, p = 0.767, df = 1). Significant differences were found when exploring the association between prior experience and career goal (𝜒2 =56.658, p = 0.006, df = 2) and prior experience and class standing (𝜒2 = 26.436, p = 0.006, df = 1). Although both 253 associations were considered to be weak due to a small effect size (V = 0.216 and 𝜑 = - 0.147 respectively), the factors that contributed to significance are explored below. ) % ( l a o G r e e r a C n i h t i w t n e c r e P 100 90 80 70 60 50 40 30 20 10 0 Health & Medical Professions Engineering & Subspecialties Other Careers Yes, has had internship/shadowing experiences No, has not had internship/shadowing experiences Figure 6.4. Percent of participants who had prior experience by career goal (%). Comparison of career goals and response choice for prior experience using adjusted residuals showed that Health & Medical Professions had a statistically greater observed cell count of participants who had reported internship or shadowing opportunities than Engineering or Other career goals (A.VI.19), a trend that can also be observed in Figure 6.4. The Health & Medical Professions career goal category is comprised of many participants having the primary goal of becoming a doctor. With extracurricular activities and health-related experiences being beneficial assets to gain entrance into professional school (Association of American Medical Colleges, 2023), it does not come as a surprise that these participants have a slightly higher frequency of engaging in shadowing or internship opportunities in comparison to the other career goals. 254 ) % ( g n i d n a t S s s a l C n i h t i w t n e c r e P 100 90 80 70 60 50 40 30 20 10 0 Lower Classmen Upper Classmen Yes, has had internship/shadowing experiences No, has not had internship/shadowing experiences Figure 6.5. Percent of participants who had prior experience by class standing (n = 1216)(%). As displayed in Figure 6.5 above, upper classmen had a higher frequency of reporting they had internship/shadowing opportunities than their lower classmen counterparts. Frequencies corresponding to these results can be found in A.VI.19. This finding could be anticipated because upper classmen have had more time to engage in such activities and more opportunities available as they progress through their studies. Because upper classmen are also closer to the conclusion of their undergraduate studies, they may be more focused on seeking opportunities that will contribute to career preparation for the workforce or continuing education (e.g., professional/graduate school). Interestingly a comparison of interview and survey data reveals a higher percentage of interview participants (53%) reporting instances of hands-on experiences than survey participants (31%). While demographic variables (e.g., class standing) and participant characteristics (e.g., career goal) are similarly distributed within both survey and interview administration, this difference may be due to interview questioning allowing for exploration that surveys do not. 255 Skills Perceived as Necessary for Future Career Like interview participants, survey participants were also asked to either describe or select the skills that they perceived to be valuable for their planned career. Although surveys were not divided into non-prompted and prompted portions when exploring this line of inquiry, as the interviews were, some surveys contained questions that were open-ended, allowing students to elaborate on their thoughts (Fa20 and Fa21). Others contained prompts asking students to select from a list of EDCs (Fa19, Sp20, and Su20). Non-prompted student responses will be explored first, followed by prompted responses. The list of skills from Chapter IV was expanded to include additional skills mentioned in surveys (A.VI.20). The Fa20 and Fa21 surveys asked participants about the skills needed for their planned career using open-ended questions in the absence of prompting about specific career-related skills to probe their ideas and general awareness. Participants mentioned the same seven skill sets seen in interviews – interpersonal, intrapersonal, occupation-specific, problem-solving & critical thinking, prioritization & time management, education & learning skills, and other. Comparison of these skill sets between survey and interview participants showed similar patterns (A.VI.21). Interpersonal skills were the most frequently mentioned as a necessary skill set, followed by occupation-specific and intrapersonal skills in both surveys. These three skill sets were also the most frequently mentioned skill sets in interviews. Survey participants in Fa19, Sp20, and Su20 were prompted with a list of employer- desired competencies; 86% or greater found each of the skills to be important to their future career goal (A.VI.22). These skills included communication, teamwork, solving problems, working with data (a subset of critical thinking), prioritization & time management, self- motivation (a subset of work ethic skills), and technical skills. These skills were presented to students as desirable workforce skills and could reflect why these skills were perceived at such a high importance. Removal of this information could have altered these results and more accurately reflected how students viewed the importance of these skills. Additionally, no significant differences were found between skills selected as necessary for their future career and prior experiences, class standing, or first-generation status (A.VI.22). However, a student’s planned career goal and the learning environment students were situated in when taking the course did reveal small differences in the selection of skills perceived as valuable career competencies (A.VI.22). Examination of the skills selected by students as necessary for career 256 success for a possible relationship to students’ planned career goals (Health & Medical Professions, Engineering, Other)(A.VI.22) showed statistical significance for communication (𝜒2 = 12.360, p = 0.012, df = 2, V = 0.080) and working with data skills (𝜒2 = 14.840, p = 0.006, df = 2, V = 0.087), although the effect sizes were too small to be further investigated. 100 90 80 70 60 50 40 30 20 10 0 ) % ( t n e m n o r i v n E g n i n r a e L n i h t i w t n e c r e P In-Person Learning Emergency Remote Learning Transition Online Learning Figure 6.6. Employer-desired competencies selected as needed for planned career goals by learning environment (%). Self-motivation skills were not present in the Fa19 (in-person) survey administration as marked by an asterisk. The data presented in Figure 6.6 above show that a higher percentage of participants in in-person learning (IP) and emergency remote learning (ERL) environments selected most skills as needed for the future career than students in the online learning (OL) environment. Statistical significance was found for an association between the learning environment and whether a student selected a skill as being valuable to their future career goal for communication (𝜒2 = 17.018, p = 0.000, df = 2), teamwork (𝜒2 = 9.742, p = 0.041, df = 2), prioritization & time management (𝜒2 = 23.721, p = 0.006, df = 2), working with data (𝜒2 = 15.931, p = 0.006, df = 2), and self-motivation (𝜒2 = 14.120, p = 0.006, df = 1)(A.VI.22). However, the effect sizes for 257 communication (V = 0.093), teamwork (V = 0.071), and working with data (V = 0.090) were determined to be negligible with statistical significance attributed to large sample sizes. Effect sizes for prioritization & time management (V = 0.110) and self-motivation skills (𝜑 = -0.127) were small in magnitude. Possible reasons for the weak association between course modality and participants’ perceptions of the importance of prioritization & time management, and self- motivation are explored below. Self-motivation skills were only included in surveys administered in semesters with ERL and OL. Chi-square analysis showed that ERL students chose self-motivation as a needed skill at a higher frequency than OL students. ERL students may have recognized this skill as important more frequently than OL students due to having primarily on-campus experiences where they had physical in-person courses to attend amidst many other responsibilities causing motivation to complete these tasks a necessity. Transitioning to online learning for ERL students may have also added to this by needing to adapt to the online learning environment and maintain motivation to continue attendance and completion of courses once the transition was made. For students in OL, a lack of motivation, that has also been observed in the literature (Means et al., 2020), could have caused OL students to have overlooked this skill as an important competency. A similar trend was seen for prioritization & time management. Adjusted standardized residuals revealed that students in ERL had a higher than expected count with an adjusted residual of 3.3, which fell above the adjusted critical value of +2.6, while OL students had a less than expected count with an adjusted residual of -4.2, below the adjusted critical value of −2.6. For students in the ERL semester, a significant portion of their learning occurred in-person before the transition to online instruction. Successfully navigating class schedules, extracurriculars, jobs, and social activities of on-campus life requires time management. The rapid, unplanned switch to online learning required students to adapt their time management practices in adjusting schedules to meet new deadlines, planning for asynchronous or virtually synchronous courses, and navigating working from home and being in a learning environment where ample distractions may be present emphasizing the importance of these skills. In contrast, the more relaxed learning environment for semesters that were entirely online may have contributed to the lower counts for time management was no longer as pressing a factor as in person, possibly leading to time management being seen as less important. Fewer activities during COVID may also have made time management less of a presence in their daily lives. 258 Beyond simply asking survey participants to select the skills they perceived to be valuable, Fa20 survey and interviews, along with Sp21 surveys, also asked participants to assign numerical values (or rankings) to indicate how valuable a skill was to their future career. In addition to discussing general trends observed, the median was used as a measure of central tendency (Gravetter & Wallnau, 2017). This value describes the midpoint or middle value in a distribution of ordinal data. Rankings started with 1 being the most important. Therefore, medians with lower numerical value signify the data being distributed towards participants perceiving a skill with higher importance, while medians with a higher numerical value represent distributions corresponding to participants perceiving a skill as having lower importance. It is important to acknowledge that any interview or survey participant could have tied two or more skills in ranking (e.g., teamwork = 1, communication = 1, problem-solving & critical thinking = 2, prioritization & time management = 3), meaning that although a ranking of 3 may not seem to be low, based on a participants ranking scheme this could have been the lowest value given. Common to the Fa20 survey and interview data was the low ranking of the importance of technical skills. The largest portion of Fa20 survey participants ranked technical skills in sixth place (31%, n = 345), resulting in the highest median (median = 4) of the six EDCs provided. Technical skills also had the highest median (median = 5) (A.VI.23) in Fa20 interview participant rankings, showcasing how technical skills were perceived as less valuable in comparison to the other five skills listed. A small subset (n = 4) of Fa20 survey participants, whose open answer responses were qualitatively analyzed, offered insight into why they felt technical skills were deserving of the lowest ranking. Reasons given included opinions that technical skills were not necessary for a student’s planned career and could be easily learned if and when needed. Interview participants offered similar thoughts about why they ranked technical skills lower in importance than other skills. While the number of survey participants who provided an explanation for their low ranking of technical skills (n = 4) may seem small, the majority (n = 143, 72%) of the 200 randomly sampled open-answer responses from Fa20 provided no justification for the rankings. The skills with the lowest medians and highest perceived value, by semester and method of data collection, were teamwork (median = 2) in Fa20 interviews, problem-solving & critical thinking (median = 2) in Fa20 surveys, and communication skills (median = 1) for Sp21 surveys. It is important to note that Sp21 prompts did not contain self-motivation or technical skills, as 259 questioning had begun to focus on specific skills that were both highly regarded as needed and developed within the courses. No significant differences were found in Fa20 survey responses when assessing differences for prior internship/shadowing experience vs. none (A.VI.23). It is important to note that Fa20 survey results were tested against all mentioned student characteristics, while Sp21 was only tested against class standing and first-generation status. There were significant differences detected for Fa20 participant ranking of skills by career goal for communication (H = 23.628, p = 0.006, 𝜀2 = 0.022), teamwork (H = 23.281, p = 0.006, 𝜀2 = 0.019), and self-motivation (H = 14.209, p = 0.006, 𝜀2 = 0.011), however all effect sizes were too small to be further investigated. Similar trends were seen in Fa20 survey data when comparing rankings disaggregated by class standing for teamwork skills (H = 7.151, p = 0.037, 𝜀2 = 0.006) and first-generation status for communication skills (H = 6.725, p = 0.049, 𝜀2 = 0.006) with the effect sizes determined to be negligible (𝜀2 = 0.006), leading to the conclusion that random variances within the sample contributed to significance when comparing these samples. No significant differences were found between samples when comparing first-generation status and class standing for Sp21 survey administration (A.VI.23). Although providing participants with an opportunity to rank skills offers a way to assess relative perceived value, caution must be exercised in interpreting median values based on the assignment of integer values to rankings because the scale will depend on the participant and is nonlinear. In the accompanying open-responses to explain their rankings, some Fa20 survey participants stated that all six EDCs were valuable (n = 18), and others (n = 5) further stated that this made ranking the skills in order of importance difficult. Additionally, one participant in Fa20 and another in Sp21 noted how the difference in ranking between skills was marginal. Comparisons will not be made across semesters or the mode of data collection because the prompts were not consistent from semester to semester. For example, Sp21 only had four of the six skills provided in Fa20 survey and interviews, while also providing space for participants to add skills not listed (results included in A.VI.23). Interview participants were also given more freedom in how they spoke of their rankings, a privilege not given to survey participants. Given the results from surveys, it is evident that the college students in this study display a good understanding of the skills needed for the 21st-century workforce. In the absence of prompting, students can identify important career competencies as desired by employers, but typically their lists are incomplete. These survey results support the argument presented in 260 Chapter IV for getting students acquainted with transferable skills early in their college career to build awareness and allow for adequate opportunities to build skills throughout their college career. Development of Student-perceived Competencies (SPCs) in the General Chemistry Laboratory Courses Findings from the investigation of student-perceived competency (SPC) development in the general chemistry laboratory courses will be presented by first discussing the 1) quantitative analysis of multi-response prompts followed by 2) qualitative analysis of open-answer responses. Multi-response prompts asked students to select the skills that they developed in the general chemistry laboratory courses from a list. Follow-up open-answer questions asked students to elaborate on how these skills were developed. Multi-response selection of skills A majority of participants selected all skills listed in the Fa19, Sp20, and Su20 multi-response prompts as necessary for their career (A.VI.22). In analyzing student perceptions of skill development in the general chemistry laboratory courses, we chose to focus on skills that occurred with high frequency in interviews – communication, teamwork, solving problems, working with data (critical thinking), prioritization & time management, self-motivation (work ethic), and technical skills. Going forward these skills will be referred to as student-perceived competencies (SPCs). Table 6.1 reports the number of participants who selected these seven SPCs as valuable career competencies, and among those who selected a skill as valuable that number that also indicated having developed these skills in the general chemistry laboratory courses. Table 6.1. Participant selection of skills as valuable career competencies and selection of skills as both valuable to planned career and developed in the general chemistry laboratory courses. Percentages are reported based on the total sample size. Skill Communication Skills Teamwork Skills Solving Problems Working with Data Participants, n 1949 1949 1949 1949 Valuable for Career, n (%) 1825 (94) 1780 (91) 1827 (94) 1680 (86) Valued & Developed in General Chemistry Lab, n (%) 1588 (87) 1620 (91) 1611 (88) 1497 (89) 261 Table 6.1 (cont’d) Prioritization & Time Management Self-motivation* Technical Skills 1949 978 1949 1808 (93) 909 (93) 1796 (92) 1566 (87) 701 (77) 1407 (78) *Skill was only present in SS20 and US20 semester multi-response prompts. When exploring participant selection choices with respect to development within the courses for possible relationships with prior experience and first-generation status, no significant differences were found for any of the seven skills (A.VI.24). In some cases, associations were found between skills developed and career goal, class standing, and/or the learning environment, which will be examined below. No statistically significant differences were found between career goals and perceived development for communication, teamwork, solving problems, and prioritization & time management. Minor associations were found between career goal and development of professional/technical skills (𝜒2 = 15.409, p = 0.006, df = 2, V = 0.093); however, effect sizes were negligible indicating the absence of strong association with significance possibly arising from trivial differences within large sample sizes. Significant differences with a small effect size were found for self-motivation (𝜒2 = 20.055, p = 0.006, df = 2, V = 0.149). Adjusted residuals for self-motivation showed that Health & Medical Professions selected this skill at higher than expected counts as being developed in the courses than Engineering and Other Career Goals (A.VI.24). Engineering-based careers selected self- motivation at lower-than-expected counts in comparison to the other career goal categories (A.VI.24). Other Career Goals remained within an acceptable range when comparing expected and observed values and did not contribute to significance. Interview participants’ discussion of self-motivation may offer some insight into observed differences. Motivation is a multi-dimensional concept that contains both intrinsic (e.g., internally motivated by personal interest) and extrinsic (e.g., motivated by “external rewards”) components (National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, et al., 2018). Most participants in these samples coming from GCL1, a requirement for both engineering majors and majors generally associated with pre-medicine/pre- vet (e.g., human biology, animal science), may select self-motivation as developed within the courses based on how the course is perceived in relation to one’s major and related career goal. Participants pursuing majors in the biological sciences in preparation for careers in Health & 262 Medical Professions, may associate the laboratory courses more closely with fulfilling prerequisites required to proceed to advanced courses that will help them progress through their major towards their future career. This in turn may amplify the motivation to do well within the general chemistry laboratory courses, resulting in a higher selection of development of this skill. In contrast, students with engineering-based career goals may not perceive these courses as having value towards advancing through their major beyond fulfilling a requirement, leading to fewer participants selecting this skill as being developed. While further in-depth exploration would be needed to determine why development may not have occurred from the survey participant perspective, it is encouraging that many survey participants recognized these courses as providing opportunities for development of these skills. There were no statistically significant differences between lower and upper class standing and selection of communicating effectively, teamwork skills, solving problems, priority & time management, working with data, and self-motivation as being developed within the courses. For professional/technical skills, there were statistically significant differences for class standing with a small effect size (𝜒2 = 20.361, p = 0.006, df = 1, 𝜑 = -0.108). While this effect size is borderline trivial, a greater number of lower classmen selected professional/technical skills as being developed within the courses than upper classmen. This small difference may have arisen from upper classmen having engaged in more advanced course and laboratory work, leading to a perception of diminished value for the technical skills that may be gained from the introductory general chemistry laboratory. Whereas lower classmen may attached greater value to opportunities for technical skill development because they are taking these courses earlier in their studies as intended. Although lower classmen selected this skill as being developed more than upperclassmen, a majority of both class standings still perceived the skill as being developed within the courses. The learning environment varied during the semesters in which in the multi-response data were collected. The learning environment (ERL, OL, IP) did not appear to have a statistically significant association for perceived development of communication, teamwork, solving problems, prioritization & time management, or working with data skills. However, there were significant differences with small effect sizes found for self-motivation (𝜒2 = 10.790, p = 0.006, df = 1, 𝜑 = -0.114) and professional/technical skills (𝜒2 = 72.686, p = 0.006, df = 2, V = 0.201). 263 Although the magnitude of significance for self-motivation skills was borderline trivial, online learning (OL) participants selected self-motivation less frequently than those who took the general chemistry laboratory courses during the emergency remote learning (ERL) transition where they were half in-person and half online. This difference could be the result of generally lower motivation when taking courses remotely. Interview participants (Chapter V) reported that their self-motivation skills were lower when taking the courses virtually. For some interview participants, the online environment was perceived as a hinderance and for others it provided an opportunity for growth. Technical skills were consistently mentioned in interviews (Chapters IV and V) as the skill set most negatively affected by online learning. This was further supported by statistical testing and post hoc analysis of multi-response prompts, in which the adjusted residual of -8.5 for selecting technical skills as being developed in the OL learning environment was well below the adjusted critical value of ±2.6 (A.VI.24). Students often associated development of technical skills with physical manipulations. While ERL students had hands-on experiences during the first half of the semester, OL participants were completely removed from the laboratory environment and had no direct experience with conducting experiments. The open-ended questions that followed the multi-response prompts asked students to discuss how they developed skills that they selected in the general chemistry laboratory courses, but did not ask students why they did not select particular skills. Consequently, there is no direct evidence for why participants in OL selected technical skills less frequently. Qualitative themes from open-answer survey responses associating course components with skill development Use of multi-response questioning provided a general overview of skills developed within the course(s) that participants believed were valuable to their career goals but did not answer how development occurred. To investigate this, open-answer questions were incorporated into surveys. For open answer responses associated with multi-response items, in Fs19, Sp20, and Su20 semesters, skills that were selected for the open answer prompt by ≥20% of participants, that were also selected as valued towards their career and developed within the course within the multi-response prompt, were further explored using qualitative analysis (Table 6.2). 264 Table 6.2. Skills for which open responses were sampled from by semester. Skill Communication Skills Teamwork Skills Problem-solving & Critical Thinking Prioritization & Time Management Work Ethic Technical Skills Field-specific Knowledge Semester Su20 x x Fa19 x x x x Sp20 x x x x Fa20 x x x x x x Fa21 x x x x x x Fa21 followed similar analysis, but instead relied on selecting skills reported by ≥20% of participants to be developed within the courses that were also reported as needed for their future career goals, with the skills that were further explored for this semester represented in Table 6.2. Fa20 had a slightly different method of analysis for selecting skills to further explore development, being that students were presented with a list of six EDCs to expand upon how course elements supported development. The EDCs recognized by ≥20% of participants as being supported within the courses were further investigated and are represented in Table 6.2. In analysis, the skills of solving problems and working with data were aggregated in the skill category problem-solving & critical thinking to distinguish the skill from solving problems and working with data as course activities contributing to building a career skill. Specific sampling parameters for each of these semesters are provided in A.VI.25 - 27. In the analysis, responses for related skills investigated were combined across semesters. The number of participants included in the analysis for each skill is presented A.VI.28. Themes were generated following the same approach used in analysis of interviews. Major themes corresponded to > 60% of participants who attributed development of a specific skill to a course element. For minor themes, 20 – 60% of participants associated a course element with development of a specific skill (A.28). Table 6.3 summarizes the common themes found in both surveys and interviews. The most prevalent shared major themes included how a) collaboration in the courses contributed to communication and teamwork skill development and b) prioritization & time management were developed through individual management of various project components. Shared minor themes included how a) the collaborative environment also aided in prioritization & time management and work ethic skill development, b) working with data led to development of problem-solving 265 & critical thinking skills, c) managing one’s individual tasks and persevering to meet general course expectations (e.g., showing up to an early course) contributed to work ethic skill development, and d) use of laboratory equipment and software/applications supported growth of technical skills. Each of these themes will be expanded upon, including representative quotes, below. Table 6.3. Major and minor themes present in survey and interviews. Major Themes Present in Both Surveys & Interviews and Associated Minor Themes Major Theme 1: Development of communication & teamwork skills was attributed to the collaborative nature of the courses. Associated Minor Themes: Development of prioritization & time management and work ethic skills were also associated with collaboration. Major Theme 2: Independently managing projects, assignments, and tasks supported building prioritization & time management skills. Associated Minor Theme: An individual’s ability to manage assignments and tasks supported development of work ethic skills. Minor Themes Present in Both Surveys & Interviews and Associated Minor Themes Minor Theme 1: Working with data contributed to problem-solving & critical thinking skill development. Associated Minor Theme (Present as a Major Theme in Interviews): Problem-solving & critical thinking skill development was supported by the open inquiry learning environment. Minor Theme 2: Persevering through a college laboratory course supported work ethic skill development. Minor Theme 3: Use of laboratory equipment and software/applications fostered technical skill development. Minor Themes Only Present within Surveys Minor Theme 4: Field-specific knowledge was developed through a) learning and applying concepts, b) general lab experiences (e.g., conducting experiments), and c) the open inquiry learning environment. Major Theme 1: Development of communication & teamwork skills were attributed to the collaborative nature of the courses. Similar to interview findings, components of collaboration in the courses that led to communication and teamwork skill development centered on experiences involving survey participants’ immediate team members. Delegation and coordination of tasks and team roles was the most frequently mentioned aspect of collaborating with team members that resulted in communication and teamwork skill development. Participants discussed organization of team duties, which included working with teammates to complete tasks in an efficient and timely manner, making sure to check in and ensure everyone was on the same page, communicating what was done and still needed to be completed, sharing information and results, and assuming 266 the leadership position when needed. Scheduling time to collaborate outside of normal class meeting times for things such as preparing and practicing presentations was also provided as another course activity that built these skills. Comments from EDC_3997, EDC_3780, and EDC_3128 are representative of how delegating and coordinating teamwork led to communication and teamwork skill development. In terms of communication skills this course has helped for the reason that in order to complete assignments I need to collaborate with my teammates and ensure that everyone is on the same page. [. . .](EDC_3997, Communication Skills, Fa21 FRA Survey) Teamwork skills are employed in the chem lab because four people have to work together to accomplish tasks and assignments. This goes hand in hand with communication skills. The group members need to communicate effectively in order to understand the roles of everyone in their groups as well as complete tasks. (EDC_3780, Communication & Teamwork Skills, Fa20 Survey) [. . .] Usually I don't enjoy being the team leader because I prefer that everyone in the group contribute and "self police" to stay on task if need be, but sometimes when my group would go off task, I would have to keep my teammates on track. It taught me how to speak to them in a manner where I didn't feel uncomfortable or‚"bossy" but still got my point across and got everyone back on track. (EDC_3128, Teamwork Skills, Sp20 Survey) Two quotes from EDC_4451 and EDC_3129 reflected on how learning to work in a team environment led to development of prioritization & time management skills. In each of the labs, my group mates and I all divide up the work/roles of the lab evenly to ensure enough time is given to the lab and that there is enough time to be able to get everything completed that is needed to be. [. . .] (EDC_4451, Prioritization & Time Management Skills, Fa19 Survey) 267 In the General chemistry lab, I think that managing time and priorities are very important because we only have a limited amount of time when we are in the lab (virtually). Meaning that there is only a limited time when all your teammates are available. So it is important to prioritize what needs to get done as a team and how long is it going to take. (EDC_3129, Prioritization & Time Management Skills, Fa20 Survey) Another component of collaboration that led to teamwork skill development, and to a lesser extent communication skill development, was learning how to be a teammate and work with others. Through navigation of a team-based environment, students gained experience that helped them become comfortable in a team setting through working with people they did not know or were unlike them (e.g., people they may not like, people who take on different roles or have different knowledge and skill sets, people who learn or work differently, and people of different cultures), resolving team conflicts and working through disagreements, being a proactive and contributing member, and becoming a leader when needed. Students became better team members by learning to be patient with their teammates, helping and guiding one another, praising others for doing well, and recognizing each person’s contributions as important to the team dynamic. Additionally, students had to learn to depend on, support, and trust group members. The various components of learning how to work with others is exemplified through EDC_0700, EDC_3345, and EDC_4204’s responses below. My lab partners and I work as a team every time we meet. Without working together and helping each other out we would never be able to complete an experiment. This has helped me grow in this area by understanding that people work at their own pace and understand things differently and by being patient, things work out more smoothly [. . .] (EDC_0700, Teamwork Skills, Fa19 Survey) Learning to work with a team, especially with people from different areas of you that come from different backgrounds is something I learned in my general chemistry lab. (EDC_3345, Teamwork Skills, Fa20 Survey) 268 [. . .] we communicate to work through issues and misunderstandings [. . . ] (EDC_4204, Communication Skills, Fa20 Survey) Learning to work in the team environment was also the component of collaboration most frequently mentioned by survey participants as contributing to the minor theme of work ethic skill development. Work ethic skills were developed by motivating oneself to complete work because their team relied on them, being a motivating force for teammates, and coming to class prepared in order to help others. EDC_1922 provides a nice description of how learning to work with others led to work ethic skill development. [. . .] There are also aspects of self-motivation because there are individual assignments and team assignments with individual aspects for which your team is relying on you. (EDC_1922, Work Ethic, Fa20 Survey) Although some participants spoke of listening and sharing ideas with their teammates as contributing to communication and teamwork skill development in survey responses, such responses were less prevalent in surveys than interviews and were not further explored. While a few students in the sample mentioned that interactions with teaching assistants contributed to communication skill development, the number was not large enough to constitute a finding (A.VI.28). To summarize, students were able to build valuable communication, teamwork, prioritization & time management, and work ethic skills through coordination of their efforts to complete group projects and navigation of the team-centered learning environment. The presence of these themes in surveys provides further evidentiary support for interview findings from a larger and more generalizable sample of students. Major Theme 2: Prioritization & time management development was related to students having to independently manage projects, assignments, and tasks. Another major theme present within both interview and survey findings was that through individual management of various aspects of a project, including assignments and tasks, students were able to build prioritization & time management skills. This theme focuses on aspects of project management that are independent from the team environment. Students spoke of how they had to complete assignments, tasks, or projects in an efficient manner and prioritize what 269 needed to be completed within the time constraints (e.g., having to generate a procedure in which tasks can be completed within the 3-hour time frame). They also associated development of time management skills with multi-tasking to complete the various components of lab (e.g., performing experiments and recording work in their laboratory notebook), being aware of and making sure to meet deadlines, being prepared and working on assignments in advance to allow for sufficient time to receive help, making schedules or plans for work that needed to be completed, and leaving enough time during lab in case errors occurred. Managing large workloads and breaking down larger assignments into more manageable pieces was another aspect of project management identified by participants. Participants discussed the need to be organized during experimentation (e.g., keep lab stations clean and organized, organizing lab procedures and evidence) to complete tasks. Quotes from EDC_3305, EDC_3994, and EDC_1859’s provide participant perspectives on how various components of project management contributed to prioritization & time management skill development. When completing tasks in chemistry there is a limited amount of time. Before beginning it is important to estimate how much time each task will approximately take and ensure that there is time to do all of it. [. . .] (EDC_3305, Prioritization & Time Management Skills, Fa19 Survey) [. . .] there are often deadlines in the class for turning in assignments. These deadline vary but it is important that a person is able to effectively manage their time in such a way where they can get all assignments turned in while meeting each deadline (EDC_3994, Prioritization & Time Management Skills, Fa20 Survey) Chemistry lab has helped me a lot with time management and organization especially because of the limited amount of time versus the amount of work there needs to be done, therefore being organized and planning my schedule for this class has helped a lot. (EDC_1859, Prioritization & Time Management Skills, Fa21 Survey) Participants related managing assigned tasks independently to work ethic skills. They wrote about requiring motivation to complete assignments and tasks on time, making sure work 270 was done well, holding oneself accountable to complete work, staying on task, coming to class prepared (e.g., going over material prior to coming to lab and studying for the course), meeting assignment deadlines, and keeping track of work. EDC_1353 and EDC_0518’s commentaries below showcase how independently managing projects, assignments, and tasks contributed to work ethic skill development. [. . .] self motivation is used to get your work not only done but also effectively and well [. . .] (EDC_1353, Work Ethic, Fa20 Survey) In my chemistry laboratory class, it certainly helped my [sic.] develop and expand the skill of an effective work ethic. In doing such detailed experiments in a limited time frame, making sure to record each and every piece of date [sic.], it taught me to work effectively and smoothly, using the entire time I had, working hard every minute of it. It certainly helped develop that skill, which I will later use as a nurse. (EDC_0518, Work Ethic, Fa21 Survey) Course projects contain various components from assignments to experimental tasks that need to be completed throughout the semester. Although these are designed with the intention to be too large to be completed by one person, encouraging the use of a collaborative effort to complete, there are many components of each project in which an individual is responsible for completing their own independently assigned tasks. This allows students to develop highly desirable prioritization & time management and work ethic skills, as evident in both interview and survey responses. Further, themes found within surveys corroborate interview findings and provide additional evidentiary support to claims. Minor Theme 1: Working with data contributed to problem-solving & critical thinking skill development. A minor theme present in both interview and survey data was how students spoke about working with data as contributing to problem-solving & critical thinking skill development. Like interviews, when survey participants spoke of this course element as contributing to skill development, they primarily mentioned how they had to analyze, interpret, and display data. An additional component that was not as prevalent in interviews, but was more frequently mentioned 271 in surveys, was how problem-solving & critical thinking was developed through data collection and record keeping. After conducting experiments and collecting evidence, students mentioned how they engaged in data analysis by sorting through data, determining how to use the evidence collected and decide what data was useful (or having to find and make sure to have the “correct” results), and using mathematical equations to perform calculations. Students also spoke of having to interpret data by finding trends, drawing conclusions, and making sense of their observations. Participant EDC_2493’s quote below references both data analysis and interpretation when elaborating on problem-solving & critical thinking skill development. This class has allowed me to work with data in a way I never have before. Because of the Laboratory setting, I have been able to take and analyze my own date with experiments such as the mystery liquid lab where we used spectrometers to take readings of wavelength and frequency of a liquid and use that data to determine the composition of a mystery liquid (EDC_2493, Problem-solving & Critical Thinking Skills, Sp20 Survey) Visually representing and displaying data using tables and graphs, captured in EDC_2291’s excerpt below, was an additional component of working with data attributed to problem-solving & critical thinking skill development. [. . .] The oral presentation where we needed to make a poster required me to make a lot of graphs out of the data we had collected from lab. In order to make those graphs I had to put the data into line plot or dot plot makers. Then I had to label axes and also title graphs correctly (EDC_2291, Problem-solving & Critical Thinking Skills, Fa19 Survey) Prior to engaging in data analysis, students were tasked with collecting data (e.g, taking measurements or collecting spectra) and documenting these results by recording their observations in a laboratory notebook. Students mentioned the need to organize their data and make sure to keep accurate records when documenting evidence as contributing to problem- solving & critical thinking skill development. 272 In each lab, data collection and recording of data are critical. Our project grades are dependent on accurate records of data. (EDC_1463, Problem- solving & Critical Thinking Skills, Fa19 Survey) Most labs we would gather data about something. We would need to make sure the data is recorded correctly and neatly. (EDC_1130, Problem-solving & Critical Thinking Skills, Sp20 Survey) It is important to note that most of the participant responses in this minor theme come from the Fa19 and Sp20 semesters where working with data was one of the skills on the list provided in surveys. The working with data skill was combined with solving problems to form the overall category of problem-solving & critical thinking for these semesters. The contribution of the open inquiry learning environment to building to problem-solving & critical thinking skills also emerged as a minor theme. Survey participants reported that having to design and plan experiments contributed to problem-solving & critical thinking skill development, a finding also observed within interviews. Participants commonly wrote about devising methods, procedures, or experiments to address an issue presented in project scenarios and laboratory goals. We are given a prompt each week describing certain goals for that lab, but are given no procedure. Using problem solving skills and our knowledge of chemistry, we were able to take the right steps to perform an experiment that would reach these goals. (EDC_1054, Problem-solving & Critical Thinking, Fa19 Survey) [. . .] We use problem solving and critical thinking to come up with experimental designs and figure out how to go about preforming [sic.] our experiments [. . .] (EDC_3035, Problem-solving & Critical Thinking, Fa20 Survey) [. . .] Additionally, the use of project proposals rather than step-by-step guides for labs has introduced me to the type of critical thinking I will need to utilize as I go forward. In my high school we were not asked to hypothesize on 273 possible methodologies for solving a problem. This class has helped me to become more confident with creating and conducting experiments. (EDC_0774, Problem-solving & Critical Thinking, Fa21 Survey) Many of the examples of activities that students identified as contributing to the development of problem-solving & critical thinking can be tied to the scientific practices, particularly planning investigations and analyzing/interpreting data (National Research Council & Committee on a Conceptual Framework for New K-12 Science Education Standards, 2012). These results suggest that curricula incorporating these practices may result in development of EDCs, such as problem-solving & critical thinking. Minor Theme 2: Persevering through a college laboratory course supported work ethic skill development. When speaking of work ethic skill development, both survey and interview participants referred to persevering through general challenges common to college classes rather than specific to the project-based general chemistry laboratory curriculum. This included staying motivated, focused, and on task during a 3-hour class, attending an early class, and showing up for class on time. [. . .] Motivation to stay on task and complete quality work even though the lab is held early in the morning and lasts a long time [. . .] (EDC_1253, Work Ethic, Fa20 Survey) Other participants discussed needing the desire to complete work and assignments, working harder on assignments when having a GTA who grades hard, and gaining experiences dealing with stressful circumstances (that were not further elaborated upon). Gaining work ethic skills by continuing through the course despite various difficulties is reflected in comments from EDC_2087 and EDC_3452. [. . .] Within the course there was also a lot of individual papers and assignments. My formal report consisted of up to 3,500 words, this took a lot of self-motivation for me to get this done. (EDC_2087, Work Ethic, Fa20 Survey) 274 The chem lab course gave me extensive experiences dealing with stressful circumstances. (EDC_3452, Work Ethic, Fa21 Survey) Minor Theme 3: Using laboratory equipment and software/applications fostered technical skill development. For technical skill development, survey and interview participants mentioned using instrumentation (e.g., infrared spectrometer) and software/applications as contributing. Survey participants from the Fa20 online courses most frequently referenced using software and applications as supporting development. These experiences included software programs and applications such as LoggerPro (an application used to collect and analyze spectrum), Microsoft Office (e.g., Excel, Word), and other online resources (e.g., Zoom, D2L, OneDrive). [. . .] technical skills by being proficient in zoom, excel, word, and one drive. (EDC_2521, Technical Skills, Fa20 Survey) Interestingly, some Fa20 survey participants mentioned laboratory equipment in the context of technical skill growth. In many cases, they made broad statements that did not reference specific laboratory tools or instruments; however those who did provide specifics mentioned the use of infrared and UV-visibility spectrometers. Minor Theme 4: Field-specific knowledge was developed through a) learning and applying concepts, b) general lab experiences (e.g., conducting experiments), and c) the open inquiry learning environment. Field-specific knowledge primarily focused on scientific knowledge, specifically developing a deeper understanding of chemistry and building a general background in science and scientific principles. When students spoke of development, they believed a) learning and application of concepts, b) general lab experiences, and c) open inquiry learning were course components that contributed to skill development. The growth of new knowledge and learning through application can be seen within EDC_1594 and EDC_2954’s comments below, respectively. [. . .] I learned basic scientific principles throughout each lab (EDC_1594, Field-specific Knowledge, Fa21 Survey) 275 The general chemistry lab helped me to develop a deeper, more hands on experience of chemistry and its general topics. I am more of a hands on, do it yourself kind of learner so this class helped me with this. [. . .] (EDC_2954, Field-specific Knowledge, Fa21 Survey) Field-specific knowledge was also developed through gaining experience working in a laboratory setting (e.g., lab etiquette, techniques and safety) and having hands-on experiences. I got a basis of chemistry labs and how they work. I know how to be safe now in the lab. I know how to do dilutions very well at this point. [. . .] (EDC_3355, Field Specific Knowledge, Fa21 Survey) Lastly, the open inquiry learning environment helped field-specific knowledge skill development by having students design their own procedures. [. . .] All of the experiments required understanding of how to construct an experiment when given a scenario like identifying an unknown ionic compound, replicating food dye using spectroscopy, etc. [. . .] (EDC_0003, Field-specific Knowledge, Fa21 Survey) Applying chemistry concepts in laboratory activities, developing laboratory skills, and learning techniques are commonly cited goals of laboratory learning (Bretz et al., 2013; Bruck et al., 2010; DeKorver & Towns, 2015). Click or tap here to enter text. Summary Data from surveys provided additional evidence from a larger sample to support the claims and findings from the interview studies. These findings show that a broader sample of students associated many of the same course elements with development of specific career-relevant skills, or EDCs, as their interview counterparts. Students enrolled in the project-based general chemistry laboratory courses perceived that they developed valuable 21st-century skills such as communication, teamwork, problem-solving & critical thinking, and prioritization & time management. Although participants believed these skills to be developed regardless of the course modality (e.g., in-person, online), remote learning posed a challenge to skill development that will be further investigated in the following section. 276 Online Impact on Skill development Investigation of the impact of online learning was incorporated the study design after emergence of the COVID-19 pandemic in March of 2020 forced laboratory courses to operate remotely. Open-answer questions were added to Sp20 and Su20 surveys to probe how online learning affected skill development. Continuing with the top skills in which development was explored for these semesters in the previous section (Table 6.2), it was found that students generally reported the online learning environment to have a negative influence on their ability to develop communication, teamwork problem solving & critical thinking, and prioritization & time management skills (Table 6.4). Teamwork skills were significantly impacted, with 58% (n = 46) of participants indicating that online learning hindered development of this skill. Table 6.4. Sp20 and Su20 combined perception of the impact online learning had on skill development, n (%). Competency Communication Skills (n = 81) Teamwork Skills (n = 80) Problem-solving & Critical Thinking (n = 60) Prioritization & Time Management (n = 60) Influence Online Learning was Perceived to Have on Development No Influence on Supported Development Development 15 (19) 10 (12) Hindered Development 26 (32) 46 (58) 16 (27) 15 (25) 2 (3) 1 (2) 2 (3) 8 (10) 22 (37) 17 (28) Two primary themes emerged regarding how online learning negatively impacted skill development (A.VI.29): 1) communication and teamwork skills were hindered due barriers to collaboration with teammates and 2) the absence of hands-on experiences hindered problem- solving & critical thinking and prioritization & time management skill. When investigating how online learning supported the development of communication skills, the most prevalent theme that emerged was how students had to adapt their methods of communication to overcome the barriers online learning often created. Interview participants reported the same impacts of online learning on skill development, albeit to a much lesser extent. Each of these themes and supporting evidence provided through student quotes are explored further below. 277 Theme 1: Communication & teamwork skills were hindered online due to barriers to collaboration & communication. Participants felt that a lack of face-to-face interactions, an inability to contact and connect with teammates, less time dedicated to working with others, less cooperation, and a lack of opportunities to work and communicate with teammates halted growth of these skills. EDC_4108 and EDC_2328’s comments below capture how collaboration, which was so integral to the course design, was not present online. It became a lot harder [to develop skill] because we were no longer able to meet face-to-face and most of the time not able to work on the tasks at the same time as each other. (EDC_4108, Communication Skills, Sp20 Survey) Team skills were not present online. As soon as classes went online, my chem lab group has not once talked to each other. It is hard to expect team building when classes aren't face to face though. (EDC_2328, Teamwork Skills, Sp20 Survey) Another theme that often coincided with a lack of communication and collaboration between teammates, for students only within the Sp20 semester, was that the assignments became more individual. Students within this semester experienced the emergency remote transition to online learning and the contrast between their in-person and online experiences may have contributed to this negative perception of development. This aspect is reflected in EDC_3063 and EDC_1955’s comments below. Because everything went online, most projects became individual which really took the communication with my group out of the class. We were able to text but it definitely negatively impacted the way we were able to communicate. (EDC_3063, Communication Skills, Sp20 Survey) Since classes have been online, I think the ability to build this skill has decreased. Since online class, all assignments have been individual (besides the glow stick poster.) Because of this, I haven't been in contact with my group 278 members, and haven't had to contribute anything to a "team" because all assignments have been individual. (EDC_1955, Teamwork Skills, Sp20 Survey) This negative perception of engaging in the collaborative environment online was in contrast to positive reports in the literature in which students were satisfied with their remote team-based experiences (Díez-Pascual & Jurado-Sánchez, 2022). The negative experiences reported herein could be a product of many factors. These factors include 1) the GTA facilitating the remote classroom not encouraging collaboration resulting in a lack thereof, 2) the teams themselves not finding it necessary or deciding not to engage with each other, and 3) adapting to the remote learning environment could have been difficult for some students based on living situations amongst other personal reasons. Another aspect that could have contributed to perceptions of the online environment hindering collaboration and communication in the course could have been due to students being given the option of whether to have their camera and/or mic on, a common occurrence observed by the primary author of this study. Being given this option has been reported to have deleterious impacts in student learning (Castelli & Sarvary, 2021). However, Castelli & Sarvary reported that requiring the use of cameras and mics in an online learning environment may potentially generate other problems for students, due to factors such as being afraid that other students may see the environment one is living in, and instead advised instructors to encourage use of cameras and offer alternative solutions (Castelli & Sarvary, 2021). Conversely, a small sample of students believed that growth of communication skills was supported online through the need to be flexible and adapt their methods of communication. Participants had to learn how to convey thoughts and put in a greater amount of effort when collaborating virtually. It is more difficult to communicate when we are not in person, so learning to communicate clearly over zoom calls and messaging has been a challenging but is also an important skill to be learning. (EDC_1879, Communication Skills, Su20 Survey) 279 Theme 2: Absence of hands-on experiences hindered problem-solving & critical thinking and prioritization & time management skill development Students’ perceived ability to develop problem-solving & critical thinking and prioritization & time management skills, was negatively impacted primarily due to no longer being able perform experiments and engage in data collection but instead being given videos to watch. Going to an online course made this a bit difficult because its easier to work with data when it is in front of you and you are recording and measuring it with your own hands. (EDC_0869, Problem-solving & Critical Thinking Skills, Sp20 Survey) This was tougher to build that skill because we didn't do any more labs, just watched a video of one and wrote a report on that. We watched someone do the lab without a particular time limit, and didn't have to prioritize. (EDC_1576, Prioritization & Time Management, Sp20 Survey) Summary Comparison of interview and survey results showed differing patterns. Interview participants who referenced a particular skill were often split in their perception of how online learning impacted development, while survey participants overwhelmingly believed that online learning was detrimental to the growth of skills. Survey participants were asked to focus on 1 - 2 specific skills for further explanation while interview participants asked to offer broad commentary on how online learning effected skill development, which may at least in part account for this difference. Additionally, depending on how an interview progressed, it may not have been possible to explore a participant’s perception of every relevant career skill and the influence online learning may have had. For the various themes that arose when investigating the specifics of how online learning supported or hindered skill development, themes emergent in survey responses were also observed in commentary provided by interview participants. These findings support a need to develop better methods of encouraging collaboration in online learning environments where teamwork is perceived to be difficult or lacking. By addressing these problems, communication and teamwork skill development may be better nourished. Additionally, use of gamification and 280 virtual laboratories as sub(Hawkins & Phelps, 2013; Williams et al., 2022)ps, 2013; Williams et al., 2022) may be able to offset the negative effects online learning has on problem-solving & critical thinking and prioritization & time management skills. Skill Sets Students Feel Proficient In & Areas They Would Like to Develop Skills needed for a participant’s future career goal and experiences surrounding development within the courses were often the primary focus of questioning throughout this study. However, during the Fa21 IRA survey participants were also asked what skills they already feel proficient in. Proficiency was defined as having already mastered a skill or technique and participants who indicated basic knowledge or uncertainty in their abilities (e.g., being “pretty good”) when answering this question were not considered to be proficient in the related competency. Participants could also mention that they felt proficient but would like to continue developing or building skills. Looking broadly at the seven main skill sets identified throughout this survey, most participants who identified interpersonal skills as necessary for their future careers also felt they were proficient in these skills (n = 82 of n = 124, 66%). To a lesser extent this was also observed for prioritization & time management skills (n = 39 of n = 75, 52%) and intrapersonal skills (n = 42 of n = 82, 51%). Students were less inclined to report problem-solving & critical thinking (n = 17 of n = 43, 40%), occupation-specific (n = 33 of n = 96, 34%), education & learning skills (n = 10 of n = 46, 22%), or report other skills (n = 12 of n = 27, 44%) as areas in which competencies were mastered. A complete breakdown of these skills is in A.VI.30. This was followed by a prompt asking students to elaborate on which skills outlined as needed for their planned career were also areas they would like to work on during their undergraduate degree. Overwhelmingly, those who found occupation-specific skills to be necessary career competencies wanted to focus on developing those skills during college (n = 80 of n = 96, 83%). This included skills such as gaining more experience (e.g., shadowing/lab experience), field-specific knowledge (e.g., knowledge of the sciences, math, engineering), and technical skills (e.g., computer science skills or working with software). All other skill sets were reported by approximately 50% of participants, who also recognized the skill set as valuable, as areas they would like to focus on and develop during their undergraduate degree (A.VI.30). These results indicate that for those who recognize and place importance on occupation-specific 281 skills as a necessity for their future career, there is a desire and expectation that these will be gained throughout their college education. Narrowing the scope to looking at the skills students would have liked to have developed but did not get a chance to in Su20 (n = 101) and Fa20 (n = 200) semesters, yielded the results displayed in Figure 6.7. Technical skills stand out above the rest as being the most desired skills for both Su20 (n = 42, 42%) and Fa20 (n = 68, 34%) semesters, a finding mirror by interview participants. Within this set of skills, students wished to have opportunities to hone their a) laboratory skills & techniques, b) technological skills, and c) other technical skills not related to the general chemistry laboratory courses. Specific skills mentioned within the laboratory skills & techniques included a) wanting to work with laboratory equipment, instruments, tools (e.g., pipets and flasks), and materials (e.g., chemicals), and b) having the opportunity to be hands-on in lab by being given the opportunity to carry out procedures or experiments and engage in data collection and observations. When referring to skills surrounding technology, students wanted to learn different software and programs related to chemistry and beyond. Other skills mentioned by students within this category were skills that would not be gained within the confines of the general chemistry laboratory courses and referred to learning technical skills focused on a) areas not yet explored, and b) motor skills. As observed in previous sections, and related to results presented in Chapter V, the online learning environment was either explicitly reported or implied (e.g., mentioning wanting hands-on experiences) by many participants in Su20 (n = 34, 50%) and Fa20 (n = 35, 81%) as the primary factor contributing to a lack of technical skills being present within the course. Due to the lack of in-depth exploration of student commentary, an inherent limitation of survey administration, it can be surmised that most participants attributed the remote learning environment to inhibiting technical skill development because they were no longer physically present in the laboratory space. Interestingly, many Fa20 students continued speaking of development within the courses (n = 37, 19%) or elaborated on skills they would like to further grow in the chemistry courses (n = 5, 3%) or non-chemistry courses (n = 2, 0.1%). 282 s t n a p i c i t r a P f o t n e c r e P ) % ( r e t s e m e S n i h t i w 100 90 80 70 60 50 40 30 20 10 0 Su20 Fa20 Figure 6.7. Skills Su20 and Fa20 survey participants desired to develop within the general chemistry laboratory courses but were not given an opportunity to (sample sizes of each semester is as follows; n = 101 for Su20 and n = 200 for Fa20)(%). Students Have Been Learning About EDCs within High School & College & Would Like Additional Opportunities In the absence of prompting, the rich information provided by participants in both interview and survey data surrounding skills perceived as valuable to their future career goals showed that students were highly knowledgeable of prevalent EDCs needed in today’s workforce. This prompted the Fa21 FRA question of where students had learned about EDCs beyond the general chemistry laboratory courses. As displayed in Figure 6.8, a little over half (55%) of Fa21 FRA participants responded that they learned about EDCs in both high school and during their time at MSU. Three of the four remaining choices were relatively evenly split with 13% reporting they had learned about EDCs solely in high school, 12% who had learned about EDCs only while at MSU, and 13% who were unsure if they had learned about EDCs while in high school or at MSU. A small percentage of participants (7%) had not yet been introduced to EDCs during high school or their time at MSU. All corresponding frequencies to the data described above can be found in A.VI.31. There were no associations found between response choice and career goal (𝜒2 = 19.271, p = 0.057, df = 8), class standing (𝜒2 = 3.079, p = 0.704, df = 4), or first-generation 283 status (𝜒2 = 5.439, p = 0.437, df = 4). While these findings show that students have had opportunities to learn about skills in settings other than the general chemistry laboratory courses, it is unfortunately unclear whether skills it was within college/high school courses or extracurricular college/high school activities (e.g., career fair, clubs) in which exposure to these skills occurred. Further questioning would be needed to determine what activities students engaged in that facilitated learning of EDCs in these environments. Not sure 13% Yes, at MSU 12% No 7% Yes, in high school 13% Yes, both in high school & at MSU 55% Figure 6.8. Percent of Fa21 participants who had learned about EDC outside of the general chemistry laboratory courses (n = 1,063)(%). After asking participants about their experiences being introduced to EDCs in relatively recent educational settings, it was of importance to know if integrating EDCs within the college curriculum would be valuable to students. When prompted with the question of wanting additional opportunities to learn about EDCs, a majority of Fa21 FRA participants (86%) stated they would like to have more exposure to EDCs within their college courses (A.VI.32). The remaining 14% of participants did not desire to learn more about EDCs in their undergraduate courses. This majority distribution did not differ when considering career goal (𝜒2 = 0.929, p = 0.767, df = 2) or first-generation student status (𝜒2 = 1.768, p = 0.389, df = 1). Although there was a significant difference detected between class standing and response choice (𝜒2 = 8.740, p = 0.017, df = 1), the effect size was very small (𝜑 = -0.094), indicating significance was likely due 284 to detecting minute differences within the large sample sizes. Frequencies for the data presented above can be found in A.VI.32. These findings contribute evidence that students have had exposure to EDCs prior to and throughout college and are still willing and ready to continue learning about the skills needed to be successful in their future career pursuits. EDCs are of High Value to Students This study aimed to not only question if the opportunities to learn EDCs within the courses were valuable experiences, but also to investigate if the skills desired by employers and presented throughout prompts were of personal value to students. This line of questioning influenced the addition of two questions being integrated into Fa20 interview and survey prompts. These questions revolved around asking participants if EDCs provided in a list were 1) of value to them and success in their future career and 2) valuable to a variety of careers. EDCs related to the Fa20 survey prompt included communication, teamwork, problem solving & critical thinking, priority & time management, self-motivation, and technical skills. Nearly 100% of survey participants reported that these six EDCs were not only valuable to success in their future career endeavors (n = 1,111, 99.6%) but were recognized as important for other careers as well (n = 1,112, 99.7%). Frequencies associated with the results presented above can be found in A.VI.33. This trend of survey participants highly recognizing these skills as important to themselves, their career goals, and other careers is also reflected within interview transcripts. Of the n = 16 Fa20 interview participants who were asked if the same six EDCs presented to Fa20 survey participants were of value to them, 100% felt these skills had personal value and importance. Additionally, much like the large portion of Fa20 survey participants, of the n = 14 Fa20 interview participants who were asked if the six EDCs were applicable to a variety of careers, or stated independently that they felt these skills had importance to any job, 79% (n = 11) felt that all skills listed were beneficial to any or most jobs. For the n = 3 interview participants who did not feel that all skills were applicable across any career path, they focused on how for some skills the use and importance would vary depending on the career path. This can be seen in Pax’s following statement: “I think all these are very important when it comes to engineering, but depending on what field you go into [. . .] some of them aren't gonna be as valuable.” It is important to note that not all 18 Fa20 interview participants were asked if the six 285 EDCs were applicable across career paths due to a lack of time (n = 2) or a loss of audio in the transcript (n = 2) resulting in an exclusion of responses in these results This perception of skills, either identified by participants themselves as valuable to their career goal or EDCs provided in lists, of having value that extended beyond the workplace was shared by many survey and interview respondents. Both interview and survey participants believed that the various skills explored throughout the entirety of this study are applicable and beneficial throughout many areas of life (A.VI.33) within this chapter and Chp IV. From college to career and their lives in between, these skills are seen as having high value, further emphasizing the importance of integrating these skills within college courses. By incorporating opportunities for development within a student’s undergraduate career, success may be felt in various avenues of life beyond the workplace. Building Awareness of EDCs Throughout the Courses Throughout this study, continual mention has been made to the importance of introducing students to EDCs to better prepare them for their future careers early on in their undergraduate education. Survey administration in Sp21 was created with this in mind, by integrating EDCs into project scenarios introduced to students at the beginning of each project in GCL1. The four EDCs that were continually recognized by students as important to their careers, as well as those reported as valuable 21st-century skills by employers, included communication, teamwork, problem-solving & critical thinking, and prioritization & time management. This included incorporating an excerpt (A.VI.12.1) outlining the importance of these skills to future career endeavors, stating that these skills can be gained from the course, and providing brief definitions of each skill. Toward the end of the semester, surveys inquired how being introduced to these skills helped students build awareness of 21st-century skills (A.VI.12.2). This line of questioning was followed by asking if the opportunity to develop EDCs was valuable to career preparation, ranking skills based on importance, and additional suggestions regarding EDC development in the course. While the way in which students ranked EDCs was already explored in a section above, the remaining components of Sp21 survey administration will be explored below. When asked how the introduction to EDCs throughout the course helped develop awareness of prevalent 21st century skills, three primary themes emerged. These included 1) application, development, and practice of skills, 2) learning about skills and their importance, and 3) learning how to work with others. Approximately half of the participants (n = 118, 55%) 286 spoke of applying, developing, or practicing the skills that were introduced in the course, in response to this question (A.VI.34). While many participants did not reference a particular skill (n = 47, 40%), as reflected within R5’s quote below, there were notable mentions surrounding communication (n = 35, 30%) teamwork (n = 48, 41%), problem-solving & critical thinking (n = 21, 18%), and prioritization & time management (n = 15, 13%). Students mentioned other skills they took time to reflect on developing as well, that were either categorized under the skills mentioned previously (e.g., creativity skills that is categorized under problem-solving & critical thinking) or additional skills (e.g., leadership skills or technology skills). The introduction of transferable skills through the course helped me develop awareness of the skills that are needed in today's workforce by providing me with actual opportunities to use the skills during class. (EDC_1812) Investigating development within this semester’s survey administration did not follow the same coding scheme as other semesters (e.g., referencing the open inquiry learning environment or working with data as leading to development of problem-solving & critical thinking skills). This was because statements were often vague in relation to how skills were developed or applied. Students often provided descriptions of how they became aware of skill application and development over specific course elements that contributed to development. This included recognizing areas of weakness and focusing on areas that needed improvement, along with recognizing what skills they already had. Students also spoke of how a) working with others in the course aided development or application of skills, b) application of skills aided success in course, and c) how the course simulated real world scenarios where skills would be used. Introducing EDCs throughout the course not only gave students the opportunity to develop skills, but learn how to apply these skills as well. Evidence provided using quotes from student responses and the corresponding number of participants who contributed to generation of these themes is found in Table 6.5 below. 287 Table 6.5. Specific examples of how applying or developing skills built awareness of EDCs. How Skills Were Applied/Practiced in Relation to Building Awareness of Skills Became aware of and focused on development (n = 40) Teamwork in course aided skill development (n = 14) Application of skills helped student be successful in course (n = 8) Learned how to apply skills (n = 5) Lab simulated work environment/real scenarios (n = 1) Example It was nice to have a reminder of the skills we were working on in each project because normally I do not connect the skills I am learning with the work I am doing so it was interesting and nice to have those laid out for each assignment and project. (EDC_4986) By knowing what skills I would be developing in this course, I was able to identify and improve them (EDC_1779) When i knew some of the skills that I would need in the workforce in the future, it allowed me to focus on improving in those areas in each lab. By knowing what I wanted to achieve it helped me focus more on improving in certain areas. (EDC_5047) It was nice to have a list of things to look at while I did the project. After each project I was able to go back and see which skills I was able to put to use. (EDC_2438) I was able to focus on improving those skills while working with my team members (EDC_3170) Through working with my group I was able to improve on my communication and critical thinking as a team. (EDC_4960) [. . .] If we did not display any of these skills, or at least improve them, we would not have done well in this course. (EDC_2177) This helped me to practice the same skills I will use in the workforce and allow me to start to understand how to use these skills in many ways. (EDC_2775) This labs introduction of transferable skills were very beneficial and helpful to the many students that took the course. We were introduced to many scenarios and situations that can easily be tasked for when we go out in the real world and get real jobs. (EDC_4963) 288 Table 6.6. How students learned about EDCs integrated within the courses and built awareness of these skills. Way in Which Participants Learned About Skills & Found Importance in Them Became aware of and learned what was needed /expected in the workforce and for career success (n = 55) Recognized importance of skills (n = 15) Became aware of skills that were needed for success in course (n = 12) Learned how skills present in course applied to the workforce (n = 6) Became familiar will the skills (n = 3) Example The transferable skills being listed throughout the course helped me see the importance in everything I was doing, and help me relate it all to what I will be doing in the future. (EDC_0790) The transferable skills introduced throughout this class have shown me the skills that are needed to be successful in my future. (EDC_0590) I did not realize how prominent they actually are, these are skills as a student I use almost every day. It has just become second nature to me but this class helped me realize how important it really was (EDC_5024) it highlighted the fact that some skills that I see as not heavily important are. It showed me that some things can fall through the cracks and it is important to stay on top of everything. (EDC_2050) The introduction of these skills that were listed on every project scenario showed me what exactly was expected of me in each lab. They helped me realize what it was I needed to do and how I needed to apply them to my group. (EDC_1839) It helped me understand how what we are doing in our lab applies to the "real world" and why it matters. (EDC_0268) This course helps you see and develop these transferable skills all of the time. (EDC_2177) Another theme that emerged from this area of inquiry was how students were able to learn about soft skills and become aware of their importance (n = 86, 40%). This included a) learning what was needed or expected in the workforce, b) being able to recognize the importance of EDCs, c) learning skills necessary for success in the course, d) learning how skills in the course apply to the workforce, and e) generally becoming familiar with these skills. Quotes representative of each of these categories and the related number of participants included within each theme can be found above in Table 6.6. In addition to the themes explored above, many students spoke of learning how to work with others through the introduction to EDCs (n = 82, 38%) as presented in EDC_4231 and EDC_4534’s comments below. Students reflected on how they learned to a) apply the skills they were introduced to when working with their team, b) became aware of how to become a better 289 teammate and cognizant of their actions within the group, and c) how important skills were to successful teamwork. The introduction of transferable skill set a guideline for all students to follow, and from the very beginning reminded students the effort they need to put in when working with a team. Breaking down each individual skill enforced the idea that I had of what I needed to do in order to be the best team partner I could be. (EDC_4231) The introduction to transferable skills helped me be more aware of what makes a good teammate and how to effectively work in a group. (EDC_4534) Following this, students were explicitly asked if they perceived development of EDCs as valuable to career preparation, a key questions that went beyond prior surveys focusing on what and how skills were developed. In this multiple-choice question overwhelmingly 96% (n = 205) of participants found EDC development in the course(s) to be a valuable experience (Figure 6.9). In contrast, a very small sample of participants reported that they did not develop any EDCs (n = 4, 2%) or that the EDCs defined in the study were not perceived as valuable to their future career goal (n = 5, 2%). No associations were found between response choice and class standing (𝜒2 = 0.576, p = 1.000, df = 2) or first-generation student status (𝜒2 = 3.812, p = 0.296, df = 2). Tables corresponding to frequencies of the data reported above is in A.VI.35. This finding captures the value of providing students opportunities to build and apply EDCs in an introductory laboratory course that may be perceived as far removed from a student’s potential career path. Additionally, this finding supports interview and survey findings that although participants were attending classes remotely in the Sp21 semester, they were still able to engage in opportunities to develop valuable EDCs. 290 No, did not develop any EDCs 2% No, EDCs are not valuable to career 2% Yes, opportunity to develop EDCs was valuable to career preparation 96% Figure 6.9. Percent of Sp21 participants who perceived value of opportunities to develop EDCs as valuable to career preparation in course (n = 214). When asked to elaborate, many of the n = 205 participants who felt the course provided valuable opportunities for skill development and career preparation spoke of how they applied or developed skills within the course (n = 185, 90%)(breakdown of results in A.VI.36). Most of these participants did not provide commentary or specific examples beyond vague statements or simply stating that they applied, developed, or practiced skills as seen in EDC_2032’s quote below: Learning these skills in this lab was helpful for preparing for my future career because it taught me the skills I mentioned above. (EDC_2032) Skills that were mentioned while speaking of development within the course can be referenced in A.VI.36. For the small sample of participants who provided further commentary on how skills were applied or developed, the corresponding categories in Table 6.7 emerged. The new code of practiced & developed skills in a new environment was relayed by one participant who although they felt they learned nothing new from the course, that they were able to apply valuable EDCs to a new environment when taking GCL1. 291 Table 6.7. Specific examples of how applying or developing skills contributed to development of EDCs and career preparation. How Skills Were Applied/Practiced in Relation to Building Awareness of Skills Became aware of and focused on development (n = 8) Teamwork in course aided skill development (n = 8) Lab simulated work environment/real scenarios (n = 3) Application of skills helped student be successful in course (n = 2) Learned how to apply skills (n = 2) Practiced & developed Skills in a new environment (n = 1) Example The four main transferable skills that were listed helped me recognize and build on them [. . .] (EDC_4287) I think working in a team helps with soft skills when you're not realizing it. Working in a team improves your self confidence and communication skills all are which are important for any career. (EDC_4221) The experiments that we do at the General Chemistry Laboratory are very similar to the situations that we will face in our future careers [. . .] (EDC_1534) My teammates were really helpful with almost everything [. . .] Furthermore, I also helped them in the same way and we all got through the course in the best possible way. (EDC_3732) When I was having a hard time with my group I was always able to look back at the transferable skills and figure out what i was lacking and how I can apply it. (EDC_1812) I continued to develop the skills that I mentioned in question 1 with a new environment. (EDC_4979) Another familiar theme touched upon prior, when investigating awareness, was how participants learned how to work with others (n = 64, 31%). EDC_4231’s response below provides an example of how this student learned to navigate the team environment leading to development of EDCs. I was very nervous at the beginning of this semester because I get nervous meeting new people for the first time. But as time went on with the course I thought my over all leadership and communication skill definitely improved and I will be much more confident the next time I need to work with a group. And for my future career I will without a doubt be working with many different people on projects, and goals, and a class like this has helped me not only build my transferable skills, but also just understanding how a group should work and how to handle sharing the workload for the same project. (EDC_4231) 292 The remaining participants shared how they were able to learn these skills and recognize their importance (n = 5, 2%) or provided statements too vague or contradictory to warrant a code (n = 4, 2%). Students who did not find the course to be helpful in developing EDCs (n = 4) provided the following reasons: a) the online format hindered development of skills, b) the course did not teach new skills but reinforced ones they already had, and c) the course was believed to be poorly organized and the professor was unhelpful and hard to work with leading to a lack of skill development. For the other participants who did not find the EDCs included in this prompt to be valuable to their future career goal (n = 5), most felt that a chemistry course was not related to their future career goal (e.g., vet, medicine, or data science) and one participant did not view EDCs to be something that is learned in a course but to be learned while on the job. Although there was a small percent of participants who did not feel the course was adequate in preparing them for their future career, most participants did find the course and their experience learning of, developing, and applying EDCs to be a beneficial experience. Furthermore, students were given the opportunity to provide additional comments or suggestions regarding EDCs in the GCL1 laboratory courses during Sp21 survey administration. When exploring this comments section, while many students offered no additional comments (n = 104, 49%), there was a large portion of students who spoke of how positive and beneficial their experience being introduced to EDCs and being given the opportunity to develop these skills in the course was (n = 66). This is reflected in EDC_1719, EDC_2113, and EDC_0590’s comments below. Thanks for including these so that way we could see what we needed to go by and so people could see what skills you need to work in this class and even in everyday life (EDC_1719) I feel that this course did a great job of putting these skills at the forefront of our education and showcasing the value and importance of developing these skills. (EDC_2113) Working in groups definitely helped me to develop my transferable skills and being introduced to the ones that would be used in the scenario of each project 293 was helpful in reminding me how and when they are used and for what parts of the process. (EDC_0590) In addition to expressing an appreciation for integration of EDCs within the GCL1 laboratory course, two participants reflected on how the surveys themselves were beneficial to recognizing development of skills. This is captured in EDC_4221’s comment stating that the reflective assignment was “very helpful to reflect on your own growth with transferable skills.” Another participant emphasized the importance of reflecting on these skills in recognizing development, stating “I think even though I didn’t know I was developing these transferable skills it is important to reflect on how much these skills did develop. They all are important things that everyone should learn” (EDC_4353). A handful of participants used the additional comments section to express their overall appreciation for the course (n = 11), grievances concerning their overall experience in the course (n = 1), and general suggestions for course changes not regarding EDCs integration and development (n = 4). Throughout the surveys six participants spoke of how learning the skills early on will allow for them to apply and grow these skills throughout the remainder of their college career and into their professional lives. This is reflected in EDC_5028’s statement “I think teaching students transferable skills early on in their academic career is very important, they will take those skills and use them through both their academic and occupational careers.” Overall these comments highlight how students appreciated not only being introduced to valuable 21st-century skills but how the experiences surrounding development themselves were valued and beneficial. These findings support a need to acquaint students with EDCs early and throughout their college careers, giving them opportunities to reflect, and providing instances in which they can apply these skills. A small percent (n = 11, 5%) of participants offered additional suggestions regarding EDC development in the course and how integration of these skills could be improved. These varied from person to person with no clear pattern emerging from participant responses. Some participants wished for the skills to be discussed or gone over in the course beyond being presented solely in the scenario documents. Others felt that incorporating a rubric for receiving feedback on how well students were applying skills or integrating the skills into the team contract that students complete at the beginning of the semester, would be valuable. Some also 294 desired incorporating how EDCs could be applied to each lab, in addition to the descriptions provided. Other suggestions students shared included a) wanting to develop different skills for each project, b) allowing for students to come up with their own skills they would like to work on in addition to the ones provided, c) wanting the opportunity to individually demonstrate competencies, d) wanting to explore careers that are related to what is being done within the labs, and e) wanting to speak with other teams to determine what is or is not working for them. Implications The findings from explored above show how students were able to build and develop valuable EDCs, both in-person and online, through curriculum that included elements of project-based learning and scientific practices. However, student responses highlighted areas of remote learning that could be further improved to facilitate skill development. Of key interest were the interpersonal skills of communication and teamwork and how remote hindered development of these skills. Facilitating learning environments that foster development of these skills online can be particularly challenging due to the barriers that online presents (Darby & Lang, 2019; Donelan & Kear, 2023). Results of this study show that students may need more resources and guidance to overcome these barriers and model effective teamwork. Additionally, reframing the online learning environment to show how adapting to virtual collaboration can result in beneficial 21st-century skills may also help students become accustomed to engaging in remote teamwork. Other findings from this study show that students desire integration of EDCs into their college curriculum, indicating that widescale implementation of this may be met positively by undergraduates. Although this may seem an incredible feat, recognizing small but relevant areas in which each individual courses can contribute to student career preparation and skill development can amass into an altogether valuable learning experience. While college may not be able to provide students with all the knowledge and skills needed to be successful in their chosen career, it can be but one of many experiences that contribute meaningfully to success in a student’s professional and personal life. Conclusions & Future Directions Both interview and survey participants were able to adequately identify skills that employers desire most in the modern workforce as important to their future careers, both with and without prompting. When prompted with the skills highly valued by employers, most students were able 295 find these valuable within the context of their own career goals. Not only were they able to see the importance of EDCs, but they were also able to articulate and connect experiences within the general chemistry laboratory courses to perceived development of these skills. Development as reported in both interviews and surveys primarily revolved around the collaborative learning environment aiding growth of communication and teamwork skills. Problem-solving & critical thinking and prioritization & time management were among other highly valued workplace skills that students reported to have experienced growth of in the general chemistry laboratory courses through course elements such as collaboration, open-inquiry learning, working with data, and independent management of tasks and assignments. Another key aspect investigated in this study was how the learning environment changed from in-person to online instruction. While the online learning environment returned divided perceptions of its positive or negative effects on communication, teamwork, problem-solving & critical thinking, and prioritization & time management skill development in interviews, qualitative review of survey participants found remote learning to be more detrimental to development of these skills than good. These findings emphasized how online learning resulted in deleterious effects on a student’s ability to communicate and collaborate with their teammates further resulting in a hinderance in growth of communication and teamwork skills. Additionally, while hands-on learning in the laboratory is more commonly related to development of technical skills through manual manipulation of tools and instruments, problem-solving & critical thinking skills and prioritization & time management faltered in the absence engaging with chemical equipment and conducting experiments. When continually introduced to EDCs throughout the course of a semester, students were better able to build awareness of valuable 21st-century skills, while also engaging in opportunities to develop and apply skills. Additionally, students desired more opportunities throughout their college career to develop EDCs and recognized their importance to life, career, and learning. All findings reported herein were reflective of student perceptions, lacking quantifiable measurements of student proficiencies in these skills. While this was not linked to the primary goals of this study, developing instruments to measure student abilities regarding EDCs would contribute a more accurate picture of student achievement and address prevalent areas in which education researchers and administrators can better address the skills gap. Although the literature 296 provides a great deal of commentary on methods used to measure these skills (National Academies, 2011), more standardized and cost-effective ways have yet to be produced. Additionally there is a need to come to a consensus on the definition of these skills, whether within specific disciplines or more broadly. Limitations While the various limitations surrounding specific tests have been discussed in the main body of this chapter, there are many additional limitations of administering surveys as a method of data collection. One potential limitation that can be connected to both use of surveys and interviews as primary sources of data collection is sampling bias. Sampling bias occurs because of two primary factors – coverage and non-response (Remler & Van Ryzin, 2015). Coverage bias is a result of a sample being an ill-representation of the population due to the exclusion of certain groups present within the population or inclusion of groups that are a part of the population of interest (Blair & Blair, 2020; Remler & Van Ryzin, 2015). Non-response bias is another form of sampling bias in which certain groups may have a higher likelihood to respond or participate, making the sample skewed in favorite of one group and not accurately reflecting the overall composition of the population (Gorard, 2013; Remler & Van Ryzin, 2015). Using a mixed-methods approach offered greater coverage of the student sample through gathering in-depth views of interview participants and collecting the perception of a larger student sample that offered more generalizability using surveys. To account for both coverage and non-response bias, independent samples t-tests comparing SAT scores were used to determine if there were salient differences between participating and non-participating samples, with no statistically significant differences found for either interviews or surveys. Regardless of the measures taken to ensure adequate sampling methods were employed in this study, valuable student beliefs that could have contributed to this research could have been lost due to – a) technological problems hindering a student’s ability to participate in web-based surveys or remote interviews (e.g., lack of access to Wi-Fi or study invitations being directed to spam folder), b) students who perceive the course as valuable to themselves and their future career having a greater affinity to respond over those who do not, or c) lack of time to engage in additional activities that do not pertain to relevant course work. Additionally, survey and interview responses may not be an accurate reflection of participant beliefs. This can be exacerbated by the way in which the question was formulated and presented to participants, as 297 well as survey fatigue or participants feeling they need to answer a certain way (Baxter et al., 2015; Remler & Van Ryzin, 2015). Beyond survey administration, analyzing open answer responses is interpretive in nature. It is possible that the methods of qualitative analysis employed in this study negate or overlook underlying meanings. This is an expected limitation when dealing with closed methods of data collection (e.g., surveys) that lack the ability to further explore possible meanings through use of methods such as interviews. This limitation can be seen in the notable difference between the interview and survey themes concerning development, where open inquiry learning constituted a major theme by 73% of interview participants in relation to problem-solving & critical thinking skill development but was only observed to be a minor theme by 25% of survey participants. Due to the nature of interviewing techniques allowing for more in-depth exploration of participant responses, it is possible that follow-up questioning inquiring into the often broad or vague statements provided by survey participants (e.g., “We are presented with a problem each project and we work together to figure out the best way to solve the problem” (EDC_2174) would have resulted in a higher prevalence of certain themes. This could also be said for the lack of mention of which specific course activities or project scenarios contributing to skill development that was more prevalent as minor themes in interview responses and was often a point of interest in further interview questioning versus the low recognition of these course components in survey responses. Additionally, the minor interview theme of work ethic skills being developed through maintaining integrity in work by not cheating, was not included in survey prompts, leading to a possible exclusion of this as a theme in survey data. Further, although findings from Sp21 reflect positive findings concerning EDC development and career preparation in the general chemistry laboratory courses, it is important to keep in mind that we could not validate if students were referring to EDCs in the scenario document prompt or something else. Additionally, it cannot be confirmed that students did in fact read through the skills in the scenario document over the course of the semester and is an aspect to be considered when assessing these results. Use of multiple-choice responses, as used to determine if students found EDC development in the course as valuable, are also restricted to a set number of choices and may not be reflective of a participant’s true beliefs (Treadwell, 2024). Although these limitations could have potentially affected the outcomes of these results, these findings suggest that most students who participated in Sp21 surveys found the course to 298 be beneficial in gaining awareness of EDCs and providing valuable opportunities for development. 299 Agresti, A. (2013). Following Up Chi-Square Tests. In Categorical Data Analysis (3rd ed., pp. 80–81). Wiley. REFERENCES Association of American Medical Colleges. (2023). Official Guide to Medical School Admissions: How to Prepare for and Apply to Medical School. https://store.aamc.org/downloadable/download/sample/sample_id/606/ Baxter, K., Courage, C., & Caine, K. (2015). Understanding Your Users: A Practical Guide to User Research Methods (2nd ed.). Morgan Kaufmann Publications. Blair, E., & Blair, J. (2020). Applied Survey Sampling. In Applied Survey Sampling. Sage Publications, Inc. Bretz, S. L., Fay, M., Bruck, L. B., & Towns, M. H. (2013). What Faculty Interviews Reveal About Meaningful Learning in the Undergraduate Chemistry Laboratory. Journal of Chemical Education, 90(3), 281–288. https://doi.org/10.1021/ed300384r Bruck, L. B., Towns, M., & Bretz, S. L. (2010). Faculty Perspectives of Undergraduate Chemistry Laboratory: Goals and Obstacles to Success. Journal of Chemical Education, 87(12), 1416–1424. https://doi.org/10.1021/ed900002d Bureau of Labor Statistics. (2023). Number of Jobs, Labor Market Experience, Marital Status, and Health for Those Born 1957-1964. www.bls.gov/nls Carpenter, J. R. (2020). Missing Data. In SAGE Research Methods Foundations. Sage Publications Ltd. https://methods.sagepub.com/foundations/missing-data Castelli, F. R., & Sarvary, M. A. (2021). Why Students Do Not Turn on Their Video Cameras During Online Classes and an Equitable and Inclusive Plan to Encourage Them to Do So. Ecology and Evolution, 11(8), 3565–3576. https://doi.org/10.1002/ece3.7123 Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences Second Edition. Lawrence Erlbaum Associates. Darby, F., & Lang, J. M. (2019). Small Teaching Online: Applying Learning Science in Online Classes. Jossey-Bass. DeKorver, B. K., & Towns, M. H. (2015). General Chemistry Students’ Goals for Chemistry Laboratory Coursework. Journal of Chemical Education, 92(12), 2031–2037. https://doi.org/10.1021/acs.jchemed.5b00463 Desire2Learn. (2023). Desire2Learn. https://d2l.msu.edu Díez-Pascual, A. M., & Jurado-Sánchez, B. (2022). Remote Teaching of Chemistry Laboratory Courses during COVID-19. Journal of Chemical Education, 99(5), 1913–1922. 300 https://doi.org/10.1021/acs.jchemed.2c00022 Donelan, H., & Kear, K. (2023). Online Group Projects in Higher Education: Persistent Challenges and Implications for Practice. Journal of Computing in Higher Education. https://doi.org/10.1007/s12528-023-09360-7 Evans, A. N. (2022). Inference with the F Distribution. In Using Basic Statistics in the Behavioral and Social Sciences. SAGE Publications, Inc. https://doi.org/10.4135/9781071878859 Frost, J. (2020). Hypothesis Testing: An Intuitive Guide for Making Data Driven Decisions (1st ed.). Statistics by Jim Publishing. Gault, J., Redington, J., & Schlager, T. (2000). Undergraduate Business Internships and Career Success: Are They Related? Journal of Marketing Education. Gorard, S. (2013). Research Design: Creating Robust Approaches for the Social Sciences. SAGE Publications, Inc. Gravetter, F. J., & Wallnau, L. B. (2017). Statistics for the Behavioral Sciences. In Statistics for the Behavioral Sciences . CENGAGE Learning. Gray, C. D., & Kinnear, P. R. (2012). IBM SPSS Statistics Made Simple 19 (1st ed.). Psychology Press. Hawkins, I., & Phelps, A. J. (2013). Virtual Laboratory vs. Traditional Laboratory: Which is More Effective for Teaching Electrochemistry? Chemistry Education Research and Practice, 14(4), 516–523. https://doi.org/10.1039/c3rp00070b Huizingh, E. (2012). Applied Statistics with SPSS. In Applied Statistics with SPSS. SAGE Publications, Ltd. https://doi.org/10.4135/9781446249390 IBM Corp. (2020). SPSS (27.0.1.0). Jones, E. (2006). Internships: Previewing a Profession. Occupational Outlook Quarterly, 16–18. Kim, E. B., Kim, K., & Bzullak, M. (2012). A Survey of Internship Programs for Management Undergraduates in AACSB-accredited Institutions. International Journal of Educational Management, 26(7), 696–709. https://doi.org/10.1108/09513541211263755 Kline, R. B. (2004). Beyond Significance Testing: Performing Data Analysis Methods in Behavioral Research. American Psychological Association. Kruskal, W. H., & Wallis, W. A. (1952). Use of Ranks in One-Criterion Variance Analysis. Journal of the American Statistical Association, 47(260), 583–621. 301 Kusnoor, A. V., & Stelljes, L. A. (2016). Interprofessional Learning Through Shadowing: Insights and Lessons Learned. Medical Teacher, 38(12), 1278–1284. https://doi.org/10.1080/0142159X.2016.1230186 Lee, S., & Lee, D. K. (2018). What is the Proper Way to Apply the Multiple Comparison Test? Korean Journal of Anesthesiology, 71(5), 353–360. https://doi.org/10.4097/kja.d.18.00242 Liu, R. (2022). Statistical Guideline #7 Adjust Type 1 Error in Multiple Testing. International Journal of Behavioral Medicine, 29(2), 137–140. https://doi.org/10.1007/s12529-022- 10070-0 Mader, F. H., D Mader, D. R., Alexander, E. C., & D, D. R. (2017). Job Shadowing Experiences Part of the Advertising and Promotion Management Commons, Arts Management Commons, Business and Corporate Communications Commons, Communication Commons, E-Commerce Commons, Fashion Business Commons, Marketing Commons, Operations and Supply Chain Management Commons. In Atlantic Marketing Journal (Vol. 5, Issue 3). https://digitalcommons.kennesaw.edu/amjAvailableat:https://digitalcommons.kennesaw.edu /amj/vol5/iss3/8 Means, B., Neisler, J., & with Langer Research Associates. (2020). Suddenly Online: A National Survey of Undergraduates During the COVID-19 Pandemic. https://files.eric.ed.gov/fulltext/ED610781.pdf National Academies. (2011). Assessing 21st Century Skills. In Assessing 21st Century Skills. National Academies Press. https://doi.org/10.17226/13215 National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, Board on Science Education, Cognitive Board on Behavioral, & Committee on How People Learn II: The Science and Practice of Learning. (2018). How People Learn II: Learners, Contexts, and Cultures. In How People Learn II: Learners, Contexts, and Cultures. National Academies Press. https://doi.org/10.17226/24783 National Research Council, & Committee on a Conceptual Framework for New K-12 Science Education Standards. (2012). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. In A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. National Academies Press. https://doi.org/10.17226/13165 Oswald, G. R., Alderman, L. A., & Willmering, P. (2017). Short-Term Job Shadowing Experience Benefits for Undergraduate Rehabilitation Students. Australian Journal of Rehabilitation Counselling, 23(2), 79–89. https://doi.org/10.1017/jrc.2017.2 Porter, K. E. (2018). Statistical Power in Evaluations That Investigate Effects on Multiple Outcomes: A Guide for Researchers. Journal of Research on Educational Effectiveness, 11(2), 267–295. https://doi.org/10.1080/19345747.2017.1342887 302 Qualtrics. (2023). Qualtrics. https://qualtrics.msu.edu Reid, H. M. (2022). Finding Differences With Nominal Data—II : The Chi-Square Test of Independence. In Introduction to Statistics: Fundamental Concepts and Procedures of Data Analysis. SAGE Publications, Inc. https://doi.org/10.4135/9781071878927 Remler, D., & Van Ryzin, G. G. (2015). Research Methods in Practice: Strategies for Description and Causation (2nd ed.). Sage Publications, Inc. Silva, P., Lopes, B., Costa, M., Seabra, D., Melo, A. I., Brito, E., & Dias, G. P. (2016). Stairway to employment? Internships in higher education. Higher Education, 72(6), 703–721. https://doi.org/10.1007/s10734-015-9903-9 Slavin, R., & Smith, D. (2009). The Relationship Between Sample Sizes and Effect Sizes in Systematic Reviews in Education. Educational Evaluation and Policy Analysis, 31(4), 500– 506. https://doi.org/10.3102/0162373709352369 Smyth, J. D., Christian, L. M., & Dillman, D. A. (2008). Does “Yes or No” on the Telephone Mean the Same as “Check-all-that-apply” on the Web? Public Opinion Quarterly, 72(1), 103–113. https://doi.org/10.1093/poq/nfn005 Smyth, J. D., Dillman, D. A., Christian, L. M., & Stern, M. J. (2006). Comparing Check-all and Forced-choice Question Formats in Web Surveys. Public Opinion Quarterly, 70(1), 66–77. https://doi.org/10.1093/poq/nfj007 Sullivan, G. M., & Feinn, R. (2012). Using Effect Size—or Why the P Value Is Not Enough . Journal of Graduate Medical Education, 4(3), 279–282. https://doi.org/10.4300/jgme-d-12- 00156.1 Treadwell, D. (2024). Surveys: Putting Numbers on Opinions. In Introducing Communication Research: Paths of Inquiry (5th ed., p. 123). SAGE Publications, Inc. VERBI Software. (2021). MAXQDA 2022. maxqda.com Verma, J. P., & Verma, P. (2020). Understanding Statistical Interference. In Determining Sample Size and Power in Research Studies A Manual for Researchers (pp. 25–26). Springer. Wart, A. Van, O’brien, T. C., Varvayanis, S., Alder, J., Greenier, J., Layton, R. L., Stayart, C. A., Wefes, I., & Brady, A. E. (2020). Applying Experiential Learning to Career Development Training for Biomedical Graduate Students and Postdocs: Perspectives on Program Development and Design. CBE Life Sciences Education, 19(3), 1–12. https://doi.org/10.1187/cbe.19-12-0270 Williams, N. D., Gallardo-Williams, M. T., Griffith, E. H., & Bretz, S. L. (2022). Investigating Meaningful Learning in Virtual Reality Organic Chemistry Laboratories. Journal of 303 Chemical Education, 99(2), 1100–1105. https://doi.org/10.1021/acs.jchemed.1c00476 Wolinsky-Nahmias, Y., & Auerbach, A. H. (2022). Evaluating the Design and Benefits of Internship Programs. Journal of Political Science Education, 18(4), 584–604. https://doi.org/10.1080/15512169.2022.2109481 Wu, M. J., Zhao, K., & Fils-Aime, F. (2022). Response Rates of Online Surveys in Published Research: A Meta-analysis. Computers in Human Behavior Reports, 7. https://doi.org/10.1016/j.chbr.2022.100206 304 APPENDIX A.VI.1. Response Rate of Surveys Table 6.8. Descriptive statistics of survey administration and response rates of surveys. Semester Fa19 Survey CDS Sp20 CDS Su20 CDS Fa20 CDS Sp21 Fa21 RA IRA FRA Students Invited to Participate in Survey (n) 1905 1463 442 1645 1065 580 191 123 69 1881 1432 450 1075 1075 2063 1590 473 2063 1590 473 Surveys Returned, Started, or Opened (n) 1295 995 300 1032 667 365 108 70 38 1181 900 281 233 233 1318 1028 289 1150 881 269 Response Rate of Survey (%) 68 68 68 63 63 63 57 57 55 63 63 62 22 22 64 65 61 56 55 57 Course Total GCL 1 GCL 2 Total GCL 1 GCL 2 Total GCL 1 GCL 2 Total GCL 1 GCL 2 Total GCL 1 Total GCL 1 GCL 2 Total GCL 1 GCL 2 *Abbreviations denoting survey type mean the following: CDS = course development survey, RA = reflective assignment, IRA = initial reflective assignment, and FRA = final reflective assignment. 305 A.VI.2. Survey Participant Exclusion, Missing Values, & Final Sample Sizes Managing data to account for participants who were under the age of consent, duplicate participants, and those who returned incomplete surveys, multiple steps were taken. First, participants who opted out of the study or were under the age of consent were removed from the sample (Table 6.9). After these participants were removed from the sample, each dataset was checked for duplicate participants, those who submitted more than one survey during a particular semester or survey administration. Duplicate participants were attended to in one of two ways: 1) a participant’s first recorded response was kept for analysis if the response was fully complete, leading to the second recorded response being removed from analyses, or 2) if a participant had a partially complete or blank survey recorded for the first response and the second response was fully complete, the second recorded response was kept for data analyses while the first response was removed from the dataset. First recorded responses were the most desirable to maintain in data analysis, due to the belief that participants were responding with their initial thoughts and priming was not present to potentially influence or adjust a student’s response. Duplicate participants were not commonplace and were most likely the result of either 1) some surveys lacking the presence of a “back” button, causing participants to advance through the survey and request a new link or 2) an accidental error within survey solicitations causing the investigator to send duplicate emails participants to believe they needed to submit more than one survey to receive credit. Anonymous participants, although rare, were also removed. Within the Fa21 IRA survey sample (Table 6.9) a participant failed to identify themselves. This lack of identification caused the survey participant to be removed from the sample, due to the inability to determine if this participant opted out of the study or were under the age of consent. A lack of identity confirming information also affected a small portion of the sample in Fa19, Sp20, Su20, Fa20, and Fa21 semesters, where minute subsets of the samples returned no registrar information, so the age of consent to participate was not able to be validated (Table 6.9). 306 Table 6.9. Criteria for removal of participants based on opt-out, lack of registrar information, duplicate surveys, and incomplete surveys (n). Semester Fa19 Survey Course Opt-Out CDS Total 21 Sp20 CDS Su20 CDS Fa20 CDS Sp21 RA Fa21 IRA FRA GCL 1 GCL 2 Total GCL 1 GCL 2 Total GCL 1 GCL 2 Total GCL 1 GCL 2 Total GCL 1 Total GCL 1 GCL 2 Total GCL 1 GCL 2 18 3 10 8 2 2 2 N/A 16 15 1 5 5 53 44 9 47 39 8 No Registrar Info on File Primary Cases Duplicate Cases Incomplete Surveys 4 2 2 1 1 N/A 1 1 N/A 2 1 1 N/A N/A 11* 8 2 3 2 1 1270 975 295 1021 658 363 105 67 38 1163 884 279 228 228 1251 973 278 1101 841 260 N/A N/A N/A N/A N/A N/A 4 4 N/A N/A N/A N/A N/A N/A 3 3 N/A 2 1 1 32 25 7 35 23 12 N/A N/A N/A 48 36 12 14 14 29 25 4 38 31 7 *Denotes an anonymous participant who lacked identifying information. Abbreviations denoting survey type mean the following: CDS = course development survey, RA = reflective assignment, IRA = initial reflective assignment, and FRA = final reflective assignment. Once the dataset was modified to account for obligatory participant removal and the exclusion of idiosyncratic cases, each semester was analyzed for the presence of incomplete responses or missing values. Missing values were assigned to the dataset according to the question being asked and the variable type. Open answer survey responses were considered incomplete or missing if a participant left either a blank or random alphanumeric response. Random alphanumeric responses included respondents who answered a question with random letters, numbers, or symbols as their full response (e.g., “.”, “x”, etc.). Multiple choice questions were marked as incomplete if no response was recorded. Questions in which respondents were 307 asked to rank skills based on perceived importance to their career goal were considered incomplete if a skill was missing a numerical ranking. When coding for missing responses, Fa20 presented a special case as Fa20 contained a display logic question, in which a participant was only directed to an additional question if they answered a previous question with a certain response. Participants who responded “Yes” to having had an internship or shadowing experience were advanced to an open answer prompt asking them to provide additional details of their experiences. For participants who answered “No”, no further questions asking participants to elaborate on their internship/shadowing experiences were displayed, known as skip logic. Because of this, if a participant answered “Yes” to the multiple-choice question and did not provide an open answer response, the open answer response was considered a missing or incomplete value. Additionally, because skip logic produces intentionally missing values, participants who answered “No” to the multiple-choice question did not have the follow-up open answer response treated as a missing value. Multi-response questions and additional questions were other important exceptions when investigating missing values. If a participant did not select an option within a multi-response question, it could be the outcome of a particular choice being excluded from the participant’s selection process. It is not always indicative that a participant ignored or skipped the question (Smyth et al., 2008). Questions that prompted additional comments and/or suggestions based on prior answers were not treated as missing values if a participant chose not to respond. These questions were designed to be open spaces for additional thoughts but were not designated as key survey questions. An exception to this was within the Fa21 FRA surveys, that contained a question prompting students to provide thoughts on additions that could be made to the course to better include skill development. This was considered part of the primary questions, so missing responses were treated as incomplete. After identifying and removing participants with incomplete survey responses based on the guidelines outlined above, the number of complete surveys used in analysis, completion rate, and final sample size resulted in the data compiled in Table 6.10 on the following page (page 308). 308 Table 6.10. Completion rate of surveys and final sample size. Semester Fa19 Survey CDS Sp20 CDS Su20 CDS Fa20 CDS Sp21 Fa21 RA IRA FRA Complete Surveys (n) 1238 950 288 986 635 351 101 63 38 1115 848 267 214 214 1222 948 274 1063 810 253 Completion Rate (%) 97 97 98 97 97 97 96 94 100 96 96 96 94 94 98 97 99 97 96 97 Final Sample Size (n) 1238 950 288 986 635 351 101 63 38 1115 848 267 214 214 1222 948 274 1063 810 253 Course Total GCL 1 GCL 2 Total GCL 1 GCL 2 Total GCL 1 GCL 2 Total GCL 1 GCL 2 Total GCL 1 Total GCL 1 GCL 2 Total GCL 1 GCL 2 *Abbreviations denoting survey type mean the following: CDS = course development survey, RA = reflective assignment, IRA = initial reflective assignment, and FRA = final reflective assignment. 309 A.VI.3. Missing Values Synopsis Analysis of missing values primarily resulted in expected trends. Using Figures 6.10, 6.12, 6.14, 6.16., 6.18, 6.20, and 6.22 to explore these trends, a cell is coded for either containing a response (no color) or missing a response (colored red) to depict the major response patters. Additionally, the percent of participants who fit within each patter is located in Figures 6.11, 6.13, 6.15, 6.17, 6.19, 6.21, and 6.23. The primary pattern observed was surveys in which all responses items were completed (Pattern 1). Sp21 was no exception to this, as Pattern 1 was missing an optional question, indicating that the remaining questions were fully completed. The stepwise drop off that is clearly depicted in Fa19, Sp20, Fa21 IRA, and Fa21 FRA was also an expected trend, in which participants gradually drop off and stopped responding as the questioning progressed, also referred to as a monotone pattern (Lavrakas, 2008). This is captured in the Fa20 and Sp21 data as well, although it is harder to visualize with the inclusion of optional or additional responses overshadowing this trend. Su20 presented a unique scenario in which all surveys returned were fully completed. The one pattern that contained a missing value pertains to a sole case, in which a student had responded “No” to the multiple-choice question of whether they had a shadowing/internship experience and answered the follow-up question with a dash mark. This type of response can be expected in which a follow-up response is not applicable. When addressing the percent participants who fell within each pattern of missing values, overwhelmingly Pattern 1 was the most common pattern for all semesters. Pattern 1 was representative of surveys that were returned to full completion, a trend that can also be seen within the high completion rate for each semester reflected in (Table 6.9). Although it may be perceived that Sp21 is the exception to this theme, the most common pattern, Pattern 4, was representative of participants completing all questions besides additional or optional questions. All corresponding figures are present on pages 311 – 317. 310 Figure 6.10. Fa19 missing values pattern analysis (D = question/prompt exploring development in course(s), N = question/prompt exploring skills needed for career goal). 100 97.5 ) % ( s e s a C f o t n e c r e P 90 80 70 60 50 40 30 20 10 0 1.1 7 0.7 5 1 0.4 4 Missing Values Pattern 0.2 6 0.1 3 0.1 2 Figure 6.11. Fa19 missing values percent of cases (cases denote number of participants)(%). 311 Figure 6.12. Sp20 missing values pattern analysis (D = question/prompt exploring development in course(s), N = question/prompt exploring skills needed for career goal). 100 96.6 ) % ( s e s a C f o t n e c r e P 90 80 70 60 50 40 30 20 10 0 0.9 5 0.8 7 1 0.6 0.5 0.2 8 4 9 Missing Values Pattern 0.2 3 0.2 2 0.1 6 Figure 6.13. Sp20 missing values percent of cases (cases denote number of participants)(%). 312 Figure 6.14. Su20 missing values pattern analysis (D = question/prompt exploring development in course(s), N = question/prompt exploring skills needed for career goal, C = question/prompt exploring skills wanted to develop in course but did not yet have chance to). ) % ( s e s a C f o t n e c r e P 100 90 80 70 60 50 40 30 20 10 0 99.1 1 Missing Values Pattern 0.9 2 Figure 6.15. Su20 missing values percent of cases (cases denote number of participants)(%). 313 Figure 6.16. Fa20 missing values pattern analysis (D = question/prompt exploring development in course(s), N = question/prompt exploring skills needed for career goal, C = question/prompt exploring skills wanted to develop in course but did not yet have chance to). ) % ( s e s a C f o t n e c r e P 100 90 80 70 60 50 40 30 20 10 0 62.8 33.8 9 1 1.0 18 0.8 16 0.7 19 0.3 8 0.3 11 0.2 5 0.2 17 0.1 4 Missing Values Pattern Figure 6.17. Fa20 missing values percent of cases (cases denote number of participants)(%). Only the 10 most frequently occurring patterns are shown in the chart. 314 Figure 6.18. Sp21 missing values pattern analysis (D = question/prompt exploring development in course(s)). ) % ( s e s a C f o t n e c r e P 80 70 60 50 40 30 20 10 0 73.7 7.5 1 4 6.1 5.7 5.3 6 7 Missing Values Pattern 3 1.3 2 0.4 5 Figure 6.19. Sp21 missing values percent of cases (cases denote number of participants)(%). 315 Figure 6.20. Fa21 IRA missing values pattern analysis (U = question/prompt exploring skills wanted to develop during undergraduate career). 100 97.7 ) % ( s e s a C f o t n e c r e P 90 80 70 60 50 40 30 20 10 0 1.3 6 1 0.5 5 0.3 3 0.2 2 0.1 4 Missing Values Pattern Figure 6.21. Fa21 IRA missing values percent of cases (cases denote number of participants)(%). 316 Figure 6.22. Fa21 FRA missing values pattern analysis (D = question/prompt exploring development in course(s)). 100 96.5 ) % ( s e s a C f o t n e c r e P 90 80 70 60 50 40 30 20 10 0 1.8 5 0.5 8 1 0.4 0.3 0.2 7 3 9 Missing Values Pattern 0.1 6 0.1 4 0.1 2 Figure 6.23. Fa21 FRA missing values percent of cases (cases denote number of participants)(%). 317 A.VI.4. Comparison of Survey Sample Participants to Non-participating Students The same method of testing used to determine if interview participants differed from the remaining course population, through application of independent samples t-testing, was also used for survey participants. The participating sample (survey participants) were defined as the sample of students used in analyses after data cleaning. The non-participating sample (non- participants) were survey participants removed from analyses due to incomplete or blank surveys, students who did not start or return a survey, and students who participated in the interview process for the corresponding semester under investigation. Composite SAT were initially used to determine if survey sample participants were representative of the non- participant student sample. As the study progressed, reporting SAT scores (and/or ACT scores that were sometimes used in conjunction with concordance tables to convert to SAT scores) for admission became optional. College admission shifted to a reliance on math placement scores, which affected this study’s representation for SP21 and FA21 semesters(Michigan State University: Office of Admissions, n.d.). Not all participants had either SAT or math placement scores reported, resulting in the missing values synopsis in Table 6.11. Only one comparison was made for Fa19, Sp20, Su20, Fa20, and Sp21 semesters, where one survey was administered. However, Fa21 facilitated the necessity of making multiple comparisons as it consisted of an initial (IRA) and final reflective survey (FRA), in which participants engaged in either one or both surveys. Because the results of Fa21 IRA and FRA are analyzed at separate times, two sets of comparison tests were conducted. The first comparison test comprised of only those participants who took the initial reflective assignment (Only IRA Survey Participants) against non-IRA participants (participants who could be present in the FRA sample but not the IRA sample). The second comparison test included participants who only completed the final reflective assignment (Only FRA Survey Participants) against non-FRA participants (participants who could be present in the IRA sample but not the FRA sample). The data used in this testing was assumed to be normally distributed due to the theory that large sample sizes that exceed n> 30 follow the central limit theorem and can be assumed to follow a normal distribution satisfying the assumption for normality (Ghasemi & Zahediasl, 2012; Lumley et al., 2002; Singh, 2015). Most semester samples met the assumption of equal variances (Table 6.12). However, comparison of Fa21 FRA returned a violation of the assumption of equal variances. This 318 violation facilitated the use of a different t-test value that factored in the presence of unequal variances. Fa19, Sp20, Su20, Sp21, and all Fa21 testing returned insignificant t-test results (a > 0.050) and retained the null hypothesis that there was no difference in mean test scores (Table 6.13). This indicated that the survey samples were a satisfactory representation of the population. Fa20 returned a significant result (p = 0.006), rejecting the null hypothesis that there was no difference in mean test scores. However, the effect size of this result (d = 0.168) was below the cutoff for a small effect (d=0.20) and the difference between mean test scores was determined to be trivial due to the large sample sizes used in testing. Within each semester, the participating sample was deemed an acceptable representation of the non-participating sample based on the comparison of mean SATX and math placement scores. It is important to note that these tests were not conducted to generalize study findings to the overall population, but to confirm that the sample was an accurate representation and did not deviate significantly from the non- participating sample. All corresponding tables are located on pages 320 – 321. 319 Table 6.11. Samples used in analysis of SAT scores. Semester Fa19 Sp20 Su20 Fa20 Sp21 Fa21 Sample All Survey Participants All Survey Participants All Survey Participants All Survey Participants All Survey Participants All Survey Participants Only IRA Survey Participants Only FRA Survey Participants Recurring IRA & FRA Survey Participants Total (n) 1238 Scores Used in Analysis (n) 1201 Missing Scores (n) 37 Missing scores (%) 3 627 986 661 112 72 1115 753 214 852 1469 570 1222 817 1063 976 815 1224 611 967 638 102 64 1078 728 180 719 1213 449 1003 659 897 765 686 976 16 19 23 10 8 37 25 34 133 256 121 219 158 166 211 129 248 3 2 3 9 11 3 3 16 16 17 21 18 19 16 22 16 20 Participant Non- participant Participant Non- participant Participant Non- participant Participant Non- participant Participant Non- participant Participant Non- Participant Participant Non- Participant Participant Non- Participant Participant Non- Participant Table 6.12. Levene’s test for homogeneity of variances of SAT scores. Levene's Test Semester Fa19 Sp20 Su20 Fa20 Sp21 Fa21 Sample All Survey Participants All Survey Participants All Survey Participants All Survey Participants All Survey Participants IRA Survey Participants FRA Survey Participants Outcome Equal Variances Assumed Equal Variances Assumed Equal Variances Assumed Equal Variances Assumed Equal Variances Assumed Equal Variances Assumed Equal Variances Not Assumed df 1810 1603 164 1804 897 1660 1660 Test Scores SAT Value 2.297 Significance 0.130 0.760 0.140 0.800 0.937 0.216 0.037 SAT SAT SAT Math Placement Math Placement Math Placement 0.093 2.202 0.064 0.006 1.533 4.359 320 Table 6.13. Independent samples t-test of SAT scores. Semester Fa19 Sp20 Su20 Fa20 Sp21 Fa21 Sample All Survey Participants All Survey Participants All Survey Participants All Survey Participants All Survey Participants Only IRA Survey Participants Only FRA Survey Participants Participant Non- participant Participant Non- participant Participant Non- participant Participant Non- participant Participant Non- participant Participant Non- Participant Participant Non- Participant Test Scores SAT Sample Mean (μ) 1240.17 Value 2.168 t-test Significance (p - value) 0.112 df 1810 SAT SAT SAT MP MP MP 1225.89 1217.03 1219.91 1206.26 1241.2 1224.00 1202.53 15.07 15.14 16.56 16.23 16.52 16.34 -0.441 0.791 1603 -1.755 0.219 164 3.494 0.006* 1804 -0.145 0.953 897 1.159 0.437 1660 0.637 0.704 1639. 97 *Use of t-test value that failed the assumption of homogeneity of variances caused differences seen in degrees of freedom reported for these samples. †SAT = SAT scores and MP = Math Placement scores. 321 A.VI.5. Participant Demographics Survey demographics disaggregated by semester around found below within Table 6.14. Chi- square test for association was used to determine differences between the categorical variables of gender, first-generation status, class standing, and ethnicity. Although some assumptions were outlined in the main text, further assumptions needed to be satisfied for use of this test. An additional assumption of these tests is that samples are independent of one another, meaning that a participant present in one group (e.g., first-generation student in Fa21 semester) may not be present in another (e.g., first-generation student in Sp21 semester). This assumption was violated for a small subsample of participants (Table 6.15). A total of n = 390 participants, across the six semesters in which this study was conducted, were found to have participated in surveys during a maximum of two different semesters. However, because the aim of this testing was not to determine relationships between pre/posttests, and because the affected sample size was comparatively small, this was not believed to affect the outcomes of these tests. An additional assumption is that the lowest observed cell count, of any cell in the contingency table, be ≥ 5. If an observed cell count was lower than 5, this facilitated the use of reporting the Fisher Exact Test value and associated p-value in place of either Pearson Chi- Square or Yates Correction for Continuity. While most significance detected for demographic variables explored in the main text did not have sufficient effect sizes to warrant further exploration, class standing and age prompted further investigation due to significance accompanied by a small effect size. Accounting for the differences between class standing and semester, Table 6.16 displays the semesters contributing to significance. The outcomes of the Games-Howell post hoc analysis of age across semesters in Table 6.17 displays the variables contributing to the small effect size found for the significant ANOVA test. 322 Table 6.14. Survey participant demographics, n (%). Fa19 (n = 1238) Sp20 (n = 986) Su20 (n = 112) FA20 (n = 1,115) Sp21 (n = 214) Fa21 (n = 1,469) Semester Course GCL1 GCL2 Legal Sex Female Male First Generation Status First Generation Student Continuing Generation Student Class Standing Lower Classman Upper Classman Age Ranges 18-20 21-23 24+ Mean 950 (76) 288 (23) 713 (58) 525 (42) 251 (20) 987 (80) 1,006 (81) 232 (19) 1,111 (90) 108 (9) 19 (2) 19.1 635 (64) 351 (36) 613 (62) 373 (38) 216 (22) 770 (78) 797 (81) 189 (19) 875 (89) 106 (11) 5 (1) 19.1 70 (62) 42 (38) 60 (54) 52 (46) 24 (21) 88 (79) 51 (46) 61 (54) 70 (63) 36 (32) 6 (5) 20.5 848 (76) 214 (100) 267 (24) N/A 1,131 (77) 338 (23) 661 (59) 454 (41) 138 (64) 76 (36) 223 (20) 60 (28) 902 (61) 567 (39) 325 (22) 892 (80) 154 (72) 1,144 (78) 836 (75) 279 (25) 178 (83) 36 (16) 1,168 (80) 301 (20) 956 (86) 142 (13) 17 (2) 19.3 195 (91) 18 (8) 1 (1) 18.8 1,326 (90) 119 (8) 24 (2) 19.1 Total (n = 5,134) 3,848 (75) 1,286 (25) 3,087 (60) 2,047 (40) 1,099 (21) 4,035 (79) 4,036 (79) 1,098 (21) 4,533 (88) 529 (10) 72 (1) 19.2 *The Fa21 sample contains both Fa21 IRA and FRA participants combined, with students who took both surveys during this semester only being represented once. §Sums of percentages may add up to greater than 100% due to rounding. Percentages were rounded up or down based on the tenths decimal place according to the following guidelines: 1) 0.5 the percentage was rounded up, 2) <0.5 the percentage was rounded down, and 3) if there were two categories within a variable that had 0.5 in the tenths place the percentage was rounded up if following an odd number and down if following an even number. 323 Table 6.15. Recurring participants by semester (n). First Semester Survey was Taken Fa19 Sp20 Su20 Fa20 Sp21 Last Semester Survey was Taken Sp20 167 — — — — Su20 3 13 — — — Fa20 47 Sp21 1 Fa21 10 56 7 — — — — — — 12 1 39 34 Table 6.16. Adjusted residuals of class standing by semester (Bonferroni adjusted critical value for residuals ± =2.9). Semesters Tested Fa19 x Lower Classman Fa19 x Upper Classman Sp20 x Lower Classman Sp20 x Upper Classman Su20 x Lower Classman Su20 x Upper Classman Fa20 x Lower Classman Fa20 x Upper Classman Sp21 x Lower Classman Sp21 x Upper Classman Fa21 x Lower Classman Fa21 x Upper Classman Adjusted Residual 2.6 -2.6 Expected Count 973.9 264.1 Observed Count 1006 232 1.8 -1.8 -8.5* 8.5* -3.4* 3.4* 1.6 -1.6 0.9 -0.9 775.6 210.4 79.5 21.5 877.1 237.9 168.3 45.7 1155.6 313.4 797 189 45 56 836 279 178 36 1168 301 *Adjusted residuals above or below Bonferroni adjusted critical value. 324 Table 6.17. Games-Howell post hoc analysis of age between semesters (Bonferroni adjusted critical value of α = 0.00). Reference Semester Fa19 Mean of Reference Semester 19.09 Sp20 19.11 Su20 20.61 Fa20 19.35 Sp21 18.83 Fa21 19.12 Semester Tested Against Sp20 Su20 Fa20 Sp21 Fa21 Fa19 Su20 Fa20 Sp21 Fa21 Fa19 Sp20 Fa20 Sp21 Fa21 Fa19 Sp20 Su20 Sp21 Fa21 Fa19 Sp20 Su20 Fa20 Fa21 Fa19 Sp20 Su20 Fa20 Sp21 Mean Difference -0.02 -1.53 -0.26 0.26 -0.03 0.02 -1.51 -0.24 0.28 -0.01 1.53 1.51 1.26 1.79 1.50 0.26 2.44 -1.26 0.52 0.24 -0.26 -0.28 -1.79 -0.52 -0.29 0.03 0.01 -1.50 -0.24 0.29 Significance 1.000 0.001* 0.001* 0.058 1.000 1.000 0.001* 0.001* 0.022 1.000 0.001* 0.001* 0.001* 0.001* 0.001* 0.001* 0.001* 0.001* 0.001* 0.001* 0.058 0.022 0.001* 0.001* 0.016 1.000 1.000 0.001* 0.001* 0.016 325 A.VI.6. Benjamini-Hochberg & Bonferroni Adjustment of p-values Methodology Use of the BH-FDR method was optimal for adjustment of primary hypothesis test values due to its decreased sensitivity for the large volume of testing conducted throughout this study. The BH-FDR method involved organizing p-values from smallest to largest and assigning rankings (i) to the organized p-values, with the smallest p-value having a ranking of 1 (Benjamini & Hochberg, 1995). Once rankings were assigned, the number of tests (m) and conventionally used false discovery rate (q) of 0.050 were determined and a new critical value was defined using the equation (i/m)*q. Statistical Analysis Software (SAS) was used to further compute adjusted p- values based on these parameters. All values reported were based on the SAS output. Values below 𝛼 = 0.050 were determined to be significant. The script used to perform these adjustments in SAS can be found below. Script used in SAS to adjust p-values: libname a "C:\Users\eggly\OneDrive\Desktop"; run; (IMPORT EXCEL); proc contents data=a.full_list varnum; run; proc freq data=a.full_list; table Main; run; proc means data=a.full_list; where Main EQ 'Main'; var raw_p; run; PROC MULTTEST inpvalues=a.full_list holm hoc fdr out=a.adjusted; run; (MANUAL EXPORT); To adjust for multiple comparisons, within ANOVA hypothesis tests and adjusted residuals, a new critical value was defined using the B-FWER method – the number of subtests (m) were divided by the original critical value (𝛼 = 0.050) using the formula 𝛼/𝑚 (Lewis-Beck et al., 2012). For ANOVA post hoc testing, no further calculations were made. However, since adjusted residuals were in the form of standard z scores, the adjusted critical value then had to be converted to a standardized score. Original ANOVA post hoc p-values that fell below the 326 adjusted critical value or adjusted residuals that were above or below the newly defined critical value threshold were determined to be significant. 327 A.VI.7. Guidelines for Reporting & Interpreting Effect Sizes Effect size applied throughout this dissertation relied on the following guidelines presented in Table 6.18. Further explanation regarding these values are also explored below. Table 6.18. Effect size ranges and magnitudes. Effect Size Cohen’s d (d) Related Test Independent Samples t- test Eta-squared (𝜂2) ANOVA Phi (φ) (Also applicable for Cramer’s V (V) with the lowest degree of freedom = 1) Chi-square Test for Association Cramer’s V (V)(lowest degree of freedom = 2) Chi-square Test for Association Cramer’s V (V) (lowest degree of freedom ≥ 3) Chi-square Test for Association Range 0.00 - 0.20 0.20 - 0.50 0.50 - 0.80 ≥ 0.80 0.00 - 0.01 0.01 - 0.06 0.06 - 0.14 ≥ 0.14 0.00 - 0.09 0.10 - 0.30 0.30 - 0.50 ≥ 0.50 0.00 - 0.07 0.07 - 0.21 0.21 - 0.35 ≥ 0.35 0.00 - 0.06 0.06 - 0.17 0.17 - 0.29 ≥ 0.29 Magnitude of Effect Negligible Small Effect Medium Effect Large Effect Negligible Small Effect Medium Effect Large Effect Negligible Small Effect Medium Effect Large Effect Negligible Small Effect Medium Effect Large Effect Negligible Small Effect Medium Effect Large Effect Developed by Jacob Cohen, Cohen’s d (d) is used to report effect sizes for independent samples t-tests and represents the standardized mean difference between two samples (Cohen, 1988; Lakens, 2013; Salkind, 2012). Guidelines on interpreting the magnitude of d as defined by Cohen are outlined in Table 6.18. Eta-squared (𝜂2) measures the relationship between the variance of a dependent variable (e.g., age) and how much of this variance is attributed to the independent samples being tested (e.g., semester) for analysis of variance, or ANOVA (Cohen, 1988; Gravetter & Wallnau, 2017; Lakens, 2013). The effect sizes and ranges for interpretation of 𝜂2 as characterized by Cohen are illustrated in Table 6.18 (Cohen, 1988). Originally proposed as a bias corrected measure to 𝜂2 by Truman Kelly, epsilon-squared (𝜀2) measures the relationship between the variance in mean ranks of samples and the independent ordinal variable (Allen, 2018; Jaccard & Becker, 1990; Kelley, 1935). This measure of effect size does not have 328 clearly defined ranges for interpretation but follows the convention similar to r2, where values of 0 represent a lack of relationship between the dependent and independent variable and values of 1 represent a perfect relationship (Jaccard & Becker, 1990). Effects sizes corresponding to the chi-square test of association (𝜒2) relied on two separate values depending on the size of the contingency table. Phi (φ) coefficient was reported for traditional 2x2 contingency tables, whereas Cramer’s V (V) was used for larger contingency tables (Cohen, 1988; Reid, 2022). Cohen outlined general guidelines for interpretation of φ and V located in Table 6.18 (Cohen, 1988). When interpreting Cramer’s V, the degrees of freedom play an important role in determining the range and magnitude of the effect size (Gravetter & Wallnau, 2017). To determine what magnitude to use, the lowest degree of freedom (df) for either the row or column is considered and used to interpret the ranges outlined in Table 6.18. To exemplify this, a 2 x 3 contingency table would have a row count (n) = 2 and a column count (n) = 3. The degrees of freedom (df = n - 1) for the row would equal 1 and for the column would equal 2. Since 1 is the lower degrees of freedom, the ranges outlined for phi coefficient would be applicable for this contingency table. 329 A.VI.8. Fa19 Pilot Survey Opening Text. Employers who hire MSU graduates for science, engineering, and health-related careers have identified the most important skills they are seeking in new hires. You will be asked about nine of these skills in this survey. We would like your perspective on which of these skills you are building in your general chemistry lab course (CEM 161 or CEM 162). We would also like your thoughts on which skills will be helpful in your planned career. Your individual responses will not be viewed by your TA or your course coordinator. We encourage you to take this survey on a laptop or desktop computer, which will make it easier to respond to questions, particularly open-ended questions requiring an input of text. We request your permission to examine these data for our research. Your participation is completely voluntary, anonymous, and confidential. You will receive credit for completing the survey regardless of whether you agree to participate in this research or not. Your responses will have no bearing on your course grade or any other course at Michigan State University. You must be 18 or older to participate in this research project. If you do not wish to participate or are younger than 18 years old, please email [redacted email] and include your name and course number. Question 1: What is your current career goal? (Open Answer- forced entry) Question 2: Which general chemistry lab course are you currently enrolled in? (Multiple Choice- forced entry) Multiple Choice 1: CEM 161 (General Chemistry Laboratory I) Multiple Choice 2: CEM 162 (General Chemistry Laboratory II) Question 3: Which of these skills are you developing in your general chemistry lab course? Identify skills that you will need for your future career. Check the boxes that apply. (Multi- response) Skills to select from for either “Gen Chem Lab” or “Future Career” and the descriptions provided: 330 a) Communicating Effectively: conveying information clearly, making persuasive arguments, listening carefully. b) Contributing to a Team: doing a fair share of work, valuing contributions of others, identifying and using strengths of team members to complete project, knowing when to lead and when to follow. c) Developing Professional/Technical Skills: mastering new tools and techniques to achieve goals, selecting appropriate methods or tools. d) Embracing Change: accepting change and seeing it as an opportunity, being open to new approaches. e) Acquiring Knowledge: absorbing new information and concepts, connecting related concepts, identifying appropriate outside resources. f) Managing Time and Priorities: dividing a large project into smaller manageable tasks and developing a plan, distinguishing essential from trivial, prioritizing urgent tasks, staying on task. g) Solving Problems: defining a problem, developing and executing a plan, drawing on multiple perspectives, using reliable outside sources to guide solutions. h) Working with Data: sorting through data, selecting appropriate methods for analysis, making sense of conflicting information, recognizing assumptions in analysis. i) Working in a Diverse Environment: valuing different perspectives and experiences, using differences to achieve better results, identifying and working to overcome biases. j) Validation: for survey validation please select “Gen Chem Lab” Question 4a: Select one of the nine professional skills listed below that you are developing in general chemistry lab. (Multiple Choice – forced responses) Multiple Choice 1: Communicating Effectively Multiple Choice 2: Contributing to a Team Multiple Choice 3: Developing Professional/Technical Skills Multiple Choice 4: Embracing Change Multiple Choice 5: Managing Time and Priorities Multiple Choice 6: Solving Problems Multiple Choice 7: Working with Data 331 Multiple Choice 8: Working in a Diverse Environment Multiple Choice 9: Acquiring Knowledge Question 4b: Provide specific examples of the work that you have done in general chemistry lab that has supported your growth in this area. Question 5a: Select a second professional skill from the nine listed below that you are developing in general chemistry lab. (Multiple Choice – forced responses) Multiple Choice 1: Communicating Effectively Multiple Choice 2: Contributing to a Team Multiple Choice 3: Developing Professional/Technical Skills Multiple Choice 4: Embracing Change Multiple Choice 5: Managing Time and Priorities Multiple Choice 6: Solving Problems Multiple Choice 7: Working with Data Multiple Choice 8: Working in a Diverse Environment Multiple Choice 9: Acquiring Knowledge Question 5b: Provide specific examples of the work that you have done in general chemistry lab that has supported your growth in this area. 332 A.VI.9. Sp20 Survey Opening Text. Employers who hire MSU graduates for science, engineering, and health-related careers have identified the most important skills they are seeking in new hires. You will be asked about eleven of these skills in this survey. We would like your perspective on which of these skills you are building in your general chemistry lab course (CEM 161 or CEM 162). We would also like your thoughts on which skills will be helpful in your planned career. Your individual responses will not be viewed by your TA or your course coordinator. We encourage you to take this survey on a laptop or desktop computer, which will make it easier to respond to questions, particularly open-ended questions requiring an input of text. We request your permission to examine these data for our research. Your participation is completely voluntary, anonymous, and confidential. You will receive credit for completing the survey regardless of whether you agree to participate in this research or not. Your responses will have no bearing on your course grade or any other course at Michigan State University. You must be 18 or older to participate in this research project. If you do not wish to participate or are younger than 18 years old, please email [redacted email] and include your name and course number. Question 1: What is your current career goal? (Open Answer- forced entry) Question 2: Which general chemistry lab course are you currently enrolled in? (Multiple Choice- forced entry) Multiple Choice 1: CEM 161 (General Chemistry Laboratory I) Multiple Choice 2: CEM 162 (General Chemistry Laboratory II) Question 3: Which of these skills are you developing in your general chemistry lab course? Identify skills that you will need for your future career. Check the boxes that apply. (Multi- response) Skills to select from for either “Gen Chem Lab” or “Future Career” and the descriptions provided: 333 a) Communicating Effectively: conveying information clearly, making persuasive arguments, listening carefully. b) Contributing to a Team: doing a fair share of work, valuing contributions of others, identifying and using strengths of team members to complete project, knowing when to lead and when to follow. c) Developing Professional/Technical Skills: mastering new tools and techniques to achieve goals, selecting appropriate methods or tools. d) Embracing Change: accepting change and seeing it as an opportunity, being open to new approaches. e) Acquiring Knowledge: absorbing new information and concepts, connecting related concepts, identifying appropriate outside resources. f) Managing Time and Priorities: dividing a large project into smaller manageable tasks and developing a plan, distinguishing essential from trivial, prioritizing urgent tasks, staying on task. g) Solving Problems: defining a problem, developing and executing a plan, drawing on multiple perspectives, using reliable outside sources to guide solutions. h) Working with Data: sorting through data, selecting appropriate methods for analysis, making sense of conflicting information, recognizing assumptions in analysis. i) Working in a Diverse Environment: valuing different perspectives and experiences, using differences to achieve better results, identifying and working to overcome biases. j) Curiosity: interest in how the world around us works, eagerness to learn. k) Self-motivation: completing task on own initiative without prompting from others. l) Check the box for “Future Career.” This is to confirm that you are reading the questions before responding. Question 4a: Select one of the eleven professional skills listed below that you are developing in general chemistry lab. (Multiple Choice – forced responses) Multiple Choice 1: Communicating Effectively Multiple Choice 2: Contributing to a Team Multiple Choice 3: Developing Professional/Technical Skills Multiple Choice 4: Embracing Change 334 Multiple Choice 5: Managing Time and Priorities Multiple Choice 6: Solving Problems Multiple Choice 7: Working with Data Multiple Choice 8: Working in a Diverse Environment Multiple Choice 9: Acquiring Knowledge Multiple Choice 10: Curiosity Multiple Choice 11: Self-motivation Question 4b: Provide specific examples of the work that you have done in general chemistry lab that has supported your growth in this area. Question 4c: How did going from an in-person course to an online course impact your ability to build the skill selected in your general chemistry laboratory course? Question 5a: Select a second professional skill from the eleven listed below that you are developing in general chemistry lab. Multiple Choice 1: Communicating Effectively Multiple Choice 2: Contributing to a Team Multiple Choice 3: Developing Professional/Technical Skill Multiple Choice 4: Embracing Change Multiple Choice 5: Managing Time and Priorities Multiple Choice 6: Solving Problems Multiple Choice 7: Working with Data Multiple Choice 8: Working in a Diverse Environment Multiple Choice 9: Acquiring Knowledge Multiple Choice 10: Curiosity Multiple Choice 11: Self-motivation Question 5b: Provide specific examples of the work that you have done in general chemistry lab that has supported your growth in this area. Question 5c: How did going from an in-person course to an online course impact your ability to build the skill selected in your general chemistry laboratory course? 335 A.VI.10. Su20 Survey Opening Text. Thank you for participating in this survey! We would like your perspectives on skills you may be building in your general chemistry lab course (CEM 161 or CEM 162). We would also like your thoughts on which skills will be helpful in your planned career. Your individual responses will not be viewed by your TA or your course coordinator. We encourage you to take this survey on a laptop or desktop computer, which will make it easier to respond to questions, particularly open-ended questions requiring an input of text. We request your permission to examine these data for our research. Your participation is completely voluntary, anonymous, and confidential. You will receive credit for completing the survey regardless of whether you agree to participate in this research or not. Your responses will have no bearing on your course grade or any other course at Michigan State University. You must be 18 or older to participate in this research project. If you do not wish to participate or are younger than 18 years old, please email [redacted email] and include your name and course number. Question 1: What is your current future career goal (after graduation)? (Open Answer) Question 2a: Have you had any opportunities to intern/shadow in the field of your future career goal? (Multiple Choice) Multiple Choice 1: Yes Multiple Choice 2: No Question 2b: Please elaborate on your intern/shadowing experience. (If answered "No", respond N/A.) (Open Answer) Question 3: What attributes or skills do you believe a future employer would expect from recent college graduates applying for jobs in your planned career field? Select all that apply. (Multi- Response) Multi-response Choice 1: Communicating Effectively: conveying information clearly, making persuasive arguments, listening carefully. External communication, outside of teamwork. 336 Multi-response Choice 2: Contributing to a Team: doing a fair share of work, valuing contributions of others, identifying and using strengths of team members to complete project, knowing when to lead and when to follow. Multi-response Choice 3: Developing Professional/Technical Skills: mastering new tools and techniques to achieve goal, selecting appropriate methods or tools. Multi-response Choice 4: Embracing Change: accepting change and seeing it as an opportunity, being open to new approaches. Multi-response Choice 5: Acquiring Knowledge: absorbing new information and concepts, connecting related concepts, identifying appropriate outside resources. Multi-response Choice 6: Managing Time and Priorities: dividing a large project into smaller manageable tasks and developing a plan, distinguishing essential from trivial, prioritizing important and urgent tasks, staying on task. Multi-response Choice 7: Solving Problems: defining a problem, developing and executing a plan, drawing on multiple perspectives, using reliable outside sources to guide solutions. Multi-response Choice 8: Working with Data: sorting through data, selecting appropriate methods for analysis, making sense of conflicting information, recognizing assumptions in analysis. Multi-response Choice 9: Working in a Diverse Environment: valuing different perspectives and experiences, using difference to achieve better results, identifying and working to overcome biases. Multi-response Choice 10: Curiosity: interest in how the world around us works, eagerness to learn. Multi-response Choice 11: Self-motivation: completing task on own initiative without prompting from others. Multi-response Choice 12: Check this box. This is to confirm that you are reading the questions before responding. Question 4: What general chemistry laboratory course are you currently enrolled in? (Multiple Choice) Multiple Choice 1: CEM 161 Multiple Choice 2: CEM 162 337 Question 5: Which of the attributes or skills listed below do you believe that you are building in your general chemistry laboratory course? Select all that apply. (Multi-Response) Multi-response Choice 1: Communicating Effectively: conveying information clearly, making persuasive arguments, listening carefully. External communication, outside of teamwork. Multi-response Choice 2: Contributing to a Team: doing a fair share of work, valuing contributions of others, identifying and using strengths of team members to complete project, knowing when to lead and when to follow. Multi-response Choice 3: Developing Professional/Technical Skills: mastering new tools and techniques to achieve goal, selecting appropriate methods or tools. Multi-response Choice 4: Embracing Change: accepting change and seeing it as an opportunity, being open to new approaches. Multi-response Choice 5: Acquiring Knowledge: absorbing new information and concepts, connecting related concepts, identifying appropriate outside resources. Multi-response Choice 6: Managing Time and Priorities: dividing a large project into smaller manageable tasks and developing a plan, distinguishing essential from trivial, prioritizing important and urgent tasks, staying on task. Multi-response Choice 7: Solving Problems: defining a problem, developing and executing a plan, drawing on multiple perspectives, using reliable outside sources to guide solutions. Multi-response Choice 8: Working with Data: sorting through data, selecting appropriate methods for analysis, making sense of conflicting information, recognizing assumptions in analysis. Multi-response Choice 9: Working in a Diverse Environment: valuing different perspectives and experiences, using difference to achieve better results, identifying and working to overcome biases. Multi-response Choice 10: Curiosity: interest in how the world around us works, eagerness to learn. Multi-response Choice 11: Self-motivation: completing task on own initiative without prompting from others. 338 Multi-response Choice 12: Check this box. This is to confirm that you are reading the questions before responding. Question 6a: Select a skill or attribute that you believe you are developing in the general chemistry lab. (Multiple Choice) Multiple Choice 1: Communicating Effectively Multiple Choice 2: Contributing to a Team Multiple Choice 3: Developing Professional/Technical Skills Multiple Choice 4: Embracing Change Multiple Choice 5: Managing Time and Priorities Multiple Choice 6: Solving Problems Multiple Choice 7: Working with Data Multiple Choice 8: Working in a Diverse Environment Multiple Choice 9: Acquiring Knowledge Multiple Choice 10: Curiosity Multiple Choice 11: Self-motivation Question 6b: Provide specific examples of the work that you have done in your general chemistry laboratory course that has supported your growth in this area. Question 6c: What impact do you think taking the general chemistry lab course online had on your ability to develop the skill or attribute that you selected? (Open Answer) Question 7: What skills did you want to gain from the general chemistry laboratory course that you have not had a chance to build upon during the current semester? Please elaborate. (You can mention skills seen previously or those not included on the list.)(Open Answer) 339 A.VI.11. Fa20 Survey Opening Text. Thank you for participating in this survey! We would like your perspectives on skills you may be building in your general chemistry laboratory course (GCL1/GCL2). We would also like your thoughts on which skills will be helpful in your planned career. Your individual responses will not be viewed by your TA or your course coordinator. We encourage you to take this survey on a laptop or desktop computer, which will make it easier to respond to questions, particularly open-ended questions requiring an input of text. We request your permission to examine these data for our research. Your responses will also be used to guide enhancements to the general chemistry laboratory curriculum to improve the experience for future students. Your participation is completely voluntary, anonymous, and confidential. You will receive credit for completing the survey regardless of whether you agree to participate in this research or not. Your responses will have no bearing on the grade in this course beyond the credit received for completing the survey or in any other course at Michigan State University. You must be 18 or older to participate in this research project. If you do not wish to participate or are younger than 18 years old, please email [redacted email] and include your name and course number. Question 1: What is your career goal following graduation? (Open Answer) Question 2a: Have you had an internship or job shadowing experience? (Multiple Choice) Multiple Choice 1: Yes Multiple Choice 2: No Question 2b: Please elaborate on your shadowing experience. (Open Answer) Question 3: What skills will you need to be successful in your planned career? (Please elaborate. You can mention as many skills as you would like.) (Open Answer) Question 4a: Rank these skills from most important to least important, with 1 being the most important (can have duplicate numbers if skills are equally important). (Open Answer) EDC 1 to Rank: Teamwork EDC 2 to Rank: Critical Thinking and Problem Solving EDC 3 to Rank: Managing Time and Priorities EDC 4 to Rank: Communication Skills 340 EDC 5 to Rank: Self Motivation EDC 6 to Rank: Technical Skills Question 4b: This space is provided if you would like to provide additional comments on your ranking decisions above. (Open Answer) Visual Aid for Question 5: Question 5: If an employer showed you the list of these skills (shown above) and asked you to provide examples, how would you connect your experiences in your general chemistry laboratory course to this list? (You are welcome to talk about as many as you would like.) (Open Answer) Question 6: Do you believe these skills are valuable for success in your future career? (Multiple Choice) Multiple Choice 1: Yes Multiple Choice 2: No Question 7: Do you believe that these skills are generally valuable in a variety of different careers? (Multiple Choice) Multiple Choice 1: Yes Multiple Choice 2: No 341 Question 8: What skills did you want to gain from the general chemistry laboratory course that you have not had a chance to build upon during the current semester? Please elaborate. (You can mention skills seen previously or those not included on the list.) (Open Answer) Question 9: Has online learning affected your experience in your general chemistry lab course? (Please elaborate on your experience with your online general chemistry lab.) (Open Answer) 342 A.VI.12.1. Sp21 Scenario Prompt The following prompt was provided to students throughout the course of the semester at the beginning of each project within their scenario documents. Transferable Skills Development: Several skills that are highly valued by employers, graduate schools, and professional schools when evaluating college graduates are listed below. Such skills are often referred to as transferable skills because they are necessary for success in the 21st century workforce in STEM, health-related fields, and beyond. This laboratory project provides opportunities for you to develop these skills. 1. Communication Skills: communication in written, oral, and visual formats; presenting to different audiences; conveying information clearly and concisely to others; listening well. 2. Teamwork: working well with others; recognizing one’s own strengths and weaknesses, as well of those of others; working in diverse and multidisciplinary teams to solve a problem. 3. Problem Solving and Critical Thinking: defining a problem; proposing and testing hypotheses and solutions; analyzing and interpreting data; drawing conclusions from data. 4. Managing Time and Priorities: prioritizing tasks; delegating time efficiently; completing work in a timely fashion; being well organized. 343 A.VI.12.2. Sp21 Reflective Assignment Opening Text. Throughout this semester you were introduced to transferable skills that are necessary for advancement in the 21st century workforce, as well as success in graduate and professional programs. The following questions ask you to reflect on your experiences in developing skills throughout the course. Question 1: How did the introduction to transferable skills throughout the course help you develop awareness of the skills that are needed in today’s workforce? (Open Answer) Question 2a: Do you think that the opportunities to develop transferable skills in your general chemistry laboratory course are valuable in helping to prepare for your future career? Circle, underline, or bold one of the following choices: (Multiple Choice) Multiple Choice 1: Yes Multiple Choice 2: No, I did not develop any transferable skills. Multiple Choice 3: No, these skills will not contribute to success in my future career. Question 2b: Elaborate on your choice in part 2a. (Open Answer) Question 3: Rank the following transferable skills in order from most important (value of 1) to least important for your planned career, values can be repeated if skills are equally important (For example: Teamwork and Communication Skills could both be ranked as 1). Space is provided for you to add any skills that you think are important for success in the workforce and/or graduate and professional schools, but are missing from the list. (Open Answer) Rank ______ ______ ______ ______ Skill Teamwork Communication Skills (Other) ___________________ (Other) ___________________ Skill Problem Solving and Critical Thinking Rank ______ ______ Managing Time and Priorities (Other) ___________________ ______ (Other) ___________________ ______ Question 4: Additional comments or suggestions about transferable skill development in your gen chem lab course. (Open Answer) 344 A.VI.13.1. Fa21 Initial Reflective Assignment (Fa21 IRA) Question 1: What career or additional schooling do you plan to pursue following graduation from MSU? (Open Answer) Question 2: What skills are needed to be successful in your planned career or additional schooling?(Open Answer) Question 3: In which of the skills listed above are you already proficient? (Open Answer) Question 4: Identify any skills from your list above that you would like to further develop during your college career. (Open Answer) Closing Text. We request your consent to use your responses in our research to enhance the experience in this course for future students. Participation in this study is completely voluntary, anonymous, and will not affect your grade in this or any other courses. Your general chemistry laboratory TA will not be notified of your answers or participation. In order to participate in this study you must be 18 years or older. If you agree to participate in this research, there is nothing further for you to do. If you are under 18 years old or do not wish to participate in the study, you can still complete the survey for extra credit, but please email [redacted email] with the subject line Opt Out of GCL1/2 Study from your MSU email to be excluded from the study. If you have general questions regarding the extra credit activity, please contact [author of study] at [redacted email] or [advisor of study] at [redacted email]. 345 A.VI.13.2. Fa21 Final Reflective Assignment (Fa21 FRA) Question 1: What career or additional schooling do you plan to pursue following graduation from MSU? (Open Answer) Question 2: What skills are needed to be successful in your planned career or additional schooling? (Open Answer) Question 3: In what ways did your general chemistry laboratory course help you develop the skills listed above? Please provide specific examples of how activities, assignments, the classroom environment, course structure, or other experiences helped you build these skills. (Open Answer) Question 4: What additional things could be done in the general chemistry laboratory course(s) that could help you better develop the skills that you discussed above? (Open Answer) Question 5: Have you learned about skills sought by employers at MSU or in high school? (Multiple Choice) Multiple Choice 1: Yes, at MSU (excluding the general chemistry laboratory courses) Multiple Choice 2: Yes, in high school Multiple Choice3: Yes, both in high school and at MSU (excluding the general chemistry laboratory courses) Multiple Choice 4: No Multiple Choice 5: I’m not sure Question 6: Would you like to have more opportunities in your college courses to learn about the skills needed for your planned career goal and/or additional schooling? (Multiple Choice) Multiple Choice 1: Yes Multiple Choice 2: No Closing Text. We request your consent to use your responses for our research to enhance the experience in this course for future students. Participation in this study is completely voluntary, anonymous, and will not affect your grade in this or any other courses. Your general chemistry laboratory TA will not be notified of your answers or participation. In order to participate in this study you must be 18 years or older. If you agree to participate in this research, there is nothing further for you to do. If you are under 18 years old or do not wish to participate in the study, you 346 can still complete the survey for extra credit, but please email [redacted email] with the subject line Opt Out of CEM 161/162 Study from your MSU email to be excluded from the study. If you have general questions regarding the extra credit activity, please contact [author of study] at [redacted email] or [advisor of study] at [redacted email]. 347 A.VI.14. Related Survey Questions Used as Groupings for Analysis Student Reported Career Goal a. Included in all surveys and reflective assignments, except for Sp21 Final Reflective Assignment. Multiple Choice and Open Answers Related to Prior Experience a. Su20 Question 2a and 2b. • Question 2a: Have you had any opportunities to intern/shadow in the field of your future career goal? (Multiple Choice) • Question 2b: Please elaborate on your intern/shadowing experience. (If answered "No", respond N/A.) (Open Answer) b. Fa20 Question 2a and 2b. • Question 2a: Have you had an internship or job shadowing experience? (Multiple Choice) • Question 2b: Please elaborate on your shadowing experience. (Open Answer) Open Answer Questions Relating to Skills Needed for Career Goal a. Fa20 Question 3. • Question 3: What skills will you need to be successful in your planned career? (Please elaborate. You can mention as many skills as you would like.) (Open Answer) b. Fa21 IRA Question 2. • Question 2: What skills are needed to be successful in your planned career or additional schooling?(Open Answer) c. Fa21 FRA Question 2. • Question 2: What skills are needed to be successful in your planned career or additional schooling? (Open Answer) Ranking of EDCs a. Fa20 Question 4a and 4b. • Question 4a: Rank these skills from most important to least important, with 1 being the most important (can have duplicate numbers if skills are equally important). (Open Answer) 348 • Question 4b: This space is provided if you would like to provide additional comments on your ranking decisions above. (Open Answer) b. Sp21 FRA Question 3. • Question 3: Rank the following transferable skills in order from most important (value of 1) to least important for your planned career, values can be repeated if skills are equally important (For example: Teamwork and Communication Skills could both be ranked as 1). Space is provided for you to add any skills that you think are important for success in the workforce and/or graduate and professional schools, but are missing from the list. (Open Answer) Multi-Response EDC Development in Course and Skills Needed for Career Goal a. Fa19 Question 3. • Question 3: Which of these skills are you developing in your general chemistry lab course? Identify skills that you will need for your future career. Check the boxes that apply. (Multi-response) b. Sp20 Question 3. • Question 3: Which of these skills are you developing in your general chemistry lab course? Identify skills that you will need for your future career. Check the boxes that apply. (Multi-response) c. Su20 Question 3 and 5. • Question 3: What attributes or skills do you believe a future employer would expect from recent college graduates applying for jobs in your planned career field? Select all that apply. (Multi-Response) • Question 5: Which of the attributes or skills listed below do you believe that you are building in your general chemistry laboratory course? Select all that apply. (Multi-Response) Questions Related to EDC Development in Course a. Fa19 Survey Question 4b and 5b. • Question 4b and 5b: Provide specific examples of the work that you have done in general chemistry lab that has supported your growth in this area. • The above question is in relation to a skill selected from Question 4a and 5a. 349 • Question 4a: Select one of the nine professional skills listed below that you are developing in general chemistry lab. (Multiple Choice – forced responses) • Question 5a: Select a second professional skill from the nine listed below that you are developing in general chemistry lab. (Multiple Choice – forced responses) b. Sp20 Survey Question 4b and 5b. • Question 4b and 5b: Provide specific examples of the work that you have done in general chemistry lab that has supported your growth in this area. • The above question is in relation to a skill selected from Question 4a and 5a. • Question 4a: Select one of the eleven professional skills listed below that you are developing in general chemistry lab. (Multiple Choice – forced responses) • Question 5a: Select a second professional skill from the eleven listed below that you are developing in general chemistry lab. c. Su20 Survey Question 6b. • Question 6b: Provide specific examples of the work that you have done in your general chemistry laboratory course that has supported your growth in this area. • The above question is in relation to a skill selected from Question 6a. • Question 6a: Select a skill or attribute that you believe you are developing in the general chemistry lab. (Multiple Choice) d. Fa20 Survey Question 5. • Question 5: If an employer showed you the list of these skills (shown above) and asked you to provide examples, how would you connect your experiences in your general chemistry laboratory course to this list? (You are welcome to talk about as many as you would like.) (Open Answer) e. Sp21 Reflective Assignment Question 2a, 2b, and 4. • Question 2b: Elaborate on your choice in part 2a. • The above question is in relation to a skill selected from Question 6a. • Question 2a: Do you think that the opportunities to develop transferable skills in your general chemistry laboratory course are valuable in helping to prepare 350 for your future career? Circle, underline, or bold one of the following choices: (Multiple Choice) • Question 4: Additional comments or suggestions about transferable skill development in your gen chem lab course. (Open Answer) f. Fa21 Initial Reflective Assignment N/A. • N/A g. Fa21 Final Reflective Assignment Question 3 and 4. • Question 3: In what ways did your general chemistry laboratory course help you develop the skills listed above? Please provide specific examples of how activities, assignments, the classroom environment, course structure, or other experiences helped you build these skills. (Open Answer) • Question 4: What additional things could be done in the general chemistry laboratory course(s) that could help you better develop the skills that you discussed above? (Open Answer) Open Answer Questions Related to Online Impact on EDC Development and General a. Sp20 Question 4c and 5c. Laboratory Experience • Question 4c and 5c: How did going from an in-person course to an online course impact your ability to build the skill selected in your general chemistry laboratory course? b. Su20 Question 6c. • Question 6c: What impact do you think taking the general chemistry lab course online had on your ability to develop the skill or attribute that you selected? (Open Answer) c. Fa20 Question 9. • Question 9: Has online learning affected your experience in your general chemistry lab course? (Please elaborate on your experience with your online general chemistry lab.) (Open Answer) Open Answer Questions Related to Skills Students Would Like to Develop in Course or a. Su20 Question 7. Wanted to Develop in College 351 • Question 7: What skills did you want to gain from the general chemistry laboratory course that you have not had a chance to build upon during the current semester? Please elaborate. (You can mention skills seen previously or those not included on the list.)(Open Answer) b. Fa20 Question 8. • Question 8: What skills did you want to gain from the general chemistry laboratory course that you have not had a chance to build upon during the current semester? Please elaborate. (You can mention skills seen previously or those not included on the list.) (Open Answer) c. Fa21 IRA Question 4. • Question 4: Identify any skills from your list above that you would like to further develop during your college career. (Open Answer) Miscellaneous Questions that Differed Throughout the Surveys c. Fa20 Question 6 and 7. • Question 6: Do you believe these skills are valuable for success in your future career? (Multiple Choice) • Question 7: Do you believe that these skills are generally valuable in a variety of different careers? (Multiple Choice) d. Sp21 IRA Question 4. • Question 4: How do you think that these skills could contribute to success in your future career? (Open Answer) e. Fa21 IRA Question 3. • Question 3: In which of the skills listed above are you already proficient? (Open Answer) f. Fa21 FRA Question 5 and 6. • Question 5: Have you learned about skills sought by employers at MSU or in high school? (Multiple Choice) • Question 6: Would you like to have more opportunities in your college courses to learn about the skills needed for your planned career goal and/or additional schooling? (Multiple Choice) 352 A.VI.15. Multi-response participants who passed/failed the validation prompt. Table 6.19. Multi-response participants who either passed or failed validation prompt, n, (%). Semester Fa19 Sp20 Su20 Total (n) Passed 971 (78) 884 (90) 94 (93) 1949 Passed/Failed Validation Failed 267 (22) 102 (10) 7 (7) 376 Total (n) 1238 986 101 2325 353 A.VI.16. Duplicate participants removed from analysis from Fa19, Sp20, and Su20 open answer survey responses regarding development within the course. Table 6.20. Overall participants who selected duplicate responses in their multiple-choice skill selection, n (%). Duplicate Skill Picked Yes No Total (n) Fa19 9 (1) 1229 (99) 1238 Sp20 1 (0.1) 985 (99.9) 986 Su20 0 (0) 101 (100) 101 Total (n) 10 2315 2325 Semester Table 6.21. Participants who selected duplicate responses in their multiple-choice skill selection by skill choice (n). Skill Communicating Effectively Teamwork Skills Solving Problems Priority & Time Management Professional/Technical Skills Acquiring Knowledge Embracing Change Working with Data Working in a Diverse Environment Curiosity Self-motivation Total (n) Fa19 3 2 1 — — — — 3 — — — 9 Semester Su20 — — — — — — — — — — — 0 Sp20 — — — — — — — — — — 1 1 Total 3 2 1 — — — — 3 — — 1 10 354 Due to the complexity that can emerge in coding thousands of open answer responses, the A.VI.17. Coding of Survey Participant Career Goals generation of a codebook containing guidelines for categorizing participant career goals was synthesized and is outlined below. For participants who mentioned additional schooling (i.e. medical school), it was assumed that the their future career prospects were closely linked to the additional schooling they planned to pursue and were coded as such (e.g., Health & Medical Profession). Responses in which participants had mentioned two or more possible career goals were coded for in one of two ways; 1) participant responses with career goals that fell under the same researcher-defined theme were coded accordingly or 2) participant responses that mentioned career goals that spanned across multiple themes were coded as Other Career Goals. There were some exceptions to this rule, with a small number of participants who had mentioned wanting to become a physician and also having the hopes of opening a side business or being an entrepreneur being coded as Health & Medical Professions. Additionally, forward slashes were used within some participant responses (e.g., Vet/Dancer) to denote two possible career goals and were coded interpretively based on the participant response. For example, participants who desired to go into the health and medical field often used graduate school and medical school interchangeably (i.e.: Graduate School/Medical School) or used a forward slash to denote the specialty they wanted to go into (e.g., Kinesiology/Occupational Therapist). For those who mentioned career goals in two or more separate categories (e.g., Vet/Dancer), participants were coded as Other Career Goals and interpreted as meaning “and/or” or that a participant was in between those two possible career goals. Context of participant responses being present or absent was an important factor when deciding how to code a participant’s career goal. With those guidelines in mind, the initial coding scheme consisted of the following six emergent themes; • Health & Medical Professions – includes participants pursuing careers in health and medical practitioners/professionals (e.g., medical doctor, occupational/physical therapist, physician assistant, nursing, those attending medical school, psychologist, psychiatrist) and those involved in clinical studies due to these types of studies actively involve working with and providing treatment to patients. Medical research was a topic mentioned by participants and was identified as too broad of a category to be coded as 355 Health & Medical Professions resulting in these participants being coded as All Other Career Goals. Healthcare administration was coded under All Other Career Goals due to the underlying business nature of this occupation. • Engineering & Subspecialties – includes participants who desired careers in engineering and all subspecialties of engineering occupations such as computer science, supply chain management, and packaging. Subspecialties of engineering include biomedical, biotechnical, biosystems, mechanical, electrical, civil, software, computer, automotive, applied, logistics, sales engineer, environmental engineering, etc. • Other Career Goals – includes participants who desired careers in data science and analytics (due to the multidisciplinary nature of the career), careers based in a laboratory or research that could not be classified in any other categories (e.g., biomedical lab science/technician, medical lab diagnostics, pharmacology, toxicology, and biotechnical research), natural sciences (e.g., biological sciences and all associated disciplines such as human biology, physiology, geneticist, neuroscientist, molecular genomics and genetics, epidemiology, pathology, chemistry, physics, and multidisciplinary natural science fields), business related careers (e.g., actuary or entrepreneur, pharmaceutical or medical device sales), continuing education through graduate school without specifying specialty or discipline, security and law-based careers (e.g., forensic science), education or teaching careers, academia, professor, and academic research. This category also includes obtaining a summer job/internship, working for a specified company/industry without alluding to a specific position, agriculture, natural resources, and conservation, army, air force, or navy, communication and journalism (e.g., technical writing), culinary entertainment and personal services (e.g., chef), family and consumer services (e.g., human development and family studies), social sciences (e.g., political science and psychology if not stating becoming a psychologist or any other career), social services (e.g., social work), sports and leisure (e.g., personal trainer), visual and performing arts (e.g., art conservation), mathematics (e.g., statistics), architecture, construction, and landscape (e.g., Landscape architecture), and legal professionals (e.g., Law professional). • Did Not Specify a Career – includes participants who stated that they wanted a job but providing no further description or replying that they having no career goal (e.g., “N/A” or “Get a good job”). Includes students specifying a career goal of obtaining a summer 356 job/internship with no indication of what they would be doing or what field the internship/job is in. Additionally, if participant states they would like to work for a company but did not specify the company or what they would be doing, the response was categorized here. • Unsure or Did Not Know – includes participants stating uncertainty or not knowing what career goal they would like to pursue. If a participant stated interest in a field or wanting to go into a certain field but being uncertain, they were coded for that field of interest. Some participants stated a possibility (e.g., Possibly medical school) or probability (e.g., I am not sure yet, but probably would like to work for a pharmaceutical company in the future) - and were coded for the field of interest (e.g., “To graduate with an environmental engineering degree. I am not sure where I want to work.” - was coded for Engineering & Subspecialties). These participants generally had no inkling or idea of where they wanted to go or do and provided no frame of reference of the potential career they would like to pursue. • Undergraduate Education Goals – includes students stating they would like to complete their degree without mentioning specialty, are in the process of currently changing major/degree, determining what major they would like to pursue, getting through the semester, completing the course, improve skills and prepare for career goal, and/or wanting to receive a good grade/GPA. If a student noted that they wanted to get a degree in their specialty it was assumed that they wanted to pursue a career in the same field as their major and the participant was coded accordingly (e.g., “Graduate with a bachelor's degree in Dietetics” was coded for Health & Medical Professions or “Get into school of Engineering” was coded for Engineering & Subspecialties). From these six categories, those who did not specify a career, were unsure of what future occupation they desired to be in, or those who stated goals surrounding their undergraduate education were combined and categorized under Other Career Goals. This resulted in the formation of three distinct career categories, as also seen within interviews, of Health & Medical Professions, Engineering & Subspecialties, and Other Career Goals. 357 A.VI.18. Fa21 survey participant career goal changes. Table 6.22. Recurring Fa21 IRA and FRA participants who reported a change in career goal (n = 815)(n). Fa21 FRA Health & Medical Professionals Engineering & Associated Subspecialties Fa21 IRA Health & Medical Professionals Engineering & Associated Subspecialties Other Career Goals 398 0 29 2 93 29 Other Career Goals 42 22 200 358 A.VI.19. Prior experience multiple-choice survey results disaggregated and tested by various demographic variables. Table 6.23. Overall Su20 & Fa20 multiple-choice results of whether a student has had prior experiences in the field of their anticipated career, n, (%). Prior Experience Yes, has had prior experiences No, has not had prior experiences n (%) 374 (31) 842 (69) 1216 Total (n) Table 6.24. Su20 & Fa20 multiple-choice results of whether a student has had prior experiences in the field of their anticipated career by semester, n, (%). Response Yes, has had prior experiences No, has not had prior experiences Su20 39 (39) 62 (61) 101 Total (n) Semester, n (%) Fa20 335 (30) 780 (70) 1115 Total (n) 374 842 1216 Table 6.25. Su20 & Fa20 multiple-choice results of whether a student has had prior experiences in the field of their anticipated career by first-generation status, reported as number and related percent of participants. Response Yes, has had prior experiences No, has not had prior experiences Total (n) First-Generation Student Status, n (%) First Generation 79 (32) 166 (68) 245 Continuing Generation 295 (30) 676 (70) 971 Total (n) 374 842 1216 Table 6.26. Su20 & Fa20 multiple-choice results of whether a student has had prior experiences in the field of their anticipated career by career goal, n, (%). Response Yes, has had prior experiences No, has not had prior experiences Total (n) Health & Medical Professions Career Goal Engineering & Associated Subspecialties 263 (40) 397 (60) 660 35 (18) 161 (82) 196 Other Careers Total (n) 76 (21) 374 284 (79) 360 842 1216 359 Table 6.27. Su20 & Fa20 adjusted residuals of whether a student has had prior experiences in the field of their anticipated career by career goal (Bonferroni adjusted critical value for residuals ± =2.6). Variable Groups Tested Yes, has had prior experiences x Health & Medical Professions No, has not had prior experiences x - Health a& Medical Professions Yes, has had prior experiences x Engineering & Associated Subspecialties No, has not had prior experiences x - Engineering & Associated Subspecialties Yes, has had prior experiences x Other Career Goals No, has not had prior experiences x Other Career Goals Adjusted Residual 7.5 -7.5 -4.3 4.3 -4.7 4.7 Table 6.28. Su20 & Fa20 multiple-choice results of whether a student has had prior experiences in the field of their anticipated career by class standing, n, (%). Response Yes, has had prior experiences No, has not had prior experiences Total (n) Lower Classman 234 (27) 647 (73) 881 Class Standing Upper Classman 140 (42) 195 (58) 335 Total (n) 374 842 1216 360 A.VI.20. Skills perceived as Necessary for Future Career The list below is expanded from the list included in Chapter IV A.VI.11 to incorporate skills present in survey responses. Coding Scheme - Skills Needed for Career 1. Interpersonal Skills: • Teamwork & Collaboration: working with others, working in a team-based setting, collaboration, participation, & cooperation • Communication Skills: listening skills; verbal communication (public speaking & presenting, relaying information); written communication (includes ability to write reports and communicate via written format, & scientific written communication through reports & general scientific written communication); general scientific communication • Leadership Skills • Networking Skills • Social & People Skills: this embodies the ability to interact and socialize with others. This theme includes: building friendships & relationships (i.e. personal relations); being charismatic & persuasive; compassion & empathy (ability to understand others, be caring, sympathetic, & comforting, altruism); being friendly, kind, respectful, & helpful to others or being helpful; being personable & approachable; reliable & dependable; being trustworthy & selfless; being inclusive; providing patient care (ability to talk well to & interact with patients, providing hospitality & good bedside manner, & building relationships with patients); knowing how to read cues; being able to speak with others & have conversation skills; working with, dealing, and being willing to work with people & patients (in a non-collaborative manner), accommodation , customer service skills, togetherness, available • Miscellaneous Interpersonal Skills: o Leadership Skills: overseeing decisions, guiding people in a group, making sure group is working cohesively and on track, taking initiative, delegating tasks. o Networking Skills: utilizing outside experts to answer a question and receive guidance. 361 o Multi-cultural Competency: cultural awareness; global/intercultural fluency o General Interpersonal Skills: includes mentioning the need for interpersonal skills with no specific subskill listed. 2. Intrapersonal Skills: • Work Ethic: hard working, dedication & commitment, passion, drive, & ambition, determination (having a strong will), self-motivation, persistence & perseverance (grit, relentlessness, ability to work under pressure & handle everything being thrown their way, resilience, able to work for long hours, not giving up, diligence, able to fail and continue going), performing with integrity, disciplined, focus & concentration (attention span), taking initiative & being fast to act, ability to work hard at school, being responsible & accountable, consistency, sacrifice, effort • Personality & Character Traits: Confident/outgoing (not being afraid to ask for help), positive attitude & optimistic, calm (remain calm in high stress situations), patient, serious, tough, enthusiastic, risk taker, brave, trying ones best, being a self-advocate, being independent/independence (ability to work independently & isolate), being open- minded, being humble, honest, & genuine, rational/level headed, flexible & adaptable (knowing how to deal with & face challenges, being able to handle critical situations, improvisation, being versatile, managing challenges & adapting accordingly, embracing change), being mature, having composure, keeping intentions in the right place, being a fun person, being able to and always asking for help/not being afraid to ask for help, being careful • Miscellaneous Intrapersonal Skills: self awareness (understanding ones goals & abilities, being in touch with ones emotions), mental wellness & wellbeing (emotional intelligence, emotional stability, stress management, pressure management & having self worth), having strong emotions, ethics, enthusiasm/commitment (from list) 3. Occupation-specific Knowledge & Skills: • Technical Skills: o Professional/Technical Skills (from List) o Lab Skills & Techniques: general experimentation, knowledge of how to conduct a lab & carry out an experiment (learning lab procedures, carrying out procedures w/instructions, experimenting in-person, experimental design, experimental 362 principles, collecting data & conducting tests, situational awareness of chemical reactions), experimental techniques; lab safety & etiquette (carrying out lab safety measures & techniques, hood hygiene, knowing lab etiquette); experience with laboratory tools, instruments, & equipment (i.e. microscopy, using & being introduced to lab equipment, using CEM lab instruments) ; technical chemistry skills; experience working in a lab environment, & knowing/learning how to work in a lab environment through the course itself (this is not the same as participant saying they need lab experience which may be clinical or research lab experience as specified in exposure to field and hands on experiences); technical chemistry skills o Technological/Computer Skills: general technological/computer skills; computer science knowledge & skills; Microsoft office skills (Excel, PowerPoint, Word); engineering software programs (CAD, packaging, & general engineering software); knowledge of computer software & programs; knowledge of computer parts; computer design; computer modeling (3D models and simulations); coding knowledge & skills (coding software & programs and programming skills); knowledge of music software ▪ Hands On Skills: motor skills & hand-eye coordination (includes general motor skills and hand-eye coordination, having steady hands (steadiness), manual dexterity, and general hand skills); use of tools, knowledge of instruments used in the field, mechanical skills, using physics and engineering to build machinery ▪ Precision & Accuracy o Exposure to Field & Hands On Experiences: this includes shadowing, internship, volunteer opportunities; having work experience or job in the field; research & lab experiences (includes clinical lab experience); gaining experience in the field in general and training in their profession (i.e. experience working with animals, in automotive or packaging industry, clinical experience, laboratory training, and ability to work in a certain work environment like a hospital) o Professionalism & professional skills (professional skills are not specified) o Field Specific Knowledge: 363 ▪ Medical knowledge & skills (healthcare knowledge & skills; ability to treat patients and patient skills; nursing skills; ability to & knowledge of performing surgeries & medical procedures; knowing how to apply sutures; clinical knowledge, skills, and application; medical literacy; knowledge of administering & prescribing medicine, drugs & drug side effects (knowing how drugs work in and react within the body); knowing how to take vitals; dentistry knowledge (dental courses); knowledge of hygiene and working in a safe & hygienic manner for patients; application of medical knowledge; reading and generation formulas for medicine (health and medical career goal); having a background in medical specialties, knowledge of pharmaceutical field, pharmacology knowledge ▪ Agricultural knowledge & skills: crop knowledge (planting and maintaining crops); animal husbandry; knowledge of herbicides and pesticides, knowledge of plants & how they grow (plant identification), knowledge of plant life ▪ Animal skills & knowledge: handling & working with animals (& care techniques); knowledge of animal habitats & needs; animal behavior ▪ Business knowledge & skills: entrepreneurship; financial literacy; accounting skills; business management; sales skills & the ability to sell; economics & marketing (i.e. microeconomics & marketing skills); consulting skills; logistics (logistical knowledge) ▪ Research skills & abilities: ability to look up documentation, knowledge of how to gather information, general research & development skills; application of science to research; completing steps to identify pieces of matter (research-based career goal), developing, designing, and planning experimental procedures ▪ Psychology Knowledge: psychology courses; psychoanalysis; general knowledge of psychology ▪ Educating & Teaching Others: teaching skills; classroom management ▪ Music knowledge & skills ▪ Knowledge of Conservation 364 ▪ Following Protocol, Regulations, & Instructions ▪ Knowledge of Ethics (no context) ▪ Knowledge of Strength & Conditioning Field ▪ Science, Engineering, and Math Knowledge & Skills: • Math: general math knowledge & skills; algorithms; calculations; computation skills ; logic-based mathematics; math skills for drug dosing; numeracy skills; statistical literacy; statistics; calculus. • Engineering: general engineering knowledge & concepts; ability to understand schematics; design skills (engineering design); electrical engineering knowledge (electrical knowledge); engineering mindset; food engineering; knowledge of automobiles (general automotive knowledge, automotive & engine development knowledge & skills); knowledge of robots/robotics; material knowledge & skills (understanding the properties of materials, “understanding the different compounds & molecules that may be used in making of packaging materials”); redesigning & optimizing systems; background in & understanding of mechanical engineering. • Science: o General science knowledge & skills; understanding of scientific concepts & principals; scientific literacy; scientific method. o Life Sciences: health related sciences (human sciences, applied life sciences (kinesiology & exercise knowledge (knowing how the body moves), nutritional sciences (nutrition), animal science) gaining knowledge of science in relation to medicine); biology (ecology, ecosystems, microbes, microbiology, cell & molecular biology, anatomy & physiology (neuroanatomy, anatomy of animals) animal biology, zoological studies, genetics & genomics, human biology (knowledge of the human body 365 & how the body works, ability to build simple & complex mechanisms relating to the human body, knowledge of human body & how limbs work); biochemistry (both life & physical science): includes needing biochemistry in relation to medicine, knowledge of biochemical mechanisms; biophysics: biomechanics & how body reacts to stimuli o Physical sciences: earth sciences, physics (thermodynamics, heat & mass transfer, dynamics, fluid mechanics), chemistry (organic chemistry) 4. Problem Solving & Critical Thinking: • Creativity & Innovation: thinking outside of the box, alternative thinking, creative thinking • Reasoning Skills: includes the ability to use information to draw conclusions, using evidence to support claims, inductive reasoning skills • Systems Thinking • Analytical Skills: logical thinking (logic), analysis (data analysis), interpretation (data interpretation), analytics (in regards to a career in Health and Medicine) • Troubleshooting – working through errors during experimentation • Hands on problem solving • Being a deep thinker • Intellectual • Being resourceful • Complex thinking 5. Time Management & Organization: • Scheduling & Preparation: planning time and planning ahead; being prepared; schedule management; time dedicated to classes/studying/research, planning out ideas and making plans for career goal, generating & creating plans (general), planning projects & presentations, career management • Efficiently Using Time: working in a timely manner, timeliness, being productive • Multi-tasking 366 • Meeting deadlines: for both work and school • Managing responsibilities & priorities • Avoiding & overcoming procrastination • Organization: including documenting steps (with no specifications) & record keeping skills (to document patient progress) • Punctuality • School/Life & Work/Life Balance • Task Management 6. Education & Learning Skills: • Education & Credentials: this theme includes the following: o Obtaining a bachelors, masters degree, or needing a degree in general o General Education & Academic Background: Having a general education and academic background (includes academic competence, performance, and skills); completing required undergraduate courses & prerequisites, performing well in undergraduate courses & obtaining good grades (i.e: undergraduate education); in general having an undergraduate education and taking undergraduate courses without expanding on the courses needed for career o Applying concepts & knowledge (skills in applying knowledge and application of concepts learned) o Acquiring Knowledge: this includes the following: ▪ General Knowledge: this includes having general and/or basic knowledge & understanding; being knowledgeable; competent; intelligent; smart; having a strong understanding of concepts; terminology; foundational and fundamental knowledge & understanding (i.e. knowledgeable about needed topics, core subjects, being well versed in area of study); being a knowledgeable student in all subjects, knowledge gathered from undergraduate courses (and/or major) & having a good understanding of what was learned from college courses ▪ Continuing education post undergraduate education & continuing schooling in field of choice (i.e. medical school) 367 o Learning & School Skills & Strategies: applying learning techniques; curiosity & willingness to learn, willingness to fail and learn from mistakes; being a quick & face paced learner; reading & comprehension skills, reading scholarly faster and being able to understand and explain journal articles; study skills (memorization & knowledge retention, utilizing office hours, taking notes, being studious ); & test taking skills (standardized testing skills, testing skills to receive good grades, preparing, studying, & passing professional school and standardized exams) 7. Other Skills: • Decision-making Skills: ability to make decisions, fast and quick thinking (thinking on your feet), quick judgement, good judgement, intuition or intuitive thinking, ability to judge cost & benefits of actions, and common sense • Attention to Detail & Observation Skills: attentive • Navigating Across Boundaries • Working in Diverse Environments • Soft Skills (no context) • Excel (no context) • Interest (no context) • Practical Skills (no context) • Physical Skills (no context) • Healthy Body, Physical Stamina, Athletic Skills (Weightlifting) • Orientation (no context) • Understanding (no context) • Understanding why chemistry labs are important • Being able to not be overwhelmed looking at the lab scenario and not letting this get in the way of learning and understanding material for the labs (this was a skill listed as wanting to be gained from the course but not having a chance to and I believe that they feel overwhelmed by the course and do not want to feel this way - possibly putting more emphasis on the course itself for causing this feeling of overwhelming and not something they need/want to work on themselves). 368 • Care (uncertain if this is care for others, for their job, or where the care is directed, no context). • Data Knowledge (was cited by an engineering major so I played with having it under engineering as well but was uncertain, there could be many meanings to this, no context) • Paper Knowledge (was listed as “Logistical & paper knowledge - logistics is under business knowledge but I was unsure where to put paper knowledge, no context) • Coordination (could be with others or hand-eye coordination, no context) • Involvement (could be involved in extracurriculars or involved in group activities, no context) • Synthesis (provided by an engineer focused career goal wanting to go into research, no context) • Willingness (no context) • Product Information (engineering wrote this but I do not know what they meant, no context) • Participating in Extracurricular Activities • Wanting to see other teams to make sure they are doing the labs correctly • Work Split (no context) 369 A.VI.21. Not-prompted Student-perceived Competencies that Emerged in Survey Results Survey data surrounding open answer responses regarding skills perceived as valuable for a participant’s career goal without prompting from Fa20 and Fa21 were compiled and compared against interview participants in Figure 6.24. When compiling the survey data, first responses from the n = 200 participants who took both Fa21 IRA and FRA were combined. Proceeding this Fa21 survey results were combined with Fa20 survey data resulting in a total of n = 400 survey participants represented within the figure being compared to compiled data from the n = 53 interview participants. a t a D n i h t i w s t n a p i c i t r a P f o t n e c r e P ) % ( e p y T n o i t c e l l o C 100 90 80 70 60 50 40 30 20 10 0 Interview Participants Survey Participants Figure 6.24. Comparison of student-perceived competencies as reported by interview and survey participants without prompting (n). 370 A.VI.22. Multi-response Survey Participant Perception of Skills Needed for Future Career Goal To assess the results of multi-response prompts in Fa19, Sp20, and Su20, participants who passed validation prompts remained in analyses (A.VI.15). This method of filtering resulted in the samples disaggregated by semester in Table 6.29. When prompted to select skills perceived as needed for their future career goals, most survey participants regarded all employer-desired competencies as valuable skills (Table 6.30). Because of the prevalence of approximately 86% or greater survey participants finding these skills as important career competencies, further investigation will focus on the 6 previously identified EDCs that emerged from interviews (communication, teamwork, problem solving & critical thinking, priority & time management, self-motivation, and technical skills). Table 6.29. Participants from each career goal used in multi-response testing (n). Skill Communicating Effectively Teamwork Skills Embracing Change Solving Problems Working with Data Priority & Time Management Professional/Technical Skills Acquiring Knowledge Working in a Diverse Environment Curiosity Self-motivation # of Multi-response Participants by Semester Fa19 Sp20 Su20 Total 971 971 971 971 971 971 971 971 971 — — 884 884 884 884 884 884 884 884 884 884 884 94 94 94 94 94 94 94 94 94 94 94 1949 1949 1949 1949 1949 1949 1949 1949 1949 978 978 371 Table 6.30. Overall Multi-response results including all skills provided in prompt, n, (%)(percentages represent percent within total response participants across all semesters as provided in Table 6.29). Skill Communicating Effectively Teamwork Skills Solving Problems Priority & Time Management Professional/Technical Skills Acquiring Knowledge Embracing Change Working with Data Working in a Diverse Environment Curiosity Self-motivation Skill Selected as Needed for Career 1825 (94) 1780 (91) 1827 (91) 1808 (93) 1796 (92) 1797 (92) 1769 (91) 1680 (86) 1794 (92) 862 (88) 909 (93) When drawing parallels between survey and interview responses, it should be noted that problem solving & critical thinking, work ethic, and technical skills identified through thematic analysis in interviews were presented to survey respondents as solving problems (problem solving), working with data (critical thinking), self-motivation (work ethic), and professional/technical skills (technical skills) respectively. Although interview results contained the main skill set problem solving & critical thinking, combining survey results of solving problems and working with data into this broader skill set was not a feasible option. This was due to solving problems and working with data being identified independently in the multiple- choice question preceding the open answer prompt and the outcomes of multi-response selection being used in selection of open answer responses for qualitative analysis. Because problem solving & critical thinking is represented as two separate skills in surveys, this resulted in the 7 EDCs of communication, teamwork, solving problems, working with data, priority & time management, self-motivation, and technical skills being investigated in these multi-response results. Beyond investigating general trends, these 6 skills were also used to determine if there was a significant difference between demographic variables and whether a skill was perceived as a valuable career competency and developed within the courses. Investigation into the association of prior experience, class standing, and first-generation status regarding selection of a skill as an important career competency returned insignificant results or results with negligible 372 effect sizes (Table 6.31 – 6.33). Comparison of skills selected as valuable career competencies disaggregated by career goal also showed no discernable differences in trends. This was further confirmed using chi-square test of association, that found no significant differences between career goal and skill selection for teamwork, solving problems, priority & time management, professional/technical skills, and self-motivation (Table 6.34) and although there were significant differences detected for communication (𝜒2 = 12.360, p = 0.012, df = 2) and working with data (𝜒2 = 14.840, p = 0.006, df = 2) the effect sizes were very small (V = 0.080 and 0.087 respectively) and most likely due to random variances that can occur in such a large samples. Table 6.35 displays testing regarding skill selection and learning environment that was further expanded upon in the main body of Chapter VI. Corresponding tables can be found on pages 373 – 376. Table 6.31. Multi-response of skills selected as valuable for career goal disaggregated by participants having prior experience in field, n, (%)(percentages represent percent within prior experience). Chi-square Test p-value df Test Value 1.094 2.092 0.381 0.495 0.342 0.704 1.029 0.495 1.054 0.495 0.046 0.685 0.921 0.589 Effect Size — — — -0.130 -0.137 — — 1 1 1 1 1 1 1 Selected as Needed for Career in Multi-response Yes, has had prior experience No, has not had prior experience 35 (100) 33 (94) 33 (94) 31 (89) 32 (91) 27 (77) 31 (89) 55 (93) 48 (81) 52 (88) 46 (78) 48 (81) 43 (73) 47 (80) Skill Communicating Effectively Teamwork Skills Solving Problems Priority & Time Management Professional/ Technical Skills Working with Data Self-motivation *Sample sizes account for SU20 semester only (n = 94). 373 Table 6.32. Multi-response of skills selected as valuable for career goal disaggregated by participant’s class standing, n, (%)(percentages represent percent within class standing). Skill Communicating Effectively§ Teamwork Skills§ Solving Problems§ Priority & Time Management§ Professional/ Technical Skills§ Working with Data§ Self-motivation† Selected as Needed for Career in Multi- response Chi-square Test Lower Classman Upper Classman Test Value p-value df 1451 (94) 1416 (92) 1455 (94) 1443 (93) 1428 (92) 1336 (86) 706 (93) 374 (93) 364 (90) 372 (92) 0.195 0.27 6 1.004 0.791 0.753 0.495 365 (91) 2.569 0.271 368 (92) 344 (86) 203 (93) 0.163 0.107 0.000 0.814 0.858 1.000 1 1 1 1 1 1 1 § Sample sizes account for Fa19, Sp20, and Su20 semesters and are comprised of lower classman n = 1547, and upper classman n = 402. † Sample sizes are smaller due to skill(s) being present in Sp20 and Su20 semesters only and are comprised of lower classman n = 760, and upper classman n = 218. Table 6.33. Multi-response of skills selected as valuable for career goal disaggregated by participants being continuing- or first-generation students, n, (%) (percentages represent percent within first-generation student status). Skill Communicating Effectively§ Teamwork Skills§ Solving Problems§ Priority & Time Management§ Professional/ Technical Skills§ Working with Data§ Self-motivation† Selected as Needed for Career in Multi-response, n (%) Continuing- Generation First- Generation 1430 (94) 1392 (91) 1437 (94) 1424 (93) 1409 (92) 1313 (86) 709 (94) 395 (93) 388 (91) 390 (92) 384 (90) 387 (91) 367 (86) 200 (91) Chi-square Test Test Value p- value df 0.306 0.000 3.198 0.737 1.000 0.209 4.265 0.124 0.712 0.001 1.416 0.589 1.000 0.437 1 1 1 1 1 1 1 § Sample sizes account for Fa19, Sp20, and Su20 semesters and are comprised of continuing-generation students n = 1524, and first-generation students n = 425.† Sample sizes are smaller due to skill(s) being present in SP20 and SU20 semesters only and are comprised of lower classman n = 758, and upper classman n = 220. 374 Table 6.34. Multi-response of skills selected as valuable for career goal disaggregated by participant’s desired career, n, (%) (percentages represent percent within career goal). Selected as Needed for Career in Multi- response, n (%) Engineering & Associated Sub- specialities Health & Medical Professions Other Career Goals Chi-square Test Test Value p - value df Effect Size 911 (86) 347 (86) 441 (88) 12.360 0.012 955 (92) 369 (91) 458 (90) 2.308 0.495 931 (90) 346 (85) 445 (87) 5.214 0.209 952 (92) 349 (86) 452 (88) 14.840 0.006 918 (89) 337 (83) 437 (86) 2.733 0.444 840 (81) 447 (82) 295 (73) 132 (68) 405 (79) 6.805 0.116 168 (70) 0.125 0.986 2 2 2 2 2 2 2 0.080 — — 0.087 — — — Skill Communicating Effectively§ Teamwork Skills§ Solving Problems§ Working with Data§ Priority & Time Management§ Professional/ Technical Skills§ Self-motivation† § Sample sizes account for Fa19, Sp20, and Su20 semesters and are comprised of health & medical professionals n = 1032, engineering & associated subspecialties n = 406, and other career goals n = 511.† Sample sizes are smaller due to skill(s) being present in Sp20 and Su20 semesters only and are comprised of health & medical professionals n = 545, engineering & associated subspecialties n = 194, and other career goals n = 239. Table 6.35. Multi-response of skills selected as valuable for career goal disaggregated by learning environment, n, (%)(percentages represent percent within learning environment). Chi-square Test Selected as Needed for Career in Multi-response, n (%) Emergency Remote Learning Transition 848 (96) Online Learning 90 (96) In-Person Learning 887 (91) 874 (90) 898 (92) 827 (85) 825 (93) 844 (96) 783 (89) 81 (86) 85 (90) 70 (74) Test Value 17.018 9.742 8.914 15.931 p - value 0.000 0.041 0.052 0.006 892 (92) 839 (95) 77 (82) 23.721 0.006 877 (90) 839 (95) 80 (85) 20.251 0.259 — 831 (94) 78 (83) 14.120 0.006 Effect Size (V) 0.093 0.071 — 0.090 0.110 — -0.127 df 2 2 2 2 2 2 1 Skill Communicating Effectively Teamwork Skills Solving Problems Working with Data Priority & Time Management Professional/ Technical Skills Self-motivation† § Sample sizes are representative of each individual semesters and are comprised of in-person learning n = 971, emergency remote learning transition n = 884, and online learning n = 94. † Skill was only included in Sp20 (emergency remote learning transition) and Su20 (online learning) semesters. 375 Table 6.36. Adjusted residuals of learning environment and selection of priority & time management as a skill needed for career goal(s)(Bonferroni adjusted critical value for residuals ± = 2.6). Variable Groups Tested Priority & Time Management Chosen as Skill Needed x In-person Learning Priority & Time Management Not Chosen as Skill Needed x In-person Learning Priority & Time Management Chosen as Skill Needed x Emergency Remote Learning Transition Priority & Time Management Not Chosen as Skill Needed x Emergency Remote Learning Transition Priority & Time Management Chosen as Skill Needed x Online Learning Priority & Time Management Not Chosen as Skill Needed x Online Learning Adjusted Residual -1.5 Expected Count 900.8 Observed Count 892 1.5 3.3 -3.3 -4.2 4.2 70.2 820.0 64.0 87.2 6.8 79 839 45 77 17 376 A.VI.23. Ranking Data & Results Concerning interviews, teamwork had the highest median (median = 2) and value among the six skills, followed by communication, problem-solving & critical thinking, prioritization & time management, and self-motivation skills all sharing the same median (median = 3)(Figure 6.25). Technical skills was perceived to have the lowest importance accompanied by the highest median (median = 5). In contrast to interviews, a majority of Fa20 survey participants (43%) ranked problem solving & critical thinking with the highest importance. The value survey participants attributed to problem solving & critical thinking was also supported by this skill having lowest median (median = 2), indicating that it was identified as more valuable in comparison to the other 5 EDCs listed. Communication, teamwork, and priority & time management were ranked with more even distributions from high to low importance with an equivalent median (median = 3) for all three EDCs. Although self-motivation skills had the largest portion of participants recognizing this skill as having the highest importance (29%), the median ranking for this skill was equivalent to communication, teamwork, and priority & time management (median = 3). Continued examination of Fa20 survey participant rankings found a few areas of statistical significance explored within the main body of Chapter VI. The outcomes of these tests are displayed below (Table 6.37 – Table 6.44). Corresponding figure and tables on 475 - 481. 377 a. ) n ( t n u o C c. ) n ( t n u o C 500 400 300 200 100 0 500 400 300 200 100 0 e. ) n ( t n u o C 500 400 300 200 100 0 1 2 4 3 Ranking 5 6 1 2 3 4 Ranking 5 6 b. ) n ( t n u o C d. ) n ( t n u o C f. ) n ( t n u o C 500 400 300 200 100 0 500 400 300 200 100 0 500 400 300 200 100 0 1 2 4 3 Ranking 5 6 1 2 3 4 Ranking 5 6 1 2 4 3 Ranking 5 6 1 2 4 3 Ranking 5 6 Figure 6.25. Fa20 survey participant ranking of employer-desired competencies, represented by number of participants (n = 1115). Skills corresponding to rankings follow a. Communication skills, b. Teamwork skills, c. Problem-solving & Critical Thinking Skills, d. Priority & Time Management, e. Self-motivation, and f. Technical Skills. 378 Table 6.37. Fa20 Ranking results by prior experience (n = 1,115). Prior Experience Yes, has had prior experience No, has not had prior experience Employer-desired Competencies Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Self-motivation Technical Skills Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Self-motivation Technical Skills Ranking (n) 1 83 44 143 2 88 58 71 3 78 72 56 4 48 74 37 5 27 49 25 6 11 38 3 Total Median 335 335 335 2 3 2 50 79 61 70 52 23 335 50 52 37 47 59 43 92 47 38 40 77 187 198 168 121 129 157 163 139 114 51 335 165 139 79 54 111 29 78 11 335 335 780 780 780 123 197 161 139 125 35 780 227 108 109 73 111 128 97 121 121 118 113 234 780 780 3 3 4 3 3 2 3 3 4 Table 6.38. Kruskal-Wallis test results for prior experience (n = 1,115). Skill Communication Teamwork Problem Solving & Critical Thinking Time Management Self-motivation Technical Skills Kruskal-Wallis Test Value 0.702 3.302 0.033 1.497 0.463 0.062 Significance 0.589 0.209 0.929 0.425 0.677 0.911 379 Table 6.39. Fa20 Ranking results by first-generation status (n = 1,115). Ranking (n) First-Generation Status First- Generation Student Continuing- Generation Student Employer-desired Competencies Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Self-motivation Technical Skills Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Self-motivation Technical Skills 1 62 42 2 63 43 3 52 45 4 27 42 5 14 27 6 Total Median 5 24 223 223 2 3 92 47 38 26 17 3 223 65 35 26 43 33 31 27 38 39 32 29 69 38 25 31 90 208 223 194 142 131 172 190 171 136 11 25 72 35 92 223 223 223 892 892 386 189 157 90 59 11 892 134 211 179 171 150 47 250 123 113 111 138 157 147 137 127 273 273 86 892 892 892 2 3 3 4 3 3 2 3 3 4 Table 6.40. Kruskal-Wallis test results for first-generation status (n = 1,115). Skill Communication Teamwork Problem Solving & Critical Thinking Time Management Self-motivation Technical Skills Value 6.725 1.452 0.556 4.640 3.970 0.973 Kruskal-Wallis Test Significance 0.049 0.432 0.646 0.112 0.142 0.496 Effect Size 0.006 — — — — — 380 Table 6.41. Fa20 Ranking results by career goal (n = 1,115). Ranking Career Goal Health & Medical Professions Engineering & Associated Subspecialties Other Career Goals 2 5 44 115 124 105 106 4 77 6 Total Median 14 70 604 604 3 3 1 Employer-desired Competencies 3 Communication Skills 164 158 147 Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Self-motivation 263 123 101 86 186 84 69 42 6 604 152 108 128 83 74 89 66 26 40 87 46 45 98 36 43 83 35 32 95 90 75 26 9 35 82 195 12 12 604 604 604 181 181 83 45 26 18 8 1 181 38 41 9 80 49 34 18 36 82 55 40 25 29 63 68 30 18 31 57 76 34 32 31 34 48 5 47 45 14 34 181 181 181 330 330 132 68 68 29 26 7 330 49 92 36 90 51 50 74 47 41 51 39 44 48 48 54 18 53 105 330 330 330 2 3 3 4 3 3 2 3 4 4 2 3 2 3 3 4 Technical Skills Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Self-motivation Technical Skills Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Self-motivation Technical Skills Table 6.42. Kruskal-Wallis test results for career goal ( n = 1,115). Skill Communication Teamwork Problem Solving & Critical Thinking Time Management Self-motivation Technical Skills Kruskal-Wallis Test Value 23.628 23.281 3.679 2.177 14.209 0.514 Significance 0.006 0.006 0.361 0.510 0.006 0.884 Effect Size 0.022 0.019 — — 0.011 — 381 Table 6.43. Fa20 Ranking results by class standing (n = 1,115). Ranking (n) Class Standing Lower Classman Employer-desired Competencies Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Self-motivation Technical Skills Upper Classman Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Self-motivation Technical Skills 2 3 1 4 5 87 197 213 187 121 138 168 176 159 117 6 31 78 Total Median 836 836 3 3 360 178 142 85 61 10 836 127 214 173 154 127 41 836 253 113 114 107 118 131 124 127 127 118 258 82 9 48 73 73 38 54 47 35 4 31 58 118 59 59 53 17 46 15 836 836 279 279 279 46 62 49 55 50 17 279 66 29 45 49 32 41 33 31 52 42 51 87 279 279 2 3 3 4 2 3 2 3 3 4 Table 6.44. Kruskal-Wallis test results for class standing (n = 1,115). Skill Communication Teamwork Problem Solving & Critical Thinking Time Management Self-motivation Technical Skills Value 1.56 7.151 0.018 1.057 4.495 0.144 Kruskal-Wallis Test Significance 0.42 0.037 0.954 0.495 0.117 0.820 Effect Size — 0.006 — — — — Investigation of Sp21 rankings show a different distribution of rankings amongst four of the six EDCs explored in Fa20 interview and survey prompts (Figure 6.26 on page 382). Although Sp21 participants ranked problem solving & critical thinking similar to Fa20 survey participants with a majority participants ranking this skill as having the highest importance (42%) and having the same median rank (median = 2), this skill was not awarded the highest value. For Sp21 participants, communication skills had the lowest median (median = 1) indicating that more participants attributed this skill with the highest importance. Teamwork skills were given the same median ranking as problem-solving & critical thinking (median = 2) and prioritization & time management were perceived as having the lowest value of the four (median = 3). There were no significant differences found when considering first-generation status or class standing, the only two variables tested against for this semester (Table 6.45 – Table 6.48 on pages 383 - 385). 382 a. ) n ( t n u o C 150 135 120 105 90 75 60 45 30 15 0 c. ) n ( t n u o C 150 135 120 105 90 75 60 45 30 15 0 b. ) n ( t n u o C 150 135 120 105 90 75 60 45 30 15 0 1 2 3 4 Ranking 1 2 3 Ranking 4 5 d. 150 135 120 105 ) n ( t n u o C 90 75 60 45 30 15 0 1 2 3 Ranking 4 5 1 2 3 Ranking 4 5 Figure 6.26. Sp21 ranking of employer-desired competencies, represented by number of participants (n = 213)(n). Skills corresponding to rankings follow a. Communication skills, b. Teamwork skills, c. Problem Solving & Critical Thinking Skills, and d. Priority & Time Management. 383 Table 6.45. Sp21 Ranking results by first-generation status, represented by number of participants (n = 213)(n). Class Standing Employer-desired Competencies First-Generation Student Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Continuing- Generation Student 5 Total Median 1 3 2 33 12 14 28 17 9 27 11 14 Ranking (n) 4 1 — 6 — 8 — 60 60 60 12 22 12 14 — 6 — 96 35 16 1 73 45 20 14 1 63 45 26 18 60 153 153 153 29 39 47 36 2 153 1 2 2 2 1 2 2 3 Table 6.46. Kruskal-Wallis test results for first-generation status (n = 213). Skill Communication Teamwork Problem Solving & Critical Thinking Time Management Kruskal-Wallis Test Value 1.633 0.042 0.010 0.988 Significance 0.416 0.921 0.973 0.496 Table 6.47. Sp21 Ranking results by class standing, represented by number of participants (n = 213)(n). Class Standing First- Generation Student Continuing- Generation Student Employer-desired Competencies Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management Communication Skills Teamwork Skills Problem Solving & Critical Thinking Priority & Time Management 1 110 83 73 35 19 18 17 6 2 38 54 45 55 9 8 11 3 25 22 35 44 5 7 5 6 15 Ranking (n) 4 4 17 23 41 3 3 3 9 5 — 1 1 2 — — — — Total 177 Median 1 177 177 177 36 36 36 36 2 2 2 1 2 2 3 384 Table 6.48. Kruskal-Wallis test results for class standing (n = 213). Skill Communication Teamwork Problem Solving & Critical Thinking Time Management Kruskal-Wallis Test Value 1.390 0.004 1.160 1.250 Significance 0.437 0.987 0.478 0.453 Sp21 participants were also provided spaces to list additional skills, with accompanied rankings, that they perceived as valuable to their career. Although no discernable trends were found, the skills found within student responses are presented below in Table 6.49. Table 6.49. Additional skills students listed in the Sp21 ranking prompt, represented by number of participants (n). Main Skill Set Interpersonal Skills Subskill Category Communication Skills Teamwork & Collaboration Social Skills Misc. Interpersonal Skills Work Ethic Intrapersonal Skills Personality & Character Traits Specific Skill Mentioned Listening Skills (includes active listening & listening) Public Speaking (includes presenting to groups) Writing Splitting Up Work Equally Dedication from the Group Members Getting [Along] with Each Other Dependability Leadership Motivation Perseverance Personal Responsibility Progress and Reflection Responsibility Staying [on] Task Work Ethic Adaptability Confidence Honesty Openness to New Ideas Patience Positive Attitude 385 Ranking (n) 1 2 3 4 5 6 7 2 2 2 1 1 1 1 1 2 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 Table 6.49 (cont’d) Problem Solving & Critical Thinking — Priority & Time Management — Occupation-Specific Skills Technical Skills Field-Specific Knowledge Field-Specific Knowledge Misc. Occupation-specific Skills Education & Learning Skills Other Skills Education & Credentials — Breaking Down Problems into Manageable Parts Creativity Finding Alternatives Forecasting Summarizing Data/Information Efficiency Organization Planning Turning in Things on Time Technical Skills (includes: technology skills) Research Skills Research and Planning Preparing Reports Process Documentation Professionalism Having Educational Conversations Attention to Detail Diversity Involvement Participation Work Split 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 386 A.VI.24. Multi-response Results Below are all tables used to determine the outcomes of multi-response prompts as discussed in Chapter VI. Each skill was first filtered for participants who recognized a skill as being necessary for their future career goal before exploring development and differences in the selection process using demographic variables. This resulted in each skill containing differing number of participants. All corresponding tables on pages 387 – 394. Table 6.50. Participants who selected communication skills as being developed within the course(s) out of those who also recognized this skill as a valuable career competency within Fa19, Sp20, and Su20 semesters, n, (%). Demographic Prior Experience† Chi-square Test Skill Selected as Developed, n (%) Test Value p-value df Yes, has had prior experience (n = 55) 46 (84) No, has not had prior experience (n = 35) 32 (91) Career Goal Health & Medical Professions (n = 985) 869 (88) Engineering & Associated Subspecialties (n = 374) 317 (85) Other Career Goals (n = 466) 402 (86) First-Generation Status 0.551 0.646 3.188 0.416 First-Generation Student (n = 395) 335 (85) 1.925 0.361 Continuing-Generation Student (n = 1,430) 1,253 (88) Class Standing Lower Classman (n=1,451) 1,271 (88) Upper Classman (n = 374) 317 (85) 1.872 0.368 Learning Environment In-person Learning (n = 887) 788 (89) 5.255 0.209 Emergency Remote Learning Transition (n = 848) 722 (85) Online Learning (n = 90) 78 (87) 1 2 1 1 2 † Sample sizes are smaller due to demographic variable being accounted for in the Su20 semester only, totaling n = 90 participants who selected communication skills as needed for their future career goal within this semester and used in testing. 387 Table 6.51. Participants who selected teamwork skills as being developed within the course(s) out of those who also recognized this skill as a valuable career competency within Fa19, Sp20, and Su20 semesters, n, (%). Demographic Prior Experience† Skill Selected as Developed Test Value p-value df Chi-square Test Yes, has had prior experience (n = 33) 32 (97) 0.255 0.764 1 No, has not had prior experience (n = 48) 44 (92) Career Goal Health & Medical Professions (n = 935) 860 (92) Engineering & Associated Subspecialties (n = 378) 343 (91) Other Career Goals (n = 467) 417 (89) First-Generation Status 2.788 0.437 2 First-Generation Student (n = 388) 344 (89) 2.996 0.220 1 Continuing-Generation Student (n = 1,392) 1,276 (79) Class Standing Lower Classman (n = 1,416) 1,296 (92) Upper Classman (n = 364) 324 (89) 1.941 0.361 Learning Environment In-person Learning (n = 874) 803 (92) 741 (90) Emergency Remote Learning Transition (n = 825) Online Learning (n = 81) 76 (94) 3.020 0.425 1 2 † Sample sizes are smaller due to demographic variable being accounted for in the Su20 semester only, totaling n = 81 participants who selected teamwork skills as needed for their future career goal within this semester and used in testing. 388 Table 6.52. Participants who selected solving problems as being developed within the course(s) out of those who also recognized this skill as a valuable career competency within Fa19, Sp20, and Su20 semesters, n, (%). Demographic Prior Experience† Chi-square Test Skill Selected as Developed Test Value p-value df Yes, has had prior experience (n = 33) 26 (79) No, has not had prior experience (n = 52) 44 (85) Career Goal Health & Medical Professions (n = 978) 880 (90) Engineering & Associated Subspecialties (n = 380) 326 (86) Other Career Goals (n = 469) 405 (86) First-Generation Status 0.156 0.815 6.621 0.117 First-Generation Student (n = 390) 340 (87) 0.369 0.704 Continuing-Generation Student (n = 1,437) 1,271 (88) Class Standing Learning Environment Lower Classman (n = 1,455) 1,297 (89) Upper Classman (n = 372) 314 (84) 5.918 0.064 In-person Learning (n = 898) 808 (90) 733 (87) Emergency Remote Learning Transition (n = 844) Online Learning (n = 85) 70 (82) 6.988 0.112 1 2 1 1 2 † Sample sizes are smaller due to demographic variable being accounted for in the Su20 semester only, totaling n = 85 participants who selected solving problems as needed for their future career goal within this semester and used in testing. 389 Table 6.53. Participants who selected working with data as being developed within the course(s) out of those who also recognized this skill as a valuable career competency within Fa19, Sp20, and Su20 semesters, represented by number and the related percent of participants (n = 1,680), n, (%). Demographic Prior Experience† Chi-square Test Skill Selected as Developed Test Value p-value df Effect Size Yes, has had prior experience (n = —§ 1.000 1 — No, has not had prior experience (n = 43) 36 (84) 27) 23 (85) Career Goal Health & Medical Professions 10.980 0.022 2 0.081 (n = 861) 788 (92) Engineering & Associated Subspecialties (n = 367) 315 (86) Other Career Goals (n = 452) 394 (87) First-Generation Status First-Generation Student (n = 367) 320 (87) 1.528 0.425 1 — Continuing-Generation Student (n = 1,313) 1,177 (90) Class Standing Lower Classman (n = 1,336) 1,203 (90) Upper Classman (n = 344) 294 (86) 5.449 0.082 Learning Environment In-person Learning (n = 827) 740 (90) 698 (89) Emergency Remote Learning Transition (n = 783) Online Learning (n = 70) 59 (84) 1.796 0.589 1 2 — † Sample sizes are smaller due to demographic variable being accounted for in the Su20 semester only, totaling n = 70 participants who selected working with data as needed for their future career goal within this semester and used in testing. § Fisher exact test that returned no chi-square test value associated with significance. 390 Table 6.54. Participants who selected priority & time management as being developed within the course(s) out of those who also recognized this skill as a valuable career competency within Fa19, Sp20, and Su20 semesters, represented by number and the related percent of participants (n = 1,808), n, (%). Demographic Prior Experience† Skill Selected as Developed Test Value p-value df Chi-square Test Yes, has had prior experience (n = 31) 25 (81) 0.384 0.704 1 No, has not had prior experience (n = 46) 33 (72) Career Goal Health & Medical Professions 7.132 0.111 2 (n = 965) 854 (89) Engineering & Associated Subspecialties (n = 377) 314 (83) Other Career Goals (n = 466) 398 (85) First-Generation Status First-Generation Student (n = 384) 328 (85) 0.480 0.674 1 Continuing-Generation Student (n = 1,424) 1,238 (87) Class Standing Lower Classman (n = 1,443) 1,259 (87) Upper Classman (n = 365) 307 (84) Learning Environment In person Learning (n = 892) 777 (87) 731 (87) Emergency Remote Learning Transition (n = 839) Online Learning (n = 77) 58 (75) 2.213 0.322 8.843 0.054 1 2 † Sample sizes are smaller due to demographic variable being accounted for in the Su20 semester only, totaling n = 77 participants who selected priority & time management as needed for their future career goal within this semester and used in testing. 391 Table 6.55. Participants who selected self-motivation as being developed within the course(s) out of those who also recognized this skill as a valuable career competency within Sp20 and Su20 semesters, represented by number and the related percent of participants (n = 909), n, (%). Demographic Prior Experience Yes, has had prior experience (n = 31) No, has not had prior experience (n = 47) Career Goal Health & Medical Professions (n = 507) Engineering & Associated Subspecialties (n = 181) Other Career Goals (n = 221) First-Generation Status First-Generation Student (n = 200) Continuing-Generation Student (n = 709) Class Standing Lower Classman (n = 706) Upper Classman (n = 203) Learning Environment Emergency Remote Learning Transition (n = 831) Online Learning (n = 78) Skill Selected as Developed Test Value p-value df Effect Size Chi-square Test 20 (65) 0.040 0.921 1 — 28 (60) 419 (83) 125 (69) 157 (71) 20.055 0.006 2 0.149 150 (75) 0.507 0.664 1 — 551 (78) 556 (79) 145 (71) 4.388 0.117 1 — 653 (79) 10.790 0.006 1 -0.114 48 (62) † Sample sizes are smaller due to demographic variable being accounted for in the Su20 semester only, totaling n = 78 participants who selected self-motivation as needed for their future career goal within this semester and used in testing. Table 6.56. Adjusted residuals of career goal and selection of self-motivation as a skill developed within the course(s)(Bonferroni adjusted critical value for residuals ± = 2.6). Variable Groups Tested Self-motivation Chosen as Skill Developed x Health & Medical Professions Self-motivation Not Chosen as Skill Developed x - Health & Medical Professions Self-motivation Chosen as Skill Developed x Engineering & Associated Subspecialties Self-motivation Not Chosen as Skill Developed x Engineering & Associated Subspecialties Self-motivation Chosen as Skill Developed x Other Career Goals Self-motivation Not Chosen as Skill Developed x Other Career Goals Adjusted Residual 4.7 Expected Count 416.3 Observed Count 447 128.7 148.2 45.8 182.5 56.5 98 132 62 168 71 -4.7 -3.1 3.1 -2.5 2.5 392 Table 6.57. Participants who selected technical skills as being developed within the course(s) out of those who also recognized this skill as a valuable career competency within Fa19, Sp20, and Su20 semesters, represented by number and the related percent of participants (n = 1,796), n, (%). Demographic Prior Experience† Yes, has had prior experience (n = 32) No, has not had prior experience (n = 48) Career Goal Health & Medical Professions (n = 966) Engineering & Associated Subspecialties (n = 370) Other Career Goals (n = 460) First-Generation Status First-Generation Student (n = 387) Continuing-Generation Student (n = 1,409) Class Standing Skill Selected as Developed Test Value p-value df Effect Size Chi-square Test 11 (34) 0.055 0.916 1 — 21 (44) 782 (81) 15.409 0.006 2 0.093 263 (71) 362 (79) 301 (79) 0.367 0.704 1 — 1,106 (79) Lower Classman (n = 1,428) Upper Classman (n = 368) 1,151 (81) 256 (70) Learning Environment In-person Learning (n = 877) Emergency Remote Learning Transition (n = 839) Online Learning (n = 80) 706 (81) 669 (80) 32 (40) 20.361 0.006 1 -0.108 72.686 0.006 2 0.201 † Sample sizes are smaller due to demographic variable being accounted for in the Su20 semester only, totaling n = 80 participants who selected technical skills as needed for their future career goal within this semester and used in testing. 393 Table 6.58. Adjusted residuals of learning environment and selection of technical skills as a skill developed within the course(s)(Bonferroni adjusted critical value for residuals ± = 2.6). Variable Groups Tested Technical Skills Chosen as Skill Developed x In-person Learning Technical Skills Not Chosen as Skill Developed x - In-person Learning Technical Skills Chosen as Skill Developed x Emergency Remote Learning Transition Technical Skills Not Chosen as Skill Developed x Emergency Remote Learning Transition Technical Skills Chosen as Skill Developed x Online Learning Technical Skills Not Chosen as Skill Developed x Online Learning Adjusted Residual 2.2 Expected Count 687 Observed Count 706 -2.2 1.3 -1.3 -8.5 8.5 190 657.3 181.7 62.7 17.3 171 669 170 32 48 394 A.VI.25. Multiple Choice Selection Corresponding to Open Answer Responses Regarding Development in Fa19, Sp20, and Su20 Survey Administration Below are the results of the multiple-choice skill selections in which students chose a skill via multiple choice to elaborate on development in the courses through open answer responses. Table 6.59 contains the overall tabulations of skills students selected to talk about, while Table 6.60 filters for students who selected a skill as needed and developed within the multi-response prompt and investigates who further chose to speak about development of each particular skill in open answer responses. Open responses were sampled and any skill where ≥ 20% of participants selected the skill as needed and developed in the multi-response portion of the survey were included in analysis. Table 6.59. Combined overall frequencies for open answer skill choices by semester with no added filter (n = 2,315), n (%). Employer-desired Competency (EDC) Communication Skills Teamwork Skills Solving Problems Prioritization & Time Management Technical Skills Acquiring Knowledge Embracing Change Working with Data Working in a Diverse Environment Curiosity Self-motivation Fa19 (n = 1,229)* 489 (40) 661 (54) 323 (26) 327 (27) 69 (6) 111 (9) 79 (6) 339 (28) 60 (5) 0 (0) 0 (0) Semester, n (%) Sp20 (n = 985)* 361 (37) 540 (55) 165 (17) 219 (22) 59 (6) 73 (7) 103 (10) 295 (30) 42 (4) 27 (3) 86 (9) Su20 (n = 101) 24 (24) 31 (31) 11 (11) 8 (8) 3 (3) 2 (2) 4 (4) 13 (13) 0 (0) 1 (1) 4 (4) *Participants for FA19 (n = 1,229) and SP20 (n = 986) are represented twice (never for the same skill) due to the aggregation of skill choice 1 and 2 for these semesters resulting in column total being double the sample size. † Curiosity and self-motivation was not available for selection in Fa19 survey administration. 395 Table 6.60. Skill choice in Fa19 semester when filtering for participants who selected skill as needed for their future career and developed in the course in multi-response prompt (n = 1,229). Employer-desired Competency (EDC) Communication Skills Teamwork Skills Solving Problems Prioritization & Time Management Technical Skills Acquiring Knowledge Embracing Change Working with Data Working in a Diverse Environment Selected Skill as Needed/Developed in Multi-Response (n) 788 803 808 777 Skill Choice Multiple Choice Response for Open Answer (n) 328 449 213 222 % of Participants Who Chose to Talk About Skill in Open Answer (%) 42 56 26 29 706 748 636 740 700 44 57 52 212 35 6 8 8 29 5 Table 6.61. Skill choice in Sp20 semester when filtering for participants who selected skill as needed for their future career and developed in the course in multi-response prompt (n = 985). Employer-desired Competency Communication Skills Teamwork Solving Problems Prioritization & Time Management Technical Skills Acquiring Knowledge Embracing Change Working with Data Working in a Diverse Environment Curiosity Self-motivation Selected Skill as Needed/Developed in Multi-Response (n) 722 741 733 731 Skill Choice Multiple Choice Response for Open Answer (n) 279 414 125 172 % of Participants Who Chose to Talk About Skill in Open Answer (%) 39 56 17 24 669 697 627 698 669 556 653 48 47 76 223 29 4 16 7 7 12 32 4 1 2 396 Table 6.62. Skill choice in Su20 semester when filtering for participants who selected skill as needed for their future career and developed in the course in multi-response prompt (n = 101). Employer-desired Competency Communication Skills Teamwork Solving Problems Prioritization & Time Management Technical Skills Acquiring Knowledge Embracing Change Working with Data Working in a Diverse Environment Curiosity Self-motivation Selected Skill as Needed/Developed in Multi-Response (n) 78 76 70 58 Skill Choice Multiple Choice Response for Open Answer (n) 22 21 9 6 % of Participants Who Chose to Talk About Skill in Open Answer (%) 28 28 13 10 32 61 31 59 37 29 48 1 1 1 9 0 0 3 3 2 3 15 0 0 6 Table 6.63. Number of participants sampled from and included in open answer analyses (n). Competency Communication Skills Teamwork Skills Problem-solving & Critical Thinking Prioritization & Time Management Total Fa19 60 60 120* 60 300 Sp20 60 60 60† 60 240 Su20 22 20§ — — 42 Total 142 140 180 120 582 Semester *Combines solving problems (n = 60) & working with data (n = 60). † Just working with data (n = 60). § One participant was removed from analysis due to answers pertaining to recitation and not lab. 397 A.VI.26. Skills Fa20 Participants Spoke Most of Developing Below in Table 6.64 are the number of students who spoke of perceived development for each of the six EDCs provided. Because ≥20% of participants spoke of development for each skill, all skills were included in analysis. Table 6.64. Participants who related employer-desired competencies to development within the general chemistry laboratory courses, n, (%). Employer-desired Competency Communication Skills Teamwork Skills Problem-solving & Critical Thinking Prioritization & Time Management Self-motivation (Work Ethic) Technical Skills Participants Who Perceived Development 121 (61) 152 (76) 76 (38) 70 (35) 65 (33) 41 (21) 398 A.VI.27. Skills Fa21 Participants Spoke Most of Developing Fa21 analysis followed a similar pattern, of investigating development within the courses, as was applied to interviews. Students were asked in the Fa21 final reflective assignment survey to express the skills they perceived to be needed for their future career followed by a prompt asking students to use these skills to relate course components to skill development. The top six skills students recognized as valuable career competencies were communication skills (n = 71, 36%), teamwork skills (n = 63, 32%), work ethic (n = 62, 31%), field-specific knowledge (n = 74, 37%), problem-solving & critical thinking (n = 49, 25%), and prioritization & time management (n = 56, 28%)(Figure 6.27). Of the Fa21 survey participants who recognized these skills as valuable, most chose to relate the course to development of teamwork (n = 59, 94%), communication (n = 55, 77%), prioritization & time management (n = 40, 71%), and problem- solving & critical thinking (n = 32, 65%) skills (Table 6.64). Less frequently, those who mentioned needing work ethic skills and field-specific knowledge related development to the general chemistry laboratory courses (n = 17, 27% and n = 27, 36% respectively). The skills in which ≥20% of participants spoke of development were included in qualitative analysis and the number of participants included are found within Table 6.65. 399 ) % ( A R F 1 2 a F n i s t n a p i c i t r a P f o t n e c r e P 100 90 80 70 60 50 40 30 20 10 0 Figure 6.27. Expanded skill sets Fa21 final reflective assignment survey participants recognized as being valuable to their future career goal (%). Table 6.65. Perceived development of skills selected as valuable career competencies, represented by number and the related percent of participants (percentage is representative of percent of participants who perceived development out of those who selected a skill as valuable to career goal), n, (%). Student-perceived Competency Communication Skills (n = 71) Teamwork Skills (n = 63) Problem-solving & Critical Thinking (n = 49) Prioritization & Time Management (n = 56) Work Ethic (n = 62) Field-specific Knowledge (n = 74) Participants Who Perceived Development 55 (77) 59 (94) 32 (65) 40 (71) 17 (27) 27 (36) 400 A.VI.28. Course Element Codes To explore the ways in which participants related various course elements to perceived development of skills, survey responses across each of the five semesters included in analyses were combined for related skills. This resulted in the total sample sizes for each competency displayed in Table 6.66. Table 6.66. Number of participants represented in themes generated by qualitative analysis of open answer responses by skill and semester (n). Competency Communication Skills Teamwork Skills Problem-solving & Critical Thinking Prioritization & Time Management Work Ethic Technical Skills Field-specific Knowledge Total Fa19 60 60 120* 60 — — — 300 Sp20 60 60 60† 60 — — — 240 Semester Su20 22 20‡ — — — — — 42 Fa20 121 152 76 70 65§ 41 — 525 Fa21 55 59 32 40 17 — 27 230 Total 318 351 288 230 82 41 27 1,337 *Combines solving problems (n = 60) & working with data (n = 60). †Just working with data (n = 60). §Just self- motivation skills. ‡One participant was removed from analysis due to answers pertaining to recitation and not lab. From these aggregated results, major and minor themes were generated using the same method as applied to interviews (A.IV.17) and presented in Table 6.67 and Table 6.68. Course elements, not previously observed within interviews, that emerged in survey responses were a) solving problems & finding solutions, b) determining unknown, c) writing notebooks & lab reports, d) general lab experience, and e) other. These codes were the product of ambiguity in survey responses and the inability to further inquire the meaning behind comments - a key limitation of survey administration versus conducting interviews. Because these codes do not contribute to major or minor themes, surrounding skill development that are defined in the main body of Chapter VI, each of these codes is expanded upon below. Solving problems & finding solutions was a code created from broad remarks made by students that skill development occurred in the courses through having “to be able to use [skill] to solve the problems we are given in the lab,” (EDC_0848) without further expanding upon how this was done or factors that contributed to this action. This is distinct from the Finding Solutions to Proposed Problem code under Open Inquiry, in Table 6.70, based on additional context provided. If students referenced being given a project, scenario, or issue/problem at the 401 beginning of the project that needed to be solved with given resources, background information, and/or tools or mentioning that the course is “open-ended,” the code was sorted under open inquiry learning and defined as Finding Solutions to Proposed Problem. Another notable code that appeared in surveys was having to Determine Unknowns. This was defined by students having to determine an unknown variable that was given (e.g., determining what food dyes were present in an unknown beverage). This code was used broadly to encompass a student’s complete response, as well as in addition to other codes (e.g., using data analysis to determine unknowns present). While Writing Lab Notebooks & Reports is generally housed under Course Activities, general statements made by students in survey responses (e.g., “We had to some reports throughout the course” (EDC_2899)) facilitated generation of this code. It was through students mentioning performing experiments (e.g., implementing planned procedure, hands-on experiences with chemicals, taking measurements), learning how to work in a lab environment (e.g., lab etiquette, safety, waste disposal), and learning lab techniques when speaking of skill development that the code of General Lab Experience was created. This category was generated due to the belief that these are typical laboratory experiences one expects a student to engage in during their time in a general chemistry laboratory environment. The last code contained in Table 6.67 and 6.68 that had not been previously defined is the Other category. This code houses statements made by students that were not mentioned by many participants throughout the semesters included in analysis and those that did not fit within any other category within the coding scheme. This includes subcodes such as learning from mistakes and areas which can be improved, observing how teaching assistant’s teach and manage the classroom, having a willingness to learn, the desire to do well in the course, and overcoming challenges among others as factors in the courses that aided skill development. Major themes ( > 60 of participants who mentioned course element as contributing to development of a specific skill) and minor themes (20 – 60% of participants spoke of course element in relation to specific skill development) that contained additional subcodes were further unpacked in Table 6.69 and Table 6.70. These tables act as a point of reference for the results and discussion in Chapter VI. Course elements that contributed to perceived skill development represented by number of participants. All corresponding tables present on pages 403 – 406. 402 Table 6.67. Course elements that contributed to perceived skill development represented by number of participants (n). Skill Perceived as Developed in Course(s) Communication Skills (n = 318) 284 12 30 8 — Teamwork Skills (n = 351) 335 12 30 6 2 Problem- solving & Critical Thinking (n = 288) 50 73 28 35 4 Prioritization & Time Management (n = 230) 69 9 11 6 149 45 — 1 5 1 — 2 — — 1 2 2 — 2 — — 6 1 2 2 — — 1 1 3 — 6 1 9 107 27 25 27 — 19 19 1 3 1 — — — 8 — 2 — 1 — — 1 2 2 Course Element that Contributed to Perceived Development Collaborative Environment Open Inquiry Learning Course Activities Course Projects Individual Task, Assignment, & Project Management Conveying Information to Others Real World Context Conceptual Learning & Application Working with Data Using Course Resources Troubleshooting Reasoning & Sensemaking Persevering Through Course Solving Problems & Finding Solutions Determining Unknown Writing Lab Notebook & Reports General Lab Experience Other Key Color Percentage (%) 81 - 100 61 - 80 41 - 60 21 - 40 0 - 20 403 Table 6.68. A continuation of course elements that contributed to perceived skill development represented by number of participants (n). Skill Perceived as Developed in Course(s) Work Ethic (n = 82) 21 1 3 — 38 Technical Skills (n = 41) 2 — 2 — — Field-specific Knowledge (n = 27) 2 6 2 3 — — — 5 1 — 1 — 18 2 — — — 2 1 — 2 5 16 — 1 1 4 — 1 4 — 3 3 14 1 1 — 3 — — 1 1 7 1 Course Element that Contributed to Perceived Development Collaborative Environment Open Inquiry Learning Course Activities Course Projects Individual Task, Assignment, & Project Management Conveying Information to Others Real World Context Conceptual Learning & Application Working with Data Using Course Resources Troubleshooting Reasoning & Sensemaking Persevering Through Course Solving Problems & Finding Solutions Determining Unknown Writing Notebook & Reports General Lab Experience Other Key Color Percentage (%) 81 - 100 61 - 80 41 - 60 21 - 40 0 - 20 404 Table 6.69. Frequency of subcodes within the Collaborative Learning Environment course element category and perceived association with supporting development of specific EDCs (n). Course Elements within the Collaborative Environment Communication& Collaboration with Team Communication& Collaboration with TA Additional Collaborative Elements Not Related to One Group in Specific Subcodes Delegating & Coordinating Learning to Work with Others & Be a Team Player Listening & Sharing Ideas Recognizing Importance of Communication in Teamwork Asking Questions & For Help Building Relationships Solving Problems & Finding Solutions as a Team General Communication & Collaboration with Team Asking Questions & Receiving Guidance General Communication & Collaboration with TA Recognizing Importance of Communicating with TA Asking Questions and Receiving Help Skill Perceived as Developed in Course(s) Communication Skills (n = 318) 143 Teamwork Skills (n = 351) 122 Prioritization & Time Management (n = 230) 61 Work Ethic (n = 82) 6 6 — 1 3 — 2 4 1 — — — 11 2 — 1 — 1 2 2 — — — 43 34 32 23 1 24 92 43 14 7 4 34 76 128 9 9 3 5 1 — — 1 405 Table 6.70. Frequency of subcodes within the Open Inquiry Learning, Working with Data, Conceptual Learning & Application, and Using Course Resources course element codes and their association with development of particular EDCs (n). Skill Perceived as Developed in Course(s) Problem-solving & Critical Thinking (n = 288) 58 Technical Skills (n = 41) N/A Field-specific Knowledge (n = 27) 6 12 11 4 61 79 5 1 2 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 6 1 11 1 — — N/A N/A N/A N/A N/A 12 3 N/A N/A N/A Course Element Open Inquiry Learning Working with Data Conceptual Learning & Application Using Course Resources Subcodes Designing & Planning Experiments Less Guidance Provided in Course Investigating Driving Problem & Finding Resources Finding Solutions to Proposed Problem Data Collection & Record Keeping Analyzing, Interpreting, & Displaying Data Working with Large Amounts of Data Working With & Using Multiple Types of Data Using Data in Lab Reports & Presentations Learning New Concepts & Information Applying Knowledge Course Instrument & Tools General Use of Course Resources Computer, Software, & Application Use (e.g., LoggerPro) *N/A is used to denote where further exploration of codes within a theme were not applicable to a particular skill, whereas an Em dash (—) is used to denote absence of code application where a theme was further investigated for a particular skill. 406 A.VI.29. Influence of Online Learning on Skill Development within Sp20 and Su20 Survey Administration The influence of online learning on skill development was investigated through looking into participant experiences surrounding remote learning either hindering (Table 6.71) or supporting (Table 6.72) development. This section explores the codes relevant to Sp20 and Su20 survey administration for the skills communication, teamwork, problem-solving & critical thinking and prioritization & time management. Only communication skills were investigated when looking into how online learning supported skill development due to this being the only skill in which there was a somewhat sizeable sample that found remote learning to aid in growth of these skills. Those identified as emergent themes are discussed in full within the main chapter. All corresponding tables present on pages 408 – 409. 407 Table 6.71. Perception of how virtual learning in the laboratory hindered skill development for Sp20 and Su20 Semesters (n). Reason Online Learning Hindered Development Harder to Communicate/ Collaborate Lack of Hands-on/In- person Learning Lack of Communication/ Collaboration More Individual Learning Experience Lack of Motivation Distracting Home Environment Lack of Learning Lack of Interest, Enjoyment, & Engagement More Difficult/Challenging Feeling at a Disadvantage Easier to Procrastinate Lack of Individual Tasks Enjoys Working Individually Having More Time & Less Strict Deadlines Skill Communication Skills (n = 26) 9 Teamwork Skills (n = 46) 10 Problem- solving & Critical Thinking (n = 16) — Prioritization & Time Management (n = 15) 1 11 — — 1 — 3 — — — — — — — 4 2 — 1 2 — — — — 2 — — — 2 16 8 — — — — — 1 — — — — 2 34 14 — — — 1 1 — — 1 1 1 408 Table 6.72. Perception of how virtual learning in the laboratory supported development of communication skills for Sp20 and Su20 participants, n, (%). Reason Online Learning Supported Development Learned to Adapt/Be Flexible Harder to Communicate/ Collaborate Importance of Teamwork/Communication Online Skill Communication Skills (n = 26) 7 3 2 409 A.VI.30. Skills Necessary, Skill Proficiency & Desire for Further Development Table 6.73. Skills participants perceived as necessary for their career, skills they believe they are proficient in, and skills they would like to develop throughout their college career in Fa21 IRA survey responses (n = 200), n, (%). Main Skill Set Interpersonal Skills Intrapersonal Skills Problem-solving & Critical Thinking Prioritization & Time Management Occupation Specific Skills Education & Learning Skills Other Skills No Skills Listed Identified as Career Competency 124 (62) 82 (41) 43 (22) 75 (38) 96 (48) 46 (23) 27 (14) 2 (1) # of Respondents (%) Proficiency in Skill Set* 82 (66) 42 (51) 17 (40) 39 (52) 33 (34) 10 (22) 12 (44) — Would Like to Develop Skill Set* 66 (53) 39 (48) 21 (49) 33 (44) 80 (83) 24 (52) 12 (44) — *Percent is taken from the number of students within category out of the number of students who identified the skill set as valuable to their career. 410 A.VI.31. Areas in Which Students Have Learned of Skills Outside of GCL1/GCL2 Table 6.74. Fa21 FRA overall multiple-choice results asking participants what recent educational setting they had learned about EDCs in, n, (%). Response Yes, at MSU (excluding the general chemistry laboratory courses) Yes, in high school Yes, both in high school and at MSU (excluding the general chemistry laboratory courses) No I'm not sure Total (n) n (%) 126 (12) 138 (13) 588 (55) 77 (7) 134 (13) 1063 Table 6.75. Fa21 FRA multiple-choice results asking participants what recent educational setting they had learned about EDCs in by career goal, n, (%). Response Yes, at MSU (excluding the general chemistry laboratory courses) Yes, in high school Yes, both in high school and at MSU (excluding the general chemistry laboratory courses) No I'm not sure Total (n) Health & Medical Professions Career Goals Engineering & Associated Subspecialties Other Careers Total (n) 54 (10) 72 (13) 327 (59) 43 (8) 60 (11) 556 23 (14) 17 (10) 49 (14) 49 (14) 126 138 101 (60) 7 (4) 20 (12) 168 160 (47) 27 (8) 54 (16) 339 588 77 134 1063 Table 6.76. Fa21 FRA multiple-choice results asking participants what recent educational setting they had learned about EDCs in by class standing, n, (%). Response Yes, at MSU (excluding the general chemistry laboratory courses) Yes, in high school Yes, both in high school and at MSU (excluding the general chemistry laboratory courses) No I'm not sure Total (n) Class Standing Lower Classman Upper Classman Total (n) 97 (11) 117 (14) 464 (55) 61 (7) 108 (13) 847 29 (13) 21 (10) 124 (57) 16 (7) 26 (12) 216 126 138 588 77 134 1063 411 Table 6.77. Fa21 FRA multiple-choice results asking participants what recent educational setting they had learned about EDCs in by first-generation status, n, (%). Response Yes, at MSU (excluding the general chemistry laboratory courses) Yes, in high school Yes, both in high school and at MSU (excluding the general chemistry laboratory courses) No I'm not sure Total (n) First Generation Status Continuing Generation First Generation 96 (12) 99 (12) 470 (57) 63 (8) 103 (12) 831 30 (13) 39 (27) 118 (51) 14 (6) 31 (13) 232 Total (n) 126 138 588 77 134 1063 412 A.VI.32. Desire to Learn More About EDCs in College Courses Table 6.78. Fa21 FRA overall multiple-choice results asking participants if they would like more opportunities to learn about EDCs in college courses, n, (%). Response Yes No n (%) 915 (86) 148 (14) 1,063 Total Table 6.79. Fa21 FRA overall multiple-choice results asking participants if they would like more opportunities to learn about EDCs in college courses by career goal, n, (%). Response Yes No Total (n) Health & Medical Professions 479 (86) 77 (14) 556 Career Goals Engineering & Associated Subspecialties 148 (88) 20 (12) 168 Other Careers 288 (85) 51 (15) 339 Total (n) 915 148 1063 Table 6.80. Fa21 FRA overall multiple-choice results asking participants if they would like more opportunities to learn about EDCs in college courses by class standing, n, (%). Response Yes No Total (n) Lower Classman 743 (88) 104 (80) 847 Upper Classman 172 (12) 44 (20) 216 Total (n) 915 148 1063 Class Standing Table 6.81. Fa21 FRA overall multiple-choice results asking participants if they would like more opportunities to learn about EDCs in college courses by first-generation status, n, (%). Response Yes No Total (n) Continuing Generation 722 (87) 109 (13) 831 First Generation 193 (83) 39 (17) 232 Total (n) 915 148 1063 First Generation Status 413 A.VI.33. Student Perceived Value of Skills. Table 6.82 - 6.83 display the outcomes of two Fa20 survey questions inquiring whether participants viewed the six EDCs provided to participants as valuable to their career goal, as well as others. Table 6.82. Fa20 overall multiple-choice results for if EDCs are valuable towards future career goal, n, (%). Response Yes No Total (n) n (%) 1,111 (99.6) 4 (0.4) 1,115 Table 6.83. Fa20 overall multiple-choice results for if EDCs are valuable to a variety of careers, represented by number and the related percent of participants, n, (%). Response Yes No Total (n) n (%) 1,112 (99.7) 3 (0.3) 1,115 While asking students areas in which EDCs were perceived as important beyond their career was not a question presented in survey prompts, outside of the multiple-choice question present to Fa20 participants explored above, some Fa20 and Sp21 students included further unprompted commentary of where they place value of these skills in their lives. Focusing on and combining results from the six EDCs provided to Fa20 participants (n = 200) and the four EDCs given to Sp21 participants (including additional mentions that could be categorized within these skills, n = 214), the results present in Table 6.84 were compiled. These are qualitatively coded results in addition to the Fa20 multiple-choice questions regarding if skills were perceived as valuable to a variety of career goals explored above. Although these results are very small portions of the overall survey sample, it shows that students are thinking about how these skills are valuable in places that reach outside of the confines of their career or development in the courses without prompting and warrants further exploration. 414 Table 6.84. Value of skills beyond career goal (n). Employer-desired Competency Communication Skills (n = 414) 7 4 Teamwork Skills (n = 414) 8 2 Problem- solving & Critical Thinking (n = 414) — 2 Prioritization & Time Management (n = 414) 2 1 Work Ethic (n = 200)* — 1 5 8 3 6 1 Places in Which Skill is Perceived to be Valuable Any Job/Career College Career & Campus Life General Life Skill *Skill was only present in Fa20 survey prompts. 415 A.VI.34. Building Student Awareness of EDCs Table 6.85 displays thematic analysis of an Sp21 survey question that prompted students to discuss how they were able to build awareness of EDCs by being introduced to them in scenario documents throughout the course. These themes and evidence provided are discussed in the main chapter. Table 6.85. Qualitative themes exploring how students’ built awareness of EDCs in Sp21 survey administration (n = 214), n, (%). Theme Spoke of Applying, Developing, or Practicing Skills Learned About Skills & Recognized Their Importance Spoke of Learning How to Work with Others in the Course Did Not Expand on Question Spoke of How Awareness was Built Prior to Course/Learned Nothing from Being Introduced to Skills in Course Number of Participants, n (%) 118 (55) 86 (40) 82 (38) 7 (3) 1 (0.5%) 416 A.VI.35. Value of Developing EDCs to Career Preparation in Course Table 6.86. Sp21 overall multiple-choice results asking students if they perceived opportunities for development of EDCs to be valuable towards career preparation, n, (%). Found Opportunity to Develop Skills Valuable Yes! No, I did not develop any transferable skills. No, these skills will not contribute to success in my future career n (%) 205 (96) 4 (2) 5 (2) 214 Total (n) Table 6.87. Sp21 multiple-choice results asking students if they perceived opportunities for development of EDCs to be valuable towards career preparation by class standing, n, (%). Response Yes! No, I did not develop any transferable skills. No, these skills will not contribute to success in my future career Total (n) Lower Classman 170 (96) 4 (2) Class Standing Upper Classman 35 (97) 0 (0) 4 (2) 178 1 (3) 36 Total (n) 205 4 5 214 Table 6.88. Sp21 multiple-choice results asking students if they perceived opportunities for development of EDCs to be valuable towards career preparation by first generation student status, n, (%). Response Yes! No, I did not develop any transferable skills. No, these skills will not contribute to success in my future career Total (n) Continuing Generation 150 (97) 2 (1) First Gen Status First Generation 55 (92) 3 (5) 2 (1) 154 2 (3) 60 Total (n) 205 4 5 214 417 A.VI.36. Ways in Which Opportunities to Develop EDCs were Valuable to Career Preparation Table 6.89 contains themes that emerged from Sp21 survey questioning that prompted students to elaborate on how the opportunity to develop EDCs within the course was valuable to career preparation. Table 6.90 displays specific skills mentioned within these survey responses. Table 6.89.Qualitative Themes Exploring How Opportunities to Develop Skills were Valuable to Career Preparation, represented by number and the related percent of participants (n = 214). Theme Spoke of Applying, Developing, or Practicing Skills Spoke of Learning How to Work with Others in the Course Learned About Skills & Recognized Their Importance Did Not Expand on Question Number of Participants, n (%) 185 (90) 64 (31) 5 (2) 4 (2) Table 6.90. Qualitative Themes Surrounding Specific Skills Mentioned When Students’ Spoke of How Opportunities to Develop Skills were Valuable to Career Preparation, represent by number of participants (n = 214). Skill Teamwork Skills Did Not Expand on Specific Skill Communication Skills Problem-solving & Critical Thinking Additional Interpersonal Skills Time Management & Organization Technical Skills* Field-specific Knowledge* Other Skills Work Ethic Skills Intrapersonal Skills # of Participants Who Mentioned Skill (n) 86 77 60 35 16 16 8 6 4 3 3 *Although technical skills and field-specific knowledge are typically referenced under occupation-specific skills throughout this study, because we did not ask participants to specify skills they perceived to be valuable career competencies, during the Sp21 semester, they were not categorized in this manner. 418 REFERENCES Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society, 57(1), 289–300. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences Second Edition. Lawrence Erlbaum Associates. Ghasemi, A., & Zahediasl, S. (2012). Normality Tests for Statistical Analysis: A Guide for Non- statisticians. International Journal of Endocrinology and Metabolism, 10(2), 486–489. https://doi.org/10.5812/ijem.3505 Gravetter, F. J., & Wallnau, L. B. (2017). Statistics for the Behavioral Sciences. In Statistics for the Behavioral Sciences . CENGAGE Learning. Jaccard, J., & Becker, M. A. (1990). Nonparametric Statistical Tests . In Statistics for the Behavioral Sciences (2nd ed., pp. 398–399). Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4(NOV). https://doi.org/10.3389/fpsyg.2013.00863 Lavrakas, P. (2008). Encyclopedia of Survey Research Methods. In Encyclopedia of Survey Research Methods. Sage Publications, Inc. https://doi.org/10.4135/9781412963947 Lewis-Beck, M., Bryman, A., & Futing Liao, T. (2012). The SAGE Encyclopedia of Social Science Research Methods. In The SAGE Encyclopedia of Social Science Research Methods. Sage Publications, Inc. https://doi.org/10.4135/9781412950589 Lumley, T., Diehr, P., Emerson, S., & Chen, L. (2002). The Importance of the Normality Assumption in Large Public Health Data Sets. The Annual Review of Public Health, 23, 151–169. https://doi.org/10.1146/annurev.publheath.23.100901.140546 Michigan State University: Office of Admissions. (n.d.). ACT/SAT Optional. Retrieved August 28, 2023, from https://admissions.msu.edu/apply/first-year/act-sat-test-optional Reid, H. M. (2022). Finding Differences With Nominal Data—II : The Chi-Square Test of Independence. In Introduction to Statistics: Fundamental Concepts and Procedures of Data Analysis. SAGE Publications, Inc. https://doi.org/10.4135/9781071878927 Salkind, N. (2012). Cohen’s d Statistic. In Encyclopedia of Research Design. SAGE Publications, Inc. https://doi.org/10.4135/9781412961288.n58 Singh, K. (2015). Sampling and Sample Size Estimation. In Quantitative Social Research Methods. SAGE Publications India Pvt Ltd. https://doi.org/10.4135/9789351507741 419 Smyth, J. D., Christian, L. M., & Dillman, D. A. (2008). Does “Yes or No” on the Telephone Mean the Same as “Check-all-that-apply” on the Web? Public Opinion Quarterly, 72(1), 103–113. https://doi.org/10.1093/poq/nfn005 420 CHAPTER VII: BEYOND THE GRADUATE DEGREE: 21st-CENTURY SKILL DEVELOPMENT AS A GRADUATE TEACHING ASSISTANT FOR PROJECT-BASED GENERAL CHEMISTRY LABORATORY COURSES Introduction Graduate STEM education in the 21st century plays a critical role in developing the technically trained workforce needed to maintain the nation’s economic competitiveness and leadership in innovation to solve critical problems (National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018; Wendler et al., 2012). Master’s and doctorate degree holders typically seek careers in academia, industry, or government (ACS Presidential Commission, 2012; Ganapati & Ritchie, 2021; Wendler et al., 2012). However, students are graduating from graduate programs at a greater rate than the supply of available positions (Julie Gould, 2019) leading to challenges for some in finding full-time employment (ACS Presidential Commission, 2012), particularly in academia (Larson et al., 2014). This can pose an issue as the number of students receiving graduate degrees has been trending upwards (National Center for Science and Engineering Statistics Directorate for Social Behavioral and Economic Sciences & National Science Foundation, 2022). Graduate degrees in chemical sciences alone grew by approximately 31% for Master’s degrees and 39% for Ph.D. degrees, from 2000 to 2015 (National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018). Although obtaining a graduate degree can come with many benefits such as job stability and higher earnings (National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018; U.S. Bureau of Labor Statistics, 2022; Wendler et al., 2012), past and current research highlights a divide between the needs of students and employers and the structure of the typical current chemistry graduate program curricula (ACS Presidential Commission, 2012; Ashby & Maher, 2019). Among these concerns are a lack of adequate career preparation and opportunities to develop valuable career competencies, known as 21st-century skills. With the time and effort dedicated toward achieving a graduate degree (ACS Presidential Commission, 2012), it is important that graduate education provides students with the skills needed to be successful in the workforce. 421 Career Preparation in Graduate Education Chemistry graduate education has been criticized for outdated curricula that does not adequately prepare students for the workforce (ACS Presidential Commission, 2012; Ashby & Maher, 2019; Caserio et al., 2004; Raber, 2000), particularly those that extend beyond academia. The 2019 ACS Graduate Student Survey found that chemistry graduate students primarily intend to seek careers in academia (38%) or industry (41%)(Kuniyoshi et al., 2021). While many graduate students expressed interest in academic careers at the start of their graduate studies, there was a notable shift towards careers in industry, government, entrepreneurship, and the non-profit sector in the later stages of their graduate education. However, graduate programs may not always create an environment that supports students in preparing for careers outside of academia (ACS Presidential Commission, 2012; Ganapati & Ritchie, 2021). Busby & Harshman found that chemistry graduate students interested in careers in industry felt that their graduate program generally lacked opportunities to explore options beyond academia, leading to frustration (Busby & Harshman, 2021). Given the variety of jobs that graduates can pursue, graduate education needs to prepare students for these diverse occupations (National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018). Beyond adapting to a more diverse workforce, graduate degree holders must acquire and demonstrate proficiency in valuable 21st-century skills such as communication, teamwork, problem-solving, critical thinking, and leadership (ACS Presidential Commission, 2012; Cui & Harshman, 2020; National Research Council, Division on Earth and Life Sciences, et al., 2012). Yet, graduates are reported to be deficient in communication, teamwork, leadership, and project management skills (ACS Presidential Commission, 2012; Caserio et al., 2004; National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018; Wendler et al., 2012). These skills have been recognized for years as necessary for advancing graduate education, emphasizing how this issue persists. In 2008, the American Chemical Society’s Committee on Professional Training (ACS CPT) recognized the importance of soft skill development, advocating for graduate curricula to include opportunities for development of communication, teamwork, and critical thinking (Shulman, 2008). Further, the 2012 ACS commission on Advancing Graduate Education in the Chemical Sciences was charged with defining the purpose 422 of graduate education; one of which was to equip students with the knowledge and skills needed for modern-day careers (ACS Presidential Commission, 2012). These desired skills are broadly defined and hard to measure creating challenges for operationalizing them within graduate curricula (National Academies, 2011). Furthermore, how these skills are used in the workplace can differ based on career path (ACS Presidential Commission, 2012; Cui & Harshman, 2020) and there is little evidence to suggest transference of these skills from one context to another (Laker & Powell, 2011; National Academies, 2011). Cui & Harshman interviewed PhD level chemists employed in academic, industry, and government positions, across a variety of specialties (analytical, biochemistry, physical, inorganic, and organic), to discern the skills needed to be successful in these career paths (Cui & Harshman, 2020). While nearly all identified technical skills as relevant to their career, soft skills such as communication, teamwork, management, and problem-solving emerged as important. Unsurprisingly, the details of how participants described skills depended on their employment setting and role. For example, while participant responses about management skills centered on allocating people and resources regardless of occupation, research faculty discussed using these skills when supervising students, teaching faculty described these skills as training teaching assistants, and industry professionals elaborated on these skills in overseeing projects (e.g., budget and time management). From the findings of this study, it was recommended that graduate education needs to assess how it can fit the needs of graduate students and better prepare them for their future career goals by either incorporating a breadth of soft skill development into the curriculum that can be broadly applicable across many career goals or individualizing professional development based on a graduates intended career path. When addressing who is responsible for preparing graduate students for success in the workforce, there is consensus that all stakeholders must play their part, including graduate students themselves, faculty through the curriculum and mentorship in research, and employers through increased collaborations with graduate program developers (Caserio et al., 2004; Ganapati & Ritchie, 2021; National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018; Wendler et al., 2012). One aspect of the graduate curriculum that may provide opportunities for professional development is the teaching assistantship. However, this is an element of graduate education that is often overlooked and likely underutilized (Ashby & Maher, 2019) as an area of 423 the curriculum that could potentially be leveraged for career preparation and development of key 21st-century skills. Teaching Assignments & Professional Development Graduate teaching assistants (GTAs) are regarded as key players in undergraduate chemistry education (Lang et al., 2020) through facilitating recitation or laboratory sections (Fantone et al., 2023) and are positions filled by many graduate students primarily based on available funding (Kuniyoshi et al., 2021; National Research Council, Division on Earth and Life Sciences, et al., 2012) or in fulfillment of curricular requirements as seen within the institution in this study (Department of Chemistry at Michigan State University, 2023). Unfortunately, the GTA role and its contribution to graduate education have not always been viewed favorably, [. . .] only rarely have [GTAs] been considered actual partners in instruction or has attention been paid to the impact that teaching experiences may have on their professional development. Seldom has TAing been actively seen as integral part of chemistry graduate education, and often it seems to be reduced to a necessary evil to secure graduate researchers financial support (Sandi- Urena & Gatlin, 2013). Other contributors to negative perceptions of the GTA role held by faculty and graduate students have been documented in the literature. They include uncertainty regarding whether GTAs should be primarily considered students or employees (Duffy & Cooper, 2020), a lack of GTA autonomy in decisions regarding teaching practices (Che et al., 2023), an absence of professional development opportunities to enhance teaching skills (Denise Kendall & Schussler, 2012), and concerns that GTAs are not providing adequate instruction to undergraduate students (Bruck et al., 2010). There is, however, some evidence that serving as a GTA contributes positively to graduate students’ development as scientists and as a result supports their research. Feldon et al. analyzed written research proposals from 95 graduate students. They found that students who had held both research and teaching assistantships demonstrated stronger research skills, particularly in generating testable hypotheses and designing experiments, than those who had only served as research assistants (Feldon et al., 2011). This study suggests that teaching and research should not be viewed as separate unrelated entities in graduate education and that teaching experiences 424 should not be overlooked in preparing students for the workforce. Teaching can also help graduate students build a deeper understanding of content and refine their skills in explaining technical topics to nonexperts which is valuable for careers in industry (National Research Council, Division on Earth and Life Sciences, et al., 2012). Professional development can support a graduate student’s teaching experience. Several studies have investigated the relationship between GTA training and self-reported development of teaching or professional skills in the chemical sciences. Purdue University’s Department of Chemistry required new teaching assistants to participate in an eight-week teaching seminar that covered topics such as how students learn and active learning and incorporated peer observations between pairings of experienced and novice GTAs accompanied by feedback and reflection (Lang et al., 2020). Participants reported finding the seminar as beneficial to their growth as instructors, and experienced GTAs reported development of mentorship and teaching skills. In the Chemistry Instructional Coaching program at University of Michigan, GTAs created and implemented lesson plans under the guidance of an experienced GTA mentor and reflected on their teaching practices (Fantone et al., 2023). Mentees reported gains in teaching skills, while mentors reported developing mentoring, teaching, problem-solving, and collaboration skills. These professional development efforts focused primarily on activities directly related to supporting graduate students in their instructional role rather than intentionally trying to leverage these experiences to develop professional skills that are broadly applicable to the range of careers pursued by chemistry PhDs. Wheeler et al. explored GTA motivations for teaching through an expectancy value framework (Wheeler et al., 2019). They found that no participants reported the subjective value of utility, defined as having perceived value to future pursuits such as a participant’s career goal, in their motivations for teaching, even for those intending to pursue a career in academia. These studies indicate that there may be a disconnect between the activities GTAs perform and their perception of future value. Project-based learning (PjBL) has been reported to aid students in developing valuable skills that are highly desirable in the 21st-century job market such as communication, teamwork, problem-solving, and critical thinking (Cooper & Kerns, 2006; Jollands et al., 2012; Wurdinger, 2016). This type of learning environment employs a student-centered curriculum where students plan experiments centered on a real-world problems, collect and analyze data, and report results, with instructors serving as guides (Krajcik & Shin, 2014). While 21st-century skill development 425 in project-based learning is generally reported as a learning outcome for students (Bell, 2010; Bender, 2012; Jollands et al., 2012; Wurdinger, 2016), this qualitative phenomenological study investigated graduate student perceptions of development of valuable career competencies while serving as a GTA. During interviews, GTAs were also asked to discuss the skills that they thought their students were building to examine alignment between their perceptions and those of students (Chapter IV), because if development of specific 21st-century skills is a desired learning outcome, understanding these two perspectives is important to achieving these outcomes (National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, et al., 2018). Research Questions This study aimed to address the following research questions in the context of project-based general chemistry laboratory courses: 1. (RQ1): What career-relevant skills do graduate teaching assistants (GTAs) believe they are developing in these courses? 2. (RQ2): How do graduate students perceive they are developing skills needed for their career as a teaching assistant? 3. (RQ3): What skills do graduate teaching assistants (GTAs) believe their students are developing? 4. (RQ4) How do graduate teaching assistants (GTAs) and student perceptions of student skill development align? Theoretical Framework This study is guided by the theoretical framework of social constructivism. Attributed primarily to Lev Vygotsky, and later Jerome Bruner, social constructivism posits that knowledge is constructed and influenced through active engagement with one’s social environment (Murphy, 1997; Rannikmäe et al., 2020; Vasileva & Balyasnikova, 2019; Vygotsky, 1978). This theory of knowledge development has been recognized as important in the working world (Kraiger, 2008), where this framework has been used in development and assessment of employee training programs (Cooper et al., 2006). Further, social engagements through collaboration can facilitate the learning and transfer of key 21st-century skills (Jackson, 2016). Social constructivism was adopted as the theoretical framework for the current study because collaborative interactions within student teams and between student teams and GTAs are central to the learning 426 environment in the project-based general chemistry labs where this research was conducted (Carmel et al., 2017). Methods Methodological Framework Data collection and analysis focused on use of qualitative methods, through conducting interviews and engaging in thematic analysis (Creswell, 2013). Qualitative analysis was rooted in a transcendental phenomenological methodological approach, which aims to investigate a phenomenon (e.g., career preparation and skill development as an instructor for the general chemistry laboratory courses) through showcasing participant experiences surrounding the “what” and “how” (e.g., what skills were developed and how participants experienced development)(Creswell, 2013). Researchers identified personal experiences or biases in relation to the phenomenon under study in order to set them aside to observe the data with an unburdened or new perspective (Moustakas, 1994). This act of acknowledging one’s place within the study is known as epoché and is addressed in the researcher bias statement below. Researcher Bias Statement To provide transparency, it is imperative to acknowledge each author’s experiences in the project-based general chemistry laboratory courses investigated. The lead researcher who led data collection and analysis, BE, served as a GTA for four semesters directly facilitating the work of student teams. For one semester, BE was appointed as a senior GTA responsible for supervising undergraduate learning assistants (ULAs) and assisting course coordinators with TA training. These experiences led to BE to develop their own perception of what it means to be a GTA for these courses. To address the influence of potential bias in the results of this study, multiple measures were taken. The first was the inclusion of multiple coders and researchers to ensure reliability. Author LAP (Lynmarie A. Posey) was not directly involved with these courses and aided in analysis and interpretation of results. While author PP-L (Priya Patterson-Lee) was enrolled in these courses as an undergraduate student and acted as the second coder, any potential biases were addressed during consistently scheduled meetings to make sure the meanings prescribed by participants were independent from the author’s own experiences. Because BE held a senior GTA position during the semester in which most interviews were conducted, interviews began with guaranteeing participant anonymity and explicitly stating that participation would have no 427 impact on their GTA appointment to create a safe environment in which GTA’s could share their unfiltered thoughts. Course Contexts & TA Training The general chemistry laboratory courses from which GTAs were sampled for this study follows a cooperative project-based learning pedagogy (Carmel et al., 2017; Cooper, 2012). Teams of 3 to 4 students work together for the entire semester on multi-week projects. Each project starts with a scenario outlining a real-world problem (e.g., determining the best method for dissolution of kidney stones) and tasks teams with finding a solution to that problem guided by scaffolded planning questions. They are encouraged to ask questions of their GTA or neighboring teams if they need help. Teams then carry out their planned procedures, collect and analyze data, and then present their methods and make claims supported by experimental evidence and reasoning in a written report, poster, or oral presentation. This format requires GTAs to facilitate student learning by answering questions with guiding questions rather than providing direct instruction. GTAs must be prepared to help students navigate the project-based learning environment and assist students in working through team dynamics. Prior to starting their GTA responsibilities, all new graduate students were required to attend a university-sponsored one- to two-day professional development workshop covering the role of GTAs in undergraduate student success, pedagogical practices, and general codes of conduct and ethics in the classroom (The Graduate School at Michigan State University, 2024). The chemistry department conducted a week-long orientation with approximately six hours dedicated to general GTA training (Duffy & Cooper, 2020). This training covered general teaching responsibilities, preparation for the first day of teaching, classroom management, how people learn, and best pedagogical practices. At the beginning of each semester, GTAs for the general chemistry laboratory 1 (GCL1) and 2 (GCL2) courses participated in two days of course-specific training focused on their responsibilities as an instructor, the course structure, and the goals of project-based learning. During weekly meetings, GTAs worked in groups to plan and carry out project experiments for the upcoming week, practiced grading student work, and discussed any challenges from the previous week and/or those anticipated for the upcoming week class. When courses were taught online during the COVID pandemic (second half of Spring 2020, Fall 2020, and Spring 2021), weekly training familiarized GTAs with online resources developed to preserve the key elements 428 of the project-based curriculum. While students could not conduct experiments, they were still able to plan investigations. Videos were provided to permit students to make observations, while apps allowed students to simulate measurement data using input parameters from their experimental plans. Prior to Fall 2021, GTAs were individually responsible for overseeing three sections of 20-24 students. Subsequently, two GTAs were assigned to collaboratively facilitate the work in three sections of 40 students. Many GTAs sampled in this study either collaboratively worked with a co-GTA (n = 8) or were independently in charge of their own sections (n = 7) at least once. Data Collection This research was conducted following Institutional Review Board (IRB) review and approval. Gift cards, in the amount of $20, were offered as an incentive to participate. All participants were notified of their rights as a participant and were given a brief outline of the study. Signed consent was obtained prior to conducting interviews. Invitations to participate in interviews were initially extended by email to n = 22 graduate students who had taught the general chemistry laboratory courses (either GCL1/GCL2, or a mix of both) for more than two semesters and taught in person at least once during the preceding 5- year period. Unfortunately, this strategy for recruitment was met with a low response rate of only 14% (n = 3), resulting in a more purposeful selection of participants by reaching out to one individual directly. After the initial group of four interviews were conducted, n = 31 GTAs who were currently appointed to GCL1 or GCL2, after in-person laboratories resumed, were informed of the opportunity to participate in interviews during a weekly GTA meeting. This was followed by an email asking those who wanted to volunteer to complete an online survey indicating their availability. The response rate from this second round of recruitment was 26% (n = 8). A total of n = 12 GTAs were interviewed via Zoom in the initial and second phase of data collection. Interview Protocol Interviews followed a semi-structured format, in which the interviewer followed a set of predetermined questions (APPENDIX A.VII.1) but asked follow-up questions to clarify participant responses (Herrington & Daubenmire, 2014). Questions explored two areas: a) GTA perceptions of career-relevant skills developed by undergraduate students in the general chemistry laboratory courses, and b) GTA perceptions of skills they developed as an instructor in 429 the general chemistry laboratory courses that would be beneficial for their career. To guide discussion and further questioning, participants were provided with a link to a shared google doc, where they could enter answers when asked about skills developed by students or themselves in GCL1 or GCL2 (e.g., What skills do you believe students are developing in the general chemistry laboratory course that will be beneficial towards success in the workforce? or Focusing on your own experience as a general chemistry laboratory TA, what skills did you develop while being a TA for CEM [161/162]?). This approach was implemented during the first set of interviews, and as a result, 3 participants did not enter responses in the Google documents. During 3 interviews, a visual aid containing a list of six previously defined employer-desired competencies (EDCs)(A.VII.1) was shared to facilitate further questioning. Participants were also asked to share their thoughts on how GTA training could better foster career preparation or skill development. Qualitative Analysis of Interviews Audio recordings, from interviews conducted on Zoom, were transcribed verbatim using OTTER.ai. All transcripts and documentation were deidentified with non-binary aliases prior to engaging in data analysis. Interviews were analyzed using inductive thematic analysis, where codes were generated based on emergent themes within participant responses (Nowell et al., 2017). First, segments of interest were identified based on the research questions driving this study, and initial codes were generated that aimed to capture the essence of the participant’s experience. After identifying initial codes, similar codes were merged to create broader categories (e.g., instances related to working and coordinating with a co-GTA were condensed into the broader category of Collaborating with Co-GTAs), resulting in the first coding scheme. Using selected segments of interview transcripts, this coding scheme was applied by two independent coders and compared to determine inter-rater reliability (IRR). Iterative revisions of the coding scheme continued until Brennan Prediger’s Kappa (𝜅BP) values of 0.80 or greater were achieved, values recognized as good to near perfect agreement in the literature ((Landis & Koch, 1977; Mabmud, 2012; Sim & Wright, 2005). Once adequate IRR was reached, an independent coder coded all remaining transcripts. Coding was carried out using MAXQDA (VERBI Software, 2021). Quotes are used as evidentiary support of themes and those included in the results were cleaned to remove filler words (e.g., um, uh, like) for clarity. 430 Findings To begin, demographic variables for interview participants will be presented to characterize the sample. Next, we will examine GTA perceptions of the skills they developed as instructors that will be beneficial for their career goals and how serving as a GTA for GCL1 or GCL2 supported development of these skills. Development within the context of this study refers to participant experiences where they associated learning and/or applying a skill with specific tasks performed as a GTA. Finally, GTA and student perceptions of the career-related skills developed by students in these courses will be compared qualitatively to examine the alignment between instructor and student beliefs. Since the project-based GCL1 and GCL2 employ the same pedagogical practices and the sample is small (n = 12), data are not disaggregated by course. Qualitative results presented herein are representative of a theme appearing at least once within a participant’s transcript. Participant Characteristics & Career Aspirations The sample contained a diverse representation of career goals, in which 33% (n = 4) desired careers in academia, 25% (n = 3) in industry, and 42% (n = 5) who were undecided in their post- graduation career path. Most participants, 75% (n = 9), were active GCL1 or GCL2 GTAs when the interviews were conducted, while the remaining GTAs had served as an instructor for these courses in prior semesters. Experience as a GTA in the general chemistry laboratory courses ranged from 1 – 6 semesters with an average of approximately 2 semesters. Participants also had a variety of prior teaching experiences before their graduate studies at the institution investigated in this study. Four participants served as a either a tutor, teaching assistant, or laboratory assistant while undergraduates. Four others had experience as K-12 teachers or as an instructor in non-academic tutorial centers after obtaining their undergraduate degree. Time spent in these positions ranged from 0.5 - 4 years, with an average of approximately 2 years. While graduate students at this institution, eight participants were GTAs in courses besides GCL1 and GCL2 for 1 – 6 semesters. Career Preparation & Skill Development GTA participants reported that they developed a variety of skills as an instructor for the general chemistry laboratory courses that would be beneficial to their future careers. These skills were grouped into seven broader skill sets that included interpersonal (e.g., communication, leadership), intrapersonal (e.g., patience, positive attitude), teaching skills & techniques (e.g., 431 grading skills), time management (e.g., punctuality), critical thinking, technical (e.g., laboratory skills & techniques), and other (e.g., decision making) skills. Details on the consolidation of subskills into the broader skill set categories can be found in Appendix (A.VII.2). One participant did not provide information regarding specific skills that were developed as a GTA for these courses but provided insight into other areas explored in this chapter. Figure 7.1 displays the prevalence of the seven identified skill sets in which it can be observed that subskills comprising the interpersonal skill set were identified most frequently, with nearly all participants (n = 11, 92%) recognizing these as valuable skills that were gained as an instructor for the project-based general chemistry laboratory courses. Unpacking of interpersonal skills, revealed that participants believed they were developing communication, leadership, teamwork, and social & people skills specifically. Communication (n = 9, 82%) and leadership (n = 5, 45%) emerged as the most frequently mentioned subskills within this skill set. Communication and leadership were also mentioned by at least one participant from each career area (academia, industry, undecided). Interestingly, the other interpersonal skills of teamwork and social & people skills were only identified by participants who planned to go into industry. Our investigation of how GTAs think they have developed career-relevant skills as an instructor will focus on these two subskills. Additional details on the perceived relationships between skill development and course elements can be found in Appendix (A.VII.3). Instances of development for skills not explored below, accompanied by areas in which GTAs desired further development of career-relevant skills, can be found in Appendix. 432 ) % ( s A T e t a u d a r G f o t n e c r e P 100 90 80 70 60 50 40 30 20 10 0 Figure 7.1. Skill sets graduate TAs believed they were building as an instructor for the general chemistry laboratory courses (n = 12). Two major themes emerged in relating GTA experiences to development of communication and leadership skills – 1) learning to convey information to students resulted in growth of communication skills (n = 8) and 2) teaching in a collaborative environment where GTAs are responsible for guiding and supervising students led to perceived development of both communication (n = 7) and leadership skills (n = 5). How these experiences contributed to skill development are explored below through the voices of GTAs. Theme 1: Conveying information to others led to development of communication skills. When GTAs discussed development of communication skills, they predominantly focused on aspects of verbal communication, including adapting communication based on the audience, presentation and public speaking skills, and answering questions. GTAs reflected on how they had to recognize that students enter the course with different knowledge and skills that required revising explanations to meet student needs. 433 [. . .] this lab is built, is made up of students from different areas and so they all think differently, they all come from different, other courses that expect things differently [. . .] I have to kind of convey this to each and every student in a different way. [. . .] I just have to take the time and that patience and be like “Hmmm, how would you understand this?” (Hickory, Communication Skills) [. . .] teaching [GCL1] [. . .] I was able to notice the difference among the students. Like, one, there are some students that they [. . .] already know stuff, right? And then there, there's some students that [don’t] know in a group [. . .] So, for some students, I had to sometimes go over things repeatedly to make them understand [. . .] And then I had to [. . .] balance the group. (Kola, Communication Skills) A different perspective was offered by Pistachio, who spoke of being aware of how they communicated with students from a management perspective and learning how to converse across various levels of a hierarchical work structure. [. . .] I have to learn how to communicate more professionally with [students], ‘cause they’re not my friends, they’re my students. [. . .] if I join a workplace, I think it’ll be a good skill to know, you know people that are equivalent to me, I know how to talk to them versus people I might be managing [. . .]. (Pistachio, Communication Skills) For Almond, the project-based learning environment, where students planned experiments, was perceived to contribute to communication skill development because it facilitated interactive conversations, which allowed them to build relationships. Through this they learned how to engage with a variety of students, an experience that they believed would be beneficial for their planned career in academia. These engaging experiences were contrasted to large-enrollment lecture courses where one-on-one interactions with students were more limited. [. . .] communication with various types of students, would, I’d say be the biggest thing I got from, out of that experience. I had a wide range of students with different skill sets, different personalities, and so you can’t treat them all 434 with just, the same way. They don’t all respond to the same sort of teaching styles. So, I think this is even more so with lab, than the lecture TAing, where you get to know your students over the course of the semester. [. . .] in the lecture TA, there's so many students you have to look over and you only see them once a week in terms of recitation, most of the time, you're just in a big lecture hall. [. . .] Whereas with the lab, you actually have to sit down with the students and work them through trying to improve their experiments. (Almond, Communication Skills) One of the tactics Almond employed to facilitate communication skills and get accustomed to the variety of students they instructed was “answering questions with questions.” This approach was used during the planning of experiments to get students thinking of alternative possibilities and to guide them away from possibly deleterious methods. [. . .] a tactic that I often relied on a lot was asking what I called DM questions, which comes from Dungeon Master, […] how they answer questions is often with another question 'cause you don't want to give your players too much information. (Almond, Communication Skills) Not providing direct answers but instead guiding students to solutions has been met with a more favorable perception of GTAs and their role in student learning experiences, as reported in the literature (Cooper & Kerns, 2006). Pistachio discussed learning how to answer student questions by asking further questions to gain a better understanding of what a student meant or to prompt student thinking. Pistachio also felt they became better at providing explanations to student questions and recognized the importance of reaching a mutual understanding. [. . .] when they ask me a question I try to lead them to what the answer is. Or asking them a question back to understand what they're trying to ask me. And then learning to answer questions in a way that actually explains [. . .] the answer to them [. . .] 'cause [. . .] you might answer a question and then they have no idea what you're talking about. So, trying to answer questions in a way that the person asking can actually understand you too. [. . .] and I really do 435 enjoy asking them another question, so, they're like, "Um, okay." You know, making them think about it. (Pistachio, Communication Skills) Each lab session started with a short (~5 min.) presentation during which GTAs highlighted anything that students should be aware of as they performed their planned experiments, reviewed pertinent safety practices, and reminded students of upcoming assignments. Pine believed that being given the chance to present in the course not only built communication skills but also built a sense of connection between them and their students when contrasted to a more passive teaching assistant position. I prefer being the one to present because a lot of times when another person was presenting when the professor of record was presenting, the students would instead bypass me and go to the professor's office and ask them questions. [. . .] so with the way it's currently laid out, it means that students are much more responding to me, they actually see me as someone who's there to help them instead of seeing me as the person that is just there to make sure they're wearing their goggles. (Pine, Communication Skills) Although Chestnut felt that experiences prior to instructing the general chemistry laboratory course helped contribute to their overall development of public speaking skills, they felt that their experiences as an instructor for the general chemistry laboratory courses helped them continue to build confidence in these skills, that could further apply to their graduate career (e.g., ability to present at conferences). So, even at this stage, I have actually developed my public speaking skills in the sense that when I talk to the students, sometimes I kind of find myself building more confidence and then such that even if I go for conferences that are for grad students, I wouldn't find it difficult, because I can actually see everybody present in that conference as my students, which I'm supervising. So I talk to them basically like my students, and then I feel so free. So through that I'm developing my public speaking skills. (Chestnut, Communication Skills) 436 Kola expanded upon learning to not depend on the PowerPoint slides provided but instead rely on prior experiences teaching the course to supplement additional information and provide further explanation. [. . .] before [. . .] when I [did] the presentation it was, basically I used to read a line, or I would say something, just one sentence more about that statement or that [bullet point] in the presentation. But now, it's changed [. . .]. With the [bullet point] I can explain more in a way that from my past experience we can say "Okay, so these are the [experiences], I had problems previously. So, we can improve it." [. . .] I can just give you an example. So, we always have some kind of safety requirements at every experiment right? At first, I didn't know that why that experiment, that safety is required. So, I just read out. But, throughout the experiment, I can understand why it’s stated, even the goggles or anything, why it's needed. (Kola, Communication Skills) Becoming better at conveying information to students was also seen within Hazelnut’s experience developing communication skills, where they found that having to teach multiple sections aided them in generating clearer explanations when interacting with students. So, I taught [the] same class three times in a week. So [. . .] my last section is better than others, because my explanation is better than the first time. So, like they can have a chance to [think] about it or hear about my explanation more in detail, more like clearly. [. . .] Or so I kind of expect what they are [struggling] with [. . .] every time, so I [am] kind of prepared [with] that answer before I go to lab. So, like that process helps to improve my skill. (Hazelnut, Communication Skills) Further, Pecan believed that developing communication skills as an instructor for the course would help them in the future when presenting or working in a team environment. Well, I think teaching the course helped my communication skills. I'll have to do certain presentations and proposals probably for funding, or to communicate scientific results for my experiments or for my group. So, I 437 expect that my ability to communicate [. . .] would be helpful for me in the future in my career. Typically, we work in teams in science, you have your research group. So, communication would help you better with your team. (Pecan, Communication Skills) Having readily available resources provided by course coordinators seemed to aid communication skill development for one participant. When asked how GTA training could better prepare Hickory for career preparation and personal growth, they believed that the training provided was sufficient and instead elaborated on how the resources provided on the course instructor site helped them learn how to communicate with students by saying: I don’t even have any true critiques [. . .] because every time I look in D2L, I find my answer. Even down to how to properly communicate with students. They offer multiple times, “Well, if you don’t know what to say, you know, message us and we’ll help you out. We’ll give you these things, these referrals. (Hickory, Communication Skills) Theme 2: Guiding & supervising students led to development of communication and leadership skills. In the project-based general chemistry laboratory courses, GTAs are responsible for guiding and supervising the work of student teams, which participants associated with developing both communication and leadership skills. Many of the experiences described in the quotes found in the previous section on Theme 1 contained elements of guiding students. Participants discussed guiding students to improve upon their experimental procedures (Almond), helping students work through problems (Hazelnut), overseeing laboratory safety (Kola), asking questions as a method of advising students (Pistachio), and Pine recounting being the primary presenter as helping them to feel in a place of being there to help students instead of being a passive observer as they had experienced when being a teaching assistant for a course with an instructor of record (Pine). Additional experiences included Palm checking in on different teams and providing instruction when needed as leading to development of communication skills - a sentiment reflected in Kola’s accounts of communication skill development as well. 438 Communication skills, it's important here [. . .] And even we go to the team, each of the [teams], and then, some of the stuff we need to teach them, so in a way that improves the communication. [. . .] it's more like "Okay," you know, asking questions like, "Okay, what should you be doing?","How do you feel about this?" (Palm, Communication Skills) To adequately supervise students, Pistachio noted the importance of communication with their co-GTA to ensure that they had a coordinated message when working with students to avoid confusion. [. . .] especially with co-TAs, we kind of have to be on the same page when we get into lab so that our students aren't confused (Pistachio, Communication Skills) Specific leadership skills developed through guiding and supervising students included mentoring, management, and coordination skills. Almond and Chestnut explicitly mentioned mentoring students as they navigated the project-based laboratory environment. Almond related their experiences directly to being a professor responsible for managing a research group. [. . .] it's sort of like a trial run of being a PI, to an extent, where you're overlooking many groups, experiments, and it's your job as a professor to sort of nudge them in the right direction to what will work. And so that sort of is like training of just like running your own group of just saying, like, how to deal with students, and how to help them design their experiments to be more successful. So, I think that's another useful skill to get out of it. (Almond, Leadership Skills) Chestnut saw themselves as a direct supervisor situated between students and the laboratory coordinator. They answered questions, managed conflict, and even provided advice concerning matters outside the laboratory course (e.g., helping students find undergraduate research experiences). Like Almond, Chestnut believed these experiences were valuable to future careers as an academic professor overseeing grad students, but Chestnut also expanded on how development of these skills would have relevance when managing others in industry. 439 [. . .] developing [mentoring skills] through my interaction with the students I am supervising, because I always see myself as someone [closer] to the student than the lab coordinator [themselves]. So, if the students have a first question, have a question whatsoever, the first person they reach out to is me. [. . .] if I figured I have a group, that the team members or there is one or two team members, that is not really happy with the group or not, just cooperating with a group [. . .] I see myself as a mentor in the sense that I walk up to the student that is having issues with the group, talk to the student, and then maybe if I have to pull the student out of the group, and talk to him aside, and then bring him back, him or her back into the group or address the group generally. [. . .] Recently [another] student just walk up to me and say, he's just so confused right now about from getting a research group to join in preparation for pre- med. So, I have to also be able to stand in that place and be able to advise him [. . .] looking ahead [. . .] if I end up in academia being a lecturer at several points in time, I'm going to be having students, which I'm going to be a PI, too. And then being a PI to some students means that I should definitely be able to listen to their needs, give them advice, talk to them on what they should do. So, and then even in industry, when placed as [a] head, over a group of people, I have to be able to guide them in fulfilling the task or achieving the project goals that we have at hand. (Chestnut, Leadership Skills) Meanwhile, Pistachio felt that remaining observant to ensure student safety and making sure students were staying on task contributed to development of the leadership skill of management. I feel like, kind of a manager sometimes. I have to be watching 20 to 40 people to make sure they're not spilling anything or breaking anything/hurting themselves. So, you have to be attentive the whole lab session, make sure everyone's wearing goggles. So, I think it's really good for gaining management skills, because we have to be paying attention all the time and making sure that we're answering questions and helping people throughout the whole session. (Pistachio, Leadership Skills) 440 Kola provided a rich account of developing leadership skills and the associated subskill of coordination. By ensuring each team was progressing through the lab, managing their time wisely during experimentation, and being motivated as an instructor to help teams work better and more efficiently, these skills were developed. So, we have to make sure everyone is [. . .] on the same page or everyone is doing the same thing [. . .]. As a TA, I should come and tell them how [. . .] how working as a team can be helpful for you [. . .] so that they can work much more efficiently. [. . .]. Time management is really important [. . .] we have to tell them to how they can manage it. (Kola, Leadership Skills) While not all participants explicitly associated their role with mentorship when discussing communication and leadership skill development, each of the excerpts explored above contained elements of guiding undergraduate students. Other studies have found that GTAs in inquiry-based laboratory courses view themselves as mentors and facilitators (Sandi-Urena, Cooper, Gatlin, et al., 2011; Sandi-Urena & Gatlin, 2013). The report from an NRC workshop on chemistry graduate education notes that newly appointed faculty members desired greater exposure to mentoring of undergraduates during their graduate career (National Research Council, Division on Earth and Life Sciences, et al., 2012). The structure of the project-based laboratory courses such as those in this study, which situates GTAs in the role of mentor, may be beneficial towards preparing graduate students for future careers in academia. GTA & Student Perceptions of Skills Developed by Students During interviews, GTAs were also asked to discuss what career skills, if any, were developed by undergraduate students in their general chemistry laboratory sections. Participant responses were characterized using the coding scheme found in Appendix (A.VII.5), which consists of skill and associated subskill categories. This coding scheme was adapted from a related study that investigated student perceptions of career-relevant skills developed in the GCL1 and GCL2 courses (Chapter IV and accompanying Appendix A.IV.11). 441 ) % ( s A T G f o t n e c r e P 100 90 80 70 60 50 40 30 20 10 0 Figure 7.2. Skills that GTA participants reported were developed by undergraduate students in the project-based general chemistry laboratory courses. Figure 7.2 shows the skills that 2 or more GTA participants, in the current study, reported as being developed by students in the project-based general chemistry laboratory courses. GTA participants most frequently reported that their students built teamwork (n = 9, 75%), problem-solving & critical thinking (n = 8, 67%), technical (n = 7, 58%), and communication (n = 7, 58%) skills. The frequency of occurrence for all the skills mentioned by GTA participants are provided in Appendix A.VII.6. These skills overlap with those identified by a majority of undergraduate interview participants as both relevant to their future career goals and from their perspective developed in the general chemistry laboratory courses in a separate study (Chapter IV). Table 7.1 summarizes the number of undergraduate interview participants who discussed a particular skill as relevant to their planned career and among those participants the number who also explained how they thought they had gained this skill in the general chemistry laboratory courses. From this table (Table 7.1) we can see that undergraduate interview participants primarily identified problem-solving & critical thinking, communication, 442 teamwork, and prioritization & time management as being developed, with work ethic and technical skills being mentioned to a lesser extent. Table 7.1. Skills undergraduate interview participants identified as valuable towards future career goal and developed within the general chemistry laboratory courses. Skill Communication Skills Teamwork Skills Work Ethic Problem-solving & Critical Thinking Prioritization & Time Management Technical Skills Participants, n 53 53 53 53 53 53 Valuable for Career, n (%) 44 (81) 46 (87) 46 (87) 48 (91) Valued & Developed in General Chemistry Lab, n (%) 36 (82) 37 (80) 19 (41) 40 (83) 42 (79) 36 (68) 26 (62) 13 (36) The majority of undergraduate students and GTAs interviewed perceived that the project- based general chemistry laboratory courses supported students in development of problem- solving & critical thinking, teamwork, and communication skills. However, students and GTA instructors had different perspectives regarding prioritization & time management and technical skills. A greater percentage of undergraduate student participants pointed to building prioritization & time management skills than their GTAs. While time management is an inescapable aspect of navigating college for many students, the project-based structure of the general chemistry laboratory courses requires students to attend to management of time and prioritization of tasks. Tasks must be divided among team members and individuals must organize their work both during and outside of lab in order to complete projects within the allotted time. Conversely, GTAs were more likely to report development of technical skills as a learning outcome in these courses. Differences between the interview protocols for GTAs and students may have been a contributing factor. GTAs were first asked about the career-relevant skills they had developed as instructors; questioning the skills that they thought their students had developed followed. Building laboratory skills and gaining experience with techniques are often the primary learning goals and outcomes in traditional laboratory instruction from both student and instructor perspectives (Bretz et al., 2013; Bruck et al., 2010b). Consequently, some GTAs may be predisposed to believing that building technical skills is the primary purpose of laboratory courses based on their prior experiences. In contrast, undergraduate interview 443 participants were first asked to identify the skills needed for their future career and subsequently relate development of these skills to the general chemistry laboratory courses. Asking the undergraduate students to discuss how they may have developed career-related skills after they were asked about the skills required for specific career paths may have influenced their responses. For example, they may not have viewed technical skills used in the general chemistry laboratory as relevant to their planned career. The GTAs may also be more aware of the technical skills that their students are gaining because they have the perspective of knowing how these skills can be applied in subsequent courses and STEM-based careers. For example, Pine related technical skill development to research, “[. . .] They had to figure out how to use a calorimeter and understand what a calorimeter is, and those are extremely important in a lot of physical chemistry research [. . .].” However, other GTAs recognized that most of their students will not pursue careers in chemistry, although some of these technical skills may be applicable in other areas. Acorn saw the benefits of learning microscale techniques to a variety of careers [. . .] even if they end up being some medical expert, or whether they end up in some laboratory or stuff, whatever they do, like, if they learn these techniques, in whatever they do, they can start with like a [small] amount of the chemicals and stuff they are using, and then go on for the more amounts so that if the experiment is a failure, they do not end up wasting the resources. (Acorn, Technical Skills) Almond wanted students to learn more about how the instrumentation used in the labs worked but at the same time acknowledged that most students will not pursue careers in chemistry. So, I mean, this is specifically in reference to [. . .] the Gatorade dye lab, but also the plastic lab works similarly. We don't teach them how an IR spectrometer works. Which I'm a little disappointed about, but also, a lot of these students aren't going to be chemists, a lot of them are from majors across various aspects of science, so it's not important for like a kinesiologist to 444 understand how IR spectroscopy works, I suppose. (Almond, Technical Skills) Additionally, many undergraduate participants may not have made a connection between the general chemistry laboratory courses and technical skill development because technical skills are not a primary focus in the project-based curriculum. Furthermore, over half of the undergraduate participants (n = 30, 57%) took the courses entirely online during the COVID-19 pandemic. A subsample of these participants (n = 9) recognized technical skills as relevant to their planned career but also noted that the online learning environment hampered technical skill development. Although there were some differences between GTA and undergraduate student perceptions of the career-related skills developed by students in the project-based general chemistry laboratory courses, there is notable alignment in the perceptions of the two groups. Among the interview participants, over half the GTAs and over 80% of the undergraduate students believed that students were gaining the following skills highly valued by employers: communication, teamwork, and problem-solving & critical thinking. Implications While this study focused on one component of graduate education in chemistry at one institution, a review of the literature and findings herein support the following recommendations. Although teaching assistantships provide financial support in the form of a stipend and tuition, they have not always been met favorably (Sandi-Urena & Gatlin, 2013). Framing teaching as a professional development activity that helps graduate students build skills needed for workplace success may alter perceptions and support more valuable and fulfilling teaching experiences. This is especially pertinent, seeing as motivations for learning can be influenced by the value one ascribes to an activity or task (National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, et al., 2018). With many graduate students pursuing industry careers (ACS Presidential Commission, 2012; Julie Gould, 2019) practices and skills associated with teaching may not seem relevant. This sentiment is reflected in Pistachio’s comment, “[. . .] I know that [. . .] a few TAs, like, aren't in grad school to become a professor, right? So, um, I think that [. . .] not saying, like, ‘Oh, we need to work on your teaching skills’ because I think that a lot of us, like, you know, we, we kind of know how to talk to students [. . .]” Graduate students place value on experiences 445 related to their future career goal (Busby & Harshman, 2021). Consequently, drawing attention to skills that can be developed through teaching assignments and are also applicable beyond teaching could result in greater value placed on teaching assignments. Strong arguments have been advanced for the need to reform graduate education to better prepare students for the diversity of career paths pursued STEM graduate degree holders (National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018). The graduate teaching experience, particularly in laboratory courses, can be intentionally leveraged to promote professional skill development. This study has shown that project-based laboratory courses where GTAs participate with their students in doing science as mentors can create an environment conducive to building valuable workplace skills offers. However, additional support may be required to derive maximum benefit from such experiences because there can be a mismatch between what GTAs are expected to do in inquiry-based laboratory courses and their perceptions about the actions of a “good” GTA (Duffy & Cooper, 2020). For example, integrating a seminar series into a graduate program that addresses student learning, evidence-based teaching practices, and professional practices can complement and reinforce course-specific training (Lang et al., 2020; Nicklow et al., 2007). Providing opportunities for novice GTAs to both observe and be observed by experienced GTAs and incorporating reflection can further support the professional growth of GTAs and their teaching practices (Lang et al., 2020). Several GTAs in this study also expressed an interest in teaching in different course to round out their teaching experience, which unfortunately isn’t always practical because of high demand in introductory courses. While it has been noted that changes to graduate curricula may be met with resistance due to an underlying fear of losing focus on research activities, many have suggested that additions can be made to graduate education without sacrificing other areas (National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018; National Research Council, Division on Earth and Life Sciences, et al., 2012). The literature suggests being more purposeful in providing opportunities for graduate students to develop soft skills valued by employers. Teaching assistantships have not been fully exploited to prepare students for careers beyond academia. When designing activities to develop soft skills, it is important to consider the potential of particular environments to elicit development, provide opportunities for reflection, and offer 446 feedback on strengths and areas for improvement (Czajka et al., 2021; Jackson, 2016; Reynders et al., 2019). Targeting specific skills that can be gained from different graduate activities, integrating them into the graduate curriculum as learning goals, and allowing for instances of reflection and feedback may be additional areas that graduate curriculum developers or specific course coordinators can pursue. Conclusions & Future Directions The interpersonal skills of communication and leadership emerged as the most prevalent skills that GTAs believed they were developing as an instructor in the project-based general chemistry laboratory courses. Interestingly, these skills were among those employers believed graduate students to be lacking (National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018), In addition, ACS has called for inclusion of more opportunities to develop these skills in chemistry graduate curricula (Kuniyoshi et al., 2021). Having these skills among desired learning outcomes of graduate education is not a recent development, as these have been recognized for years as important skills that graduate students should acquire (Caserio et al., 2004). Further, development of communication skills has been defined in the literature as a learning goal for GTA positions (Donkor & Harshman, 2023). The results of this study suggest that graduate teaching assignments in courses that inquiry-based learning techniques, such as project-based learning, show promise in supporting this learning goal. The communication skills that GTA’s spoke of developing in the project-based general chemistry laboratory courses were both general and teaching-specific. Teaching skills, or the ability to educate and influence others, are reported as valuable to both industry and academic careers (National Research Council, Division on Earth and Life Sciences, et al., 2012), and the ways in which GTA’s in this study were able to learn various methods of communication as instructors may better prepare them for the 21st-century workforce. Further research could delve deeper into professional development of GTAs through incorporation of assessment items into training modules, involving use of self-reflective and peer assessment activities to determine if personal beliefs regarding skill proficiency align with those of a trained observer. Additionally, follow-up studies with GTAs post-graduation to determine what experiences were perceived to influence gainful employment or success in the workforce may provide more conclusive profiles of how big or small of an impact the project-based general 447 chemistry course had on professional skill development or what other aspects were perceived to be more helpful. Over the years, this course has integrated undergraduate learning assistants (ULAs) who act as instructors alongside GTAs. A prerequisite for being employed as a ULA is having successfully completed least one of the general chemistry laboratory courses offered at this school. ULAs have been shown to have sufficient teaching capabilities when compared to GTAs (Wheeler et al., 2017). Since they have had experiences from both the student and instructor perspectives, an area of further inquiry could be exploration of ULA professional development and how experiences as a student and as a instructor shape skill development. Limitations During the time of data collection, the primary author of this study (BE) served as a senior teaching assistant and worked closely with the course administrator for GCL1. This may have resulted in a perceived power differential between the interviewer (BE) and interviewees, which was acknowledged. Author BE reassured participants that their identities would be protected several times and attempted to create an open environment where all thoughts and ideas were welcome. Participants were relatively forthcoming, with both positive and negative experiences, when speaking of their time as instructors for the course. As this is a primarily qualitative study, these results are not generalizable to be beliefs of all GTAs who instruct these courses. Further, the findings presented herein do not aim to be generalized across inquiry-based learning environments. Additionally, this work does not claim to provide a measure for GTA proficiency in the skills they reported developing while serving as an instructor for these courses. Further work must be done to adequately measure whether beliefs of development and skill competency align. 448 REFERENCES ACS Presidential Commission. (2012). Advancing Graduate Education in the Chemical Sciences. https://www.acs.org/education/resources/graduate.html Ashby, M. T., & Maher, M. A. (2019). Consensus, Division, and the Future of Graduate Education in the Chemical Sciences. Journal of Chemical Education, 96(2), 196–202. https://doi.org/10.1021/acs.jchemed.8b00642 Bell, S. (2010). Project-based Learning for the 21st Century: Skills for the Future. The Clearing House: A Journal of Educational Strategies, Issues, and Ideas, 83(2), 39–43. https://doi.org/10.1080/00098650903505415 Bender, W. N. (2012). Project-based Learning: Differentiating Instruction for the 21st Century. In Project-based Learning: Differentiating Instruction for the 21st Century. Corwin. Bruck, L. B., Towns, M., & Bretz, S. L. (2010). Faculty Perspectives of Undergraduate Chemistry Laboratory: Goals and Obstacles to Success. Journal of Chemical Education, 87(12), 1416–1424. https://doi.org/10.1021/ed900002d Busby, B. D., & Harshman, J. (2021). Program Elements’ Impact on Chemistry Doctoral Students’ Professional Development: A Longitudinal Study. Chemistry Education Research and Practice, 22(2), 347–363. https://doi.org/10.1039/d0rp00200c Carmel, J. H., Ward, J. S., & Cooper, M. M. (2017). A Glowing Recommendation: A Project- Based Cooperative Laboratory Activity To Promote Use of the Scientific and Engineering Practices. Journal of Chemical Education, 94(5), 626–631. https://doi.org/10.1021/acs.jchemed.6b00628 Caserio, M., Coppola, B. P., Lichter, R. L., Bentley, A. K., Bowman, M. D., Mangham, A. N., Metz, K. M., Pazicni, S., Phillips, M. F., & Seeman, J. I. (2004). Responses to Changing Needs in U.S. Doctoral Reform. Journal of Chemical Education, 81(12), 1698–1703. www.JCE.DivCHED.org Che, E. S., Brooks, P. J., Schwartz, A. M., Saltzman, E. S., & Whiteman, R. C. (2023). How do graduate students approach college teaching? Influences of professional development, teaching assistantships, and Big Five personality traits. Frontiers in Education, 8. https://doi.org/10.3389/feduc.2023.982998 Cooper, J., Basson, J., & Schaap, P. (2006). A Training Programme Based on the Principles of Social Constructivism and Focused on Developing People for the Future World of Work: An Evaluation. Human Resource Development International, 9(4), 467–483. https://doi.org/10.1080/13678860601032593 Cooper, M. M. (2012). Cooperative Chemistry Lab Manual (5th ed.). McGraw Hill. Cooper, M. M., & Kerns, T. S. (2006). Changing the Laboratory: Effects of a Laboratory Course 449 on Students’ Attitudes and Perceptions. www.JCE.DivCHED.org Creswell, J. W. (2013). Qualitative Inquiry and Research Design: Choosing Among Five Approaches (3rd ed.). Sage Publications, Inc. Cui, Q., & Harshman, J. (2020). Qualitative Investigation to Identify the Knowledge and Skills That U.S.-Trained Doctoral Chemists Require in Typical Chemistry Positions. Journal of Chemical Education, 97(5), 1247–1255. https://doi.org/10.1021/acs.jchemed.9b01027 Czajka, D., Reynders, G., Stanford, C., Cole, R., Lantz, J., & Ruder, S. (2021). A Novel Rubric Format for Providing Feedback on Process Skills to STEM Undergraduate Students. Journal of Science College Teaching, 50, 48–56. Denise Kendall, K., & Schussler, E. E. (2012). Does Instructor Type Matter? Undergraduate Student Perception of Graduate Teaching Assistants and Professors. CBE Life Sciences Education, 11(2), 187–199. https://doi.org/10.1187/cbe.11-10-0091 Department of Chemistry at Michigan State University. (2023). Graduate Program Guide: 2023- 2024. https://www.chemistry.msu.edu/_assets/_files/graduate/program-guide/2023-2024.pdf Donkor, B., & Harshman, J. (2023). Learning Goals and Priorities Identified by an Examination of Chemistry Graduate Handbooks. Journal of Chemical Education, 100(10), 3774–3783. https://doi.org/10.1021/acs.jchemed.3c00062 Duffy, E. M., & Cooper, M. M. (2020). Assessing TA Buy-in to Expectations and Alignment of Actual Teaching Practices in a Transformed General Chemistry Laboratory Course. Chemistry Education Research and Practice, 21(1), 189–208. https://doi.org/10.1039/c9rp00088g Fantone, R. C., Zaimi, I., Meserve, K., Geragosian, E. K., Álvarez-Sánchez, C. O., Spencer, J. L., & Shultz, G. V. (2023). Chemistry Instructional Coaching: Adapting a Peer-Led Professional Development Program for Chemistry Graduate Teaching Assistants. Journal of Chemical Education, 100(11), 4343–4350. https://doi.org/10.1021/acs.jchemed.3c00580 Feldon, D. F., Peugh, J., Timmerman, B. E., Maher, M. A., Hurst, M., Strickland, D., Gilmore, J. A., & Stiegelmeyer, C. (2011). Graduate Students’ Teaching Experiences Improve Their Methodological Research Skills. Science, 333(6045), 1037–1039. https://doi.org/10.1126/science.1207998 Ganapati, S., & Ritchie, T. S. (2021). Professional Development and Career-Preparedness Experiences of STEM Ph.D. Students: Gaps and Avenues for Improvement. PLoS ONE, 16(12 December). https://doi.org/10.1371/journal.pone.0260328 Herrington, D. G., & Daubenmire, P. L. (2014). Using Interviews in CER Projects: Options, Considerations, and Limitations. In Tools of Chemistry Education Research (Vol. 166, pp. 31–59). Oxford University Press. https://pubs.acs.org/sharingguidelines 450 Jackson, D. (2016). Modeling Graduate Skill Transfer from University to the Workplace. Journal of Education and Work, 29(2), 199–231. https://doi.org/10.1080/13639080.2014.907486 Jollands, M., Jolly, L., & Molyneaux, T. (2012). Project-based Learning as a Contributing Factor to Graduates’ Work Readiness. European Journal of Engineering Education, 37(2), 143– 154. https://doi.org/10.1080/03043797.2012.665848 Julie Gould. (2019, November 15). Working Scientist podcast: Too many PhDs, too few research positions. Nature. Kraiger, K. (2008). Transforming Our Models of Learning and Development: Web-Based Instruction as Enabler of Third-Generation Instruction. Industrial and Organizational Psychology, 1(4), 454–467. https://doi.org/10.1111/j.1754-9434.2008.00086.x Krajcik, J. S., & Shin, N. (2014). Project-based Learning. In The Cambridge Handbook of the Learning Sciences, Second Edition (pp. 275–297). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.018 Kuniyoshi, C., Martin, N., Schiavone, C., Schlatterer, J., & Stevens, T. (2021). 2019 ACS Graduate Student Survey Report. https://doi.org/10.1021/acs.edu.2019gssrpt Laker, D. R., & Powell, J. L. (2011). The differences between hard and soft skills and their relative impact on training transfer. Human Resource Development Quarterly, 22(1), 111– 122. https://doi.org/10.1002/hrdq.20063 Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159–174. https://about.jstor.org/terms Lang, F. K., Randles, C. A., & Jeffery, K. A. (2020). Developing and Evaluating a Graduate Student Teaching Assistant Training Course in the Chemistry Department of a Large American University. Journal of Chemical Education, 97(6), 1515–1529. https://doi.org/10.1021/acs.jchemed.9b00686 Larson, R. C., Ghaffarzadegan, N., & Xue, Y. (2014). Too Many PhD Graduates or Too Few Academic Job Openings: The Basic Reproductive Number R0 in Academia. Systems Research and Behavioral Science, 31(6), 745–750. https://doi.org/10.1002/sres.2210 Mabmud, S. M. (2012). Cohen’s Kappa. In Encyclopedia of Research Design. SAGE Publications, Inc. https://doi.org/10.4135/9781412961288 Moustakas, C. (1994). Phenomenological Research Methods. Sage Publications. Murphy, E. (1997). Constructivism: From Philosophy to Practice. https://files.eric.ed.gov/fulltext/ED444966.pdf 451 National Academies. (2011). Assessing 21st Century Skills. In Assessing 21st Century Skills. National Academies Press. https://doi.org/10.17226/13215 National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, Board on Higher Education and Workforce, Policy and Global Affairs, Leshner, A. I., & Scherer, L. A. (2018). Graduate STEM Education for the 21st Century. National Academies Press. https://nap.nationalacademies.org/catalog/25038/graduate-stem-education-for-the-21st- century National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, Board on Science Education, Cognitive Board on Behavioral, & Committee on How People Learn II: The Science and Practice of Learning. (2018). How People Learn II: Learners, Contexts, and Cultures. In How People Learn II: Learners, Contexts, and Cultures. National Academies Press. https://doi.org/10.17226/24783 National Center for Science and Engineering Statistics Directorate for Social Behavioral and Economic Sciences, & National Science Foundation. (2022). 2022 Doctorate Recipients from U.S. Universities. https://ncses.nsf.gov/pubs/nsf24300/ National Research Council, Division on Earth and Life Sciences, Board on Chemical Sciences and Technology, Committee on Challenges in Chemistry Graduate Education, & Board on Chemical Sciences. (2012). Challenges in Chemistry Graduate Education: A Workshop Summary. In Challenges in Chemistry Graduate Education. National Academies Press. https://nap.nationalacademies.org/catalog/13407/challenges-in-chemistry-graduate- education-a-workshop-summary Nicklow, J. W., Asce, M., Marikunte, S. S., & Chevalier, L. R. (2007). Balancing Pedagogical and Professional Practice Skills in the Training of Graduate Teaching Assistants. Journal of Professional Issues in Engineering Education and Practice. Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic Analysis: Striving to Meet the Trustworthiness Criteria. International Journal of Qualitative Methods, 16(1). https://doi.org/10.1177/1609406917733847 Raber, L. (2000). Status Quo Questioned at Ph.D. Workshop. Chemical & Engineering News. https://pubs.acs.org/sharingguidelines Rannikmäe, M., Holbrook, J., & Soobard, R. (2020). Social Constructivism - Jerome Bruner. In B. Akpan & T. J. Kennedy (Eds.), Science Education in Theory and Practice: An Introductory Guide to Learning Theory (pp. 260–261). Springer. Reynders, G., Suh, E., Cole, R. S., & Sansom, R. L. (2019). Developing Student Process Skills in a General Chemistry Laboratory. Journal of Chemical Education, 96(10), 2109–2119. https://doi.org/10.1021/acs.jchemed.9b00441 452 Sandi-Urena, S., Cooper, M. M., Gatlin, T. A., & Bhattacharyya, G. (2011). Students’ Experience in a General Chemistry Cooperative Problem Based Laboratory. Chemistry Education Research and Practice, 12(4), 434–442. https://doi.org/10.1039/C1RP90047A Sandi-Urena, S., & Gatlin, T. (2013). Factors Contributing to the Development of Graduate Teaching Assistant Self-image. Journal of Chemical Education, 90(10), 1303–1309. https://doi.org/10.1021/ed200859e Shulman, J. (2008). Survey of Ph.D. Programs in Chemistry. https://www.acs.org/education/students/graduate/survey-of-phd-programs-in- chemistry.html Sim, J., & Wright, C. C. (2005). The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy, 85, 257. https://academic.oup.com/ptj/article/85/3/257/2805022 The Graduate School at Michigan State University. (2024). Graduate Teaching Assistant Program. https://grad.msu.edu/gtap U.S. Bureau of Labor Statistics. (2022). Education Pays : U.S. Bureau of Labor Statistics. https://www.bls.gov/emp/chart-unemployment-earnings-education.htm Vasileva, O., & Balyasnikova, N. (2019). (Re)Introducing Vygotsky’s Thought: From Historical Overview to Contemporary Psychology. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.01515 VERBI Software. (2021). MAXQDA 2022. maxqda.com Vygotsky, L. S. (1978). Mind in Society: Development of Higher Psychological Processes. Harvard University Press. Wendler, C., Cline, F., Kent, J., & Mageean, D. (2012). Pathways Through Graduate School and Into Careers Executive Summary. https://files.eric.ed.gov/fulltext/ED531250.pdf Wheeler, L. B., Chiu, J. L., Maeng, J. L., & Bell, R. L. (2019). An Exploratory Study of Teaching Assistants’ Motivation for Inquiry-based Teaching in an Undergraduate Laboratory Context. Chemistry Education Research and Practice, 20(1), 53–67. https://doi.org/10.1039/c8rp00157j Wheeler, L. B., Maeng, J. L., Chiu, J. L., & Bell, R. L. (2017). Do Teaching Assistants Matter? Investigating Relationships Between Teaching Assistants and Student Outcomes in Undergraduate Science Laboratory Classes. Journal of Research in Science Teaching, 54(4), 463–492. https://doi.org/10.1002/tea.21373 Wurdinger, S. D. (2016). The Power of Project-based Learning: Helping Students Develop 453 Important Life Skills. In The Power . Rowman & Littlefield. 454 APPENDIX A.VII.1. Interview Protocol Greeting statement. Hello! Thank you for taking the time to meet with me today. Background statement. Today I am going to ask you questions about your experience as a TA and what skills you think that students developed while taking the course that will be beneficial for their future careers. Then I would like to look at your own professional development as an instructor for the course. Participation in this interview will not affect your TA appointment and the laboratory coordinators will not be notified of your participation. This interview will be audio-recorded so that I have an accurate record of what you say. During the interview, I may wait to make sure that you have had adequate time to think about and respond to a question and that you have finished responding. I may ask follow up questions that ask you to elaborate to make sure that I understand what you are saying. Opening questions. Future career and prior teaching experience 1. What are you specializing in during your graduate school career here at MSU? • Follow up: What interested you in that specialty? 2. What career plans do you have following graduation? 3. How long have you been a TA for the general chemistry laboratories at MSU? 4. Do you have experience TAing other courses at MSU? • Follow up: Did you have prior experience TAing classes before attending graduate school at MSU? (either undergraduate experience/work experience/ any experience as a TA or course assistant). Graduate TA perception of student experience. Skills students have developed in the course and course elements that contributed to development. 5. What aspects of the general chemistry laboratory courses do you view as beneficial for students’ career preparation? Provide TAs a link via Google documents or teams in which interviewee’s can write live responses to the following question asked below. 6. What skills do you believe students are developing in the general chemistry laboratory course that will be beneficial towards success in the workforce? 455 7. What specific aspects of the course contributed to student development of [skills mentioned]? • Follow up: Walk through each skill mentioned. 8. What skills do you believe could be more incorporated into the general chemistry laboratory curriculum for students? 9. Are there specific aspects of the course that you believe could be changed to better incorporate development of [skills mentioned]? Graduate TA Perception of Professional Development and Personal Growth. Skills developed as an instructor for the course and ways in which their experiences led to development. Continuing to use the link provided via google documents or teams, participants were asked to write their response to the following question below. 10. Focusing on your own experience as a general chemistry laboratory TA, what skills did you develop while being a TA for CEM [161/162]? • Follow up: Walk through skills mention and specific examples of development. 11. What specific aspects of the general chemistry laboratory curriculum contributed to personal development of [skill] • Side note: Can talk about TA meetings, professional development, the teaching environment, etc. 12. Are there any aspects of the curriculum or professional development that you would change to better incorporate skill development for graduate student TAs? Optional questioning (if time permits). Employer-desired competencies and continuation of skills TAs perceive themselves and students to be developing in the course. Provide TAs with visual aid containing list of 6 professional skills (Figure 7.3) and continue with line of questioning below. 13. This is a condensed list of skills that employers desire in recent graduates and new hires. Are there any skills on here that you would like to expand on, that you did not previously talk about, in relation to student development in the general chemistry laboratories at MSU? • Follow up: Specific instance of development in course. 456 14. Are there any skills on here that you would like to expand on, that you did not previously talk about, in relation to graduate student TA development teaching the general chemistry laboratories at MSU? 15. Are there any skills on here that you believe students should be given more of a chance to develop that they you believe they did not get a chance to? Follow up: What about for graduate student TAs? Stop sharing screen. End/closing miscellaneous questions (if time or line of questioning permits). 16. What learning environment did you primarily teach in when you were a general chemistry laboratory TA; in-person or online learning? 17. How can the learning environment influence skill development for students? (Can be positive or negative aspects of learning environment.) 18. How can the teaching environment influence skill development for graduate student TAs? (Can be positive or negative aspects of teaching environment.) Closing statement. In closing, is there anything else that you would like to add about anything we have talked about here today? Thank you! Close. 457 Figure 7.3. Visual aid used in interviews. 458 A.VII.2. Qualitative Coding of Skills Developed as a Teaching Assistant for Course Coding Scheme - Skills Developed as a TA for the Course 1. Interpersonal Skills: • Teamwork & Collaboration: cooperation between co-instructors in the same section, conflict management • Communication Skills: includes verbal communication (e.g., public speaking/presenting), answering student questions in a style that gets them to think, adapting communication style based on student needs, ability to clearly convey thoughts/ideas to others, communicating across various levels (e.g., instructor-instructor vs instructor-student communication) • Leadership Skills: mentoring skills (e.g., learning how to be a professor/principal investigator (PI)); coordinating, guiding, and managing students so that every team/student successfully and safely completes experiments, are managing time properly, and are staying on task, motivating students • Social & People Skills: being understanding towards undergraduate student perspectives and experiences 2. Intrapersonal Skills: • Personality & Character Traits: patience (e.g., not getting frustrated with students), adaptability/flexibility, positive attitude 3. Teaching-related Skills: • Flexibility in Learning & Teaching: having to coordinate back and forth between learning concepts again to be able to teach them to students • Course Organization: learned how to organize a course • Grading Skills: learned how to provide better feedback and read through student assignments more carefully 4. Critical Thinking 5. Time Management Skills: • Punctuality 6. Technical Skills: • Laboratory Techniques & Skills: learning how to use well plates/microscale techniques 459 7. Other Skills: • Learned from Experiments that Students were Performing (e.g., learned facts surrounding kidney stones such as how many US citizens suffer from this medical issue) • Decision-making: having to come up with solutions and make decisions when presented with conflicting information or problems arise 460 A.VII.3. Course Elements that Led to Development of Skills as Instructor for Course Below are themes found within GTA responses recounting perceived development of skills that were viewed as valuable towards participant career goals. Table 7.2 explores themes found for subskills of interpersonal skills and Table 7.3 further showcases code occurrences for all other skill sets that were less prevalent in GTA responses. Beyond the codes of conveying information to others and collaborative environment, that were described within the main body of Chapter VII, a few new codes emerged regarding development of other skills not previously explored. Bringing a positive attitude to the classroom entailed making sure to come to class with a good disposition because of the influence a GTAs personality can have on student experiences. Being punctual to class was in regard to showing up on time to instruct each laboratory session. Learned laboratory techniques encompassed learning new laboratory skills or techniques through instructing the course and learning something new (factually) was in reference to learning new facts related to experiments (e.g., prevalence of kidney stones in the population) that the GTA was unaware of before. The final code observed in GTA responses was learning how to structure a course that encompassed learning how to organize a course based on the layout, activities, and resources provided to students in the general chemistry laboratory courses. All corresponding tables are located on pages 461 – 462. Table 7.2. Course elements that contributed to perceived interpersonal skill development (n). Course Element that Contributed to Perceived Development Conveying Information to Others Collaborative Environment Skill Perceived as Developed as an Instructor for the Course Communication Skills (n = 9) 8 Leadership Skills (n = 5) 1 Teamwork Skills (n = 2) — Social & People Skills (n = 1) — General Interpersonal Skills (n = 1) — 7 5 2 1 1 461 Table 7.3. Course elements that contributed to perceived skill development of intrapersonal skills, teaching skills, critical thinking, time management, technical skills, and other. (n). Skill Perceived as Developed as an Instructor for the Course Course Element that Contributed to Perceived Development Conveying Information to Others Collaborative Environment Bringing Positive Attitude to Classroom Being Punctual to Class Learned Laboratory Techniques Learning Something New (Factually) Learning How to Structure a Course Intrapersonal Skills (n = 3) 1 Teaching Skills (n = 2) 2 Critical Thinking (n = 1) — Time Management (n = 2) — Technical Skills (n = 1) — Other (n = 2) — 2 1 — — — — — — — — — 1 1 — — — — — 1 — 1 — — — — — — 1 — — 1 — — — 1 — 462 A.VII.4. Skills Desired as an Instructor Participants in this study were also asked if there were other skills they would like to further develop as GTAs to be included within GTA training. Only 4 participants expressed an interest in additional training; the skills mentioned were teaching, communication, and leadership. Interest in development and training related to teaching fell into 4 areas, 1) more diverse teaching assignments, 2) learning how to better instruct or lecture, 3) grading student assignments, and 4) learning new pedagogical approaches applicable to STEM courses. To further develop teaching skills, two participants spoke of wanting more diverse teaching assignments outside the scope of the general chemistry laboratory courses. Although Palm’s future career goal surrounded starting a company in industry, they spoke of wanting to learn teaching skills, in particular how to better instruct in what seemed to be more lecture-based or traditional laboratory courses that contained pre-laboratory lecture components. [. . .] there should be some instruction [course], because I have already taken this lab three times. And then [GCL2] twice and then [GCL1] this semester. So, I feel I should have some instruction-based course. So, that I can improve, like work on my instruction skills and then grow more on the communication side. [Later in Interview] I would want it to take it to the next level, with some higher-level course, or some courses where more management or more instruction is involved [. . .] [Later in interview] like if there is an instruction based thing at the starting of the lab that would improve the communication skills and instruction activity of the TAs as well. (Palm) Palm also felt this would aid in further development of other desired skills such as leadership (specifically management) and communication skills. Teaching a wider variety of courses at higher levels was also relayed by Almond, a graduate student who desired going into academia and expressed a passion for teaching. Almond thought having different teaching opportunities would offer them a more well-rounded skill set that surrounded being introduced to different problems that students encounter and desiring the ability to teach topics closer to their specialty. 463 [. . .] So there's like, one thing doesn't need to teach every skill, right? So I think, with regards to that I was satisfied with, I really enjoyed TAing. Because I also like [understand] that it's an introductory course, personally, yeah sure I'd like to get experience teaching higher level courses [. . .] So, maybe allowing [. . .] grad students the opportunity to make sure that like, "Oh, we should get you to TA this lab because you've done the gen chem labs, let's upgrade you to this one,” so you get this like other kinds of experience. (Almond) Although providing more diverse teaching assignments is outside of the scope of what can be implemented in the general chemistry laboratory course trainings, these perspectives can offer an insight to how some GTAs desire a greater variety of teaching opportunities versus being placed as an instructor for the same course repeatedly. Earlier in their interview, Acorn had talked about learning how to provide better feedback to students and read through assignments more carefully as developing the teaching skill of getting better at grading student activities. Although this was cited as a source of development, Acorn felt that more thorough and robust training was needed to better develop these skills and to be more adequately prepared as an instructor for the course. This is an area course administrators could further explore to provide more sufficient training to ensure GTAs feel adequately prepared to grade and provide feedback on student assignments. Macadamia was the only participant who desired more training focused on learning new pedagogical approaches. They expressed that being provided additional training that focused on learning a variety of teaching methods and techniques to be applied within the context of the chemistry laboratory could be beneficial to their professional development. Unless there was like [. . .] some of the trainings into like the certificate for college teaching or something like that. Where, because a lot of the workshops and stuff I go to for those aren't really focused on like teaching chemistry. A lot of them are put on and hosted by English teachers/social sciences, where they're talking about essays, and I don't know [. . .] I just struggle to bring those back to chemistry. So having new, wrapping some of that in with the class specific training where "Here's the technique," you know, "Now you get a 464 chance to practice it in the week," might be helpful [. . .] So having opportunities to learn and do the professional development. Learning from people that [taught] the same subject can be really helpful [. . .]. (Macadamia) Being given opportunities to learn specific pedagogical approaches that can be applied directly to a GTA’s teaching assignment has been noted as being desired in the literature (Lang et al., 2020) and may be an area in which further evolution of the professional development provided to GTAs may be explored. 465 A.VII.5. Qualitative Coding of Skills TAs Believed Students were Developing in the Course Coding Scheme - Skills Developed as a Student in the Course 1. Interpersonal Skills: • Teamwork & Collaboration: working with people who have a difference of opinion, learning to grow as a team player • Communication Skills: includes verbal communication (e.g., public speaking/presenting skills), written communication (e.g, preparing reports, writing professionally in notebooks or reports), learning to ask questions, ability to clearly convey thoughts and ideas 2. Intrapersonal Skills: • Work Ethic: being accountable for work • Personality & Character Traits: being comfortable asking questions, learning how to be independent 3. Problem-solving & Critical Thinking: • Reasoning through connecting findings to scenario/real societal issues • Learning to navigate difficult questions • Creativity 4. Time Management Skills: • Multi-tasking • Punctuality • Meeting deadlines 5. Technical Skills: • Laboratory Skills & Techniques: basic/general chemistry lab skills (e.g., vacuum filtration, working with laboratory equipment), learning micro scale techniques to save resources and chemicals, laboratory safety (e.g., how to safely handle chemicals), • Technological/Computer Skills: Microsoft Office skills (e.g., Excel/PowerPoint) 6. Conceptual Knowledge & Research Skills: • Learning new concepts via experimentation • Building practical knowledge through connecting theoretical concepts to experimentation • Learning how to properly research and identify reliable references 466 7. Education & Learning Skills: • Learning & School Skills & Strategies: willingness to learn 467 A.VII.6. Numerical Breakdown of Skills GTAs Reported Students to be Developing within the Project-based General Chemistry Laboratory Courses Table 7.4. Skills GTAs perceived to be developed as an undergraduate student enrolled in the project-based general chemistry laboratory courses. Career Relevant Skill Communication Skills Teamwork/Collaboration Misc. Interpersonal Work Ethic Personality & Character Traits Problem Solving and Critical Thinking Time Management Technical Skills Conceptual Knowledge/Research Skills Education & Learning Skills & Strategies Participants Who Reported Course Supported Development, n (%) 7 (58) 9 (75) 2 (17) 1 (8) 3 (25) 8 (67) 4 (33) 7 (58) 2 (17) 1 (8) 468 REFERENCES Lang, F. K., Randles, C. A., & Jeffery, K. A. (2020). Developing and Evaluating a Graduate Student Teaching Assistant Training Course in the Chemistry Department of a Large American University. Journal of Chemical Education, 97(6), 1515–1529. https://doi.org/10.1021/acs.jchemed.9b00686 469 CHAPTER VIII: CONLUSIONS, IMPLICATIONS, & FUTURE DIRECTIONS Conclusions Throughout this dissertation, development of employer-desired competencies (EDCs) was explored from both the student and instructor perspective. Prior to this work, employers, who had experiences with undergraduate students in the College of Natural Science at Michigan State University, stated that they felt students were unable to articulate proficiency in skills such as communication, teamwork, problem-solving, and critical thinking (Telfor, 2017). Participants were sampled from introductory general chemistry laboratory courses to investigate how the project-based learning environment, which engages students in the scientific practices, might encourage development of EDCs. Using a mixed-methods approach, focusing on qualitative data collected through semi-structured interviews, and a social constructivist theoretical framework to guide this study, the following key findings emerged. Finding 1: Students identified EDCs relevant to their career coal & perceived development of EDCs in project-based general chemistry laboratory courses. Exploring student ability to articulate prevalent workforce skills was one of the primary goals of the research presented herein. Findings from interviews (Chapter IV) and surveys (Chapter VI) show that students are not only able to identify valuable EDCs as being relevant to their future career, both with and without prompting, but that they were also able to clearly define experiences in their general chemistry laboratory courses that supported perceived development. Communication, teamwork, work ethic, problem-solving & critical thinking, prioritization & time management, and technical skills were highly regarded as valuable career competencies by interview participants and align with skills desired by employers throughout the literature (Kondo & Fair, 2017; National Association of Colleges and Employers, 2023; National Research Council et al., 2011; World Economic Forum & in collaboration with The Boston Consulting Group, 2015). These findings were further supported by survey participants who were also able to relate many EDCs to success in their future career. Further, many interview and survey participants attributed development of these valuable 21st-century skills to course elements of the project-based general chemistry laboratory courses. Emergent in interview participant responses was how the collaborative environment led to perceived development of communication, teamwork, work ethic, problem-solving & critical thinking, and prioritization & time management skills. Additionally, the open-inquiry nature of 470 the courses – having students design and plan experiments with less explicit guidance from instructors – was believed to support problem-solving & critical thinking skill development. Many of these themes were further supported by survey participant responses regarding skill development. Although we do not offer commentary on the assessment of development of EDCs, these results continue to reinforce the benefits of students engaging in a project-based learning environment (Bell, 2010; Jollands et al., 2012) and how, when given a chance to reflect on development, students are able to pinpoint aspects of the curriculum that contribute to growth. Finding 2: Online learning positively & negatively impacted student development of EDCs. Online learning was a new format for many students and instructors during the COVID pandemic (Means et al., 2020) and was particularly challenging for those administering laboratory courses (Sansom, 2020). Investigating how participants believed engaging in an online general chemistry laboratory course influenced their ability to gain valuable EDCs (Chapter V) was a new area of inquiry that presented itself during data collection for Chapters IV and VI. The EDC most affected was technical skills; many interview participants believed that learning online was detrimental to skill development due to the loss of in-person and hands- on experiences. Taking laboratory courses online distanced students from becoming familiar with laboratory skills and techniques and did not allow them to engage in their own experimentation and data collection. A desire to further develop these skills was reported by both interview and survey participants, with many explicitly stating or implying that being online prevented them from developing technical skills. Participants had contrasting views regarding the influence online learning had on development of other EDCs, such as communication, teamwork, work ethic, problem-solving & critical thinking, and prioritization & time management. Interview participants were split on whether online learning supported, hindered, or had no influence on EDC development. Further, interview participant responses indicated that their perception of adapting to the online learning environment, either a positive or negative, seemed to influence whether they believed that remote learning aided skill development. In contrast, survey participants were more apt to see online learning as hindering development of EDCs due to barriers to collaboration and communication and a lack of hands-on opportunities. Both interview and survey responses indicated an underlying desire to engage in these laboratory courses in-person. However, learning to communicate and collaborate remotely, as observed in some student responses, may 471 be beneficial to graduates entering the 21st-century workforce as many careers require engagement in collaborative efforts that may be conducted remotely. Finding 3: Students value EDCs in various areas of their lives and desire further integration of these skills into their college experience. EDCs have been recognized in the literature as having value to many areas of a person’s life (Binkley et al., 2012; National Research Council et al., 2011; Organisation for Economic Co- operation and Development, 2019) – a perception shared by interview and survey participants. Student participants spoke of how EDCs were applicable to their academic and professional careers, along with being seen as general life skills (Chapter IV and VI). Beyond their own career pursuits, many participants found these skills to be important to a variety of careers, emphasizing how integrating opportunities to develop EDCs in higher education classrooms may be advantageous for many areas of a student’s life. Further, survey responses supported integration of EDCs into the college classroom with 86% of participants expressing a desire to continue learning about these skills during their college career (Chapter VI). With students wanting more experiences gaining relevant career competencies and the high value attributed to these skills, incorporating skill development into course curricula could provide beneficial experiences within classroom environments. Finding 4: Graduate teaching assistants perceived development of EDCs as instructors for the project-based general chemistry laboratories. Much like recent bachelor’s graduates, the skills gap has also been reported for graduates of STEM masters and doctoral programs (National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018), prompting our investigation into how graduate teaching assistants (GTAs) for the project- based general chemistry laboratory courses perceived development of valuable 21st-century skills (Chapter VII). The most prevalent skills that GTAs reported they as an instructor for these courses were the interpersonal skills of communication and leadership. By acting as a mentor, guiding and supervising undergraduate students in these courses, GTAs developed these skills. Communication skill development was further aided by learning how to convey information to various types of students and learning how to present. With teaching assignments often viewed as unfavorable requirements of a graduate student’s career (Sandi-Urena & Gatlin, 2013), 472 leveraging a project-based laboratory learning environment that encourages development of 21st- century skills, as observed in this study, may add value to these assignments. Implications While the four studies presented herein were exploratory, the findings, accompanied by a review of the literature, highlight important aspects to consider when integrating 21st-century skill development into student learning environments: 1. Frame learning in career-relevant contexts - assess curriculum and learning goals for areas in which 21st-century skill development can be integrated and use evidence-based pedagogical practices to implement these goals. 2. Build awareness of skill development by incorporating opportunities for self-reflection and feedback throughout a student’s college career. 3. Situate teaching assistantships as valuable opportunities for professional development. Implication 1: Frame learning in career-relevant contexts. Learning can be motivated by an individual’s interests and learning goals or what they hope to achieve from engaging in the learning process (National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, et al., 2018), including future career goals. These goals are not stagnant, as supported by the small sample of Fa21 survey participants in this dissertation (Chapter VI), who reported different career goals at the beginning and end of the semester. Further, the literature recognizes how learning environments can affect whether students continue toward or change from a particular career goal (Seymour & Hunter, 2019). One of the first steps in a student’s STEM education are introductory courses, such as general chemistry, that cover foundational topics and are often required to advance to more major-specific courses. These courses often carry a negative connotation of being used to “weed out” low performers and have been met with disinterest, causing students to switch to more appealing majors outside of STEM (President’s Council of Advisors on Science and Technology, 2012; Seymour & Hunter, 2019). By leveraging student learning goals, such as their future careers, to situate learning in relevant contexts, introductory STEM courses may be reframed with student interests in mind, resulting in increased engagement and retention in these courses. Additionally, integration of 21st-century skills that are believed to be applicable across 473 occupations, may equip students with skills that are valuable to career success, even if their plans change. Implementing 21st-century skills in the classroom requires careful assessment of the learning environment. Reflecting on the curriculum and learning goals in the context of student careers may reveal areas in which career preparation is thought to be self-evident but is actually lacking. Kerr & Runquist found that when they compared the skills employers desired to their science education curriculum, the curriculum focused on careers geared towards academia (Kerr & Runquist, 2005). This was a sobering finding as they reported that 75 - 85% of their graduates were not aiming for employment in an academic career and emphasized the need to assess how course curriculum can be transformed to better serve the futures of the students enrolled. Kerr & Runquist’s report points to the importance of identifying student career goals and taking them into consideration in curricula to better prepare students for the workforce. Much like Kerr & Runquist, taking inventory of one’s classroom can provide useful information to guide curriculum development. In the studies reported in this dissertation, undergraduate participants primarily desired careers in health and medicine or engineering. They identified communication, teamwork, work ethic, problem-solving & critical thinking, and prioritization & time management as 21st-century skills that were relevant to their future careers and believed to be developed in the general chemistry laboratory courses. These outcomes could be used to define learning goals, in course curricula, to bring awareness to additional outcomes that may contribute to success in future careers. Use of short surveys could be a valuable tool, for any course, to collect and assess general student characteristics and goals (e.g., beginning of semester survey to identify what career goals and skills students hope to gain from a course and end of semester surveys of what was believed to be gained from a course) that could be used in development of learning goals. To reach learning goals centered on 21st-century skill development, evidence-based pedagogical practices should be used. While results from the studies included in this dissertation did not explore use of pedagogical practices beyond project-based learning, this learning environment was believed by participants to support development of valuable career competencies (Chapter IV and VI). Project-based learning encourages a highly collaborative environment (Krajcik & Shin, 2014), and this course element was frequently discussed in student responses as contributing to development of highly valued workplace skills, such as 474 communication, teamwork, problem-solving & critical thinking, and prioritization & time management. Other course elements that students related to skill development included designing experiments, analyzing and interpreting data, and project management. Engaging in the scientific practices of designing and carrying out investigations and analyzing and interpreting data, which are also components of open inquiry learning, were believed by students to enhance their problem-solving & critical thinking skills. Further, students discussed developing problem-solving & critical thinking, communication, and teamwork skills by collaborating with team members to work through scaffolded planning documents designed to guide students through planning experiments. Additionally, being given projects that spanned multiple weeks and multiple assignments, were believed to encourage growth of prioritization & time management skills. These findings support literature reports that project-based learning can encourage development of 21st-century skills (Jollands et al., 2012; Wurdinger, 2016) and engaging in the scientific practices can support skills such as student reasoning that was related to critical thinking in these studies (Stowe & Cooper, 2017). While it may not be feasible or beneficial for skill development and career preparation to be the sole focus of a course’s curriculum, implementing a curriculum that incorporates elements of project-based learning and scientific practices may be naturally support development of EDCs. Implication 2: Build awareness of skill development, allow for self-reflection & feedback, & provide continual opportunities to develop skills. A key step in progressing toward integration of 21st-century skill development in the classroom, is first introducing students to the definitions and applications of these skills. Although EDCs often suffer from broad definitions (National Research Council et al., 2011), instructors can define specific skills within the context of their courses, draw attention to opportunities for skill development, and situate learning in career-relevant contexts. For example, in Sp21 (Chapter VI) course materials introduced a list of EDCs that could be developed within the general chemistry laboratory courses to build awareness. Integrating short and concise lists, such as the one employed in Sp21, into course activities may be beneficial because they require minimal time to implement, make use of readily available course materials, and can act as an opportunity to provide students with continual exposure to EDCs. Beyond familiarizing students with EDCs, allowing opportunities for students to reflect is another critical aspect of facilitating skill development. Through reflection, students can a) see 475 the practical application of what was learned, b) recognize areas of success, and c) identify areas that need further growth (Helyer, 2015). Integration of short reflective assignments, throughout or towards the end of a course, provides students with opportunities to look back upon their experiences and relate them to development of prevalent 21st-century skills. These assignments could give students practice in articulating skill development that could translate into demonstrating proficiency for job interviews. Further, use of reflective assignments could bring attention to students’ strengths and weaknesses. Although not referring to the reflective assignment itself, some Sp21 survey participants (Chapter VI) felt that being introduced to EDCs gave them a chance to not only identify how skills were used and developed within the course (GCL1), but also helped them identify areas for improvement. Additionally, a small sample of Sp21 survey participants (Chapter VI) explicitly mentioned how beneficial the reflective assignments were in helping them recognize growth and development. The literature goes one step farther, suggesting a combination of self-reflection and instructor feedback, through use of rubrics designed to assess skill development, as beneficial for student growth (Czajka et al., 2021; Reynders et al., 2019). Exposure to EDCs and instances of development should not be limited to one course but rather integrated throughout the undergraduate college career. As recognized in the literature, practice is a component of learning and mastery (Koedinger et al., 2023), indicating a need for recurring opportunities that go beyond the confines of singular experiences in one classroom. The desire for more opportunities was reflected in Fa21 Final Reflective Assignment (FRA) survey responses (Chapter VI), where most students in the sample stated that they wanted more experiences with valuable workplace competencies during their college career. From this, it can be surmised that a sustained effort to include 21st-century skills in higher education may be met favorably by students, further emphasizing the importance of providing and maintaining such opportunities throughout an undergraduate’s education. Although transformative efforts centered around 21st-century skills risk becoming a fad (Rotherham & Willingham, 2010), consistent reports that students are not adequately prepared in the skills needed for today’s workforce (Carlson, 2022; National Association of Colleges and Employers, 2023) underline the importance of continued efforts to include career preparation in the undergraduate curriculum. 476 Implication 3: Use graduate teaching assistantships as opportunities for professional development. Within the literature, graduate teaching assistantships have been recognized as an area that could be leveraged for professional development (Ashby & Maher, 2019); however, this is an aspect of graduate education that is often overlooked (Sandi-Urena & Gatlin, 2013). Much like Implications 1 and 2, which focused on undergraduate education, the graduate curriculum could greatly benefit from situating GTA training and related opportunities for professional development within a career-relevant context. This is important given that most chemistry graduate students serve as teaching assistants, accompanied by the fact that many will not continue to a career in academia (Kuniyoshi et al., 2021). In the studies presented herein (Chapter VII), GTAs recognized opportunities to develop valuable career competencies as instructors for the project-based general chemistry laboratory courses. These perceived instances of development spanned across GTAs who aspired to go into both academic and industry careers. Based on the outcomes of this study, using a project-based learning environment and placing GTA instructors in a mentor-like position, may provide opportunities for development of valuable skills such as communication and leadership. Although this study did not investigate how training provided to GTAs was supported skill development and instead focused on instructor experiences within the classroom, the outcomes from this study can guide future training modules. While GTA study participants were able to reflect on specific skill development as an outcome of instructing these courses (Chapter VII), opportunities such as these are not typically integrated within GTA training. Like the suggestions above for students enrolled in these courses, first drawing attention to valuable workplace competencies that can be developed as an instructor and allowing for instances of reflection could be a beneficial addition to training sessions. While it is not suggested that GTA training focus primarily on career preparation, opportunities for 21st-century skill development may already be embedded within their experiences as an instructor (Chapter VII) and it may just be necessary to draw attention to them. Calls to explicitly integrate career preparation into the graduate curriculum after often met with concerns that will distract from research opportunities (National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, et al., 2018). Enhancing the GTA 477 experience to explicitly support development of career-relevant skills would place minimal additional demands on graduate students while potentially having a large impact. Future Directions Higher education institutions are tasked with the daunting feat of continually adapting to an evolving workforce and ensuring students are adequately prepared for modern-day careers. While there are many education researchers, organizations, and institutions striving to meet student and employer needs (Chadwick et al., 2018; Kondo & Fair, 2017; Organisation for Economic Co-operation and Development, 2019; Partnership for 21st Century Learning, 2019), there is still much work left to be done. The studies reported in this dissertation show that students and graduate teaching assistants perceived that they developed EDCs when either enrolled in or instructing project-based general chemistry laboratory courses. However, there was no control (e.g., sampling participants from a traditional laboratory course) to compare these findings against, so we cannot claim that these experiences are solely the product of project- based learning or are of any higher value than other learning environments. To better assess the influence a learning environment (e.g., either traditional or inquiry-based pedagogies) may have on EDC development, further work could sample students from a variety of classrooms that employ different pedagogical approaches. Through use of items to assess specific skills and triangulation with semi-structured interviews to see if student perception of development aligns with observed development, pedagogical approaches that may better support skill development could be determined. Continuing to track students and GTAs through both their academic and professional careers in a longitudinal study could provide further insight into the most relevant experiences contributing to perceived career preparation and EDC development. Another area to explore focuses on technology used in these courses. One of the pillars of project-based learning is recognizing the importance of integrating current technology into student projects (Krajcik & Shin, 2014). Artificial intelligence (AI) is proliferating the academic sphere (Swaak, 2024). The course coordinators for the project-based general chemistry laboratory courses investigated in this dissertation are considering integration of this technology. The effects of projects or activities using AI and their influence on EDC development may be another interesting area of exploration. Additionally, the value that employers place on career preparation achieved within the classroom or through work experiences could be explored. With the literature continually 478 underscoring how employers desire higher education institutions to better prepare students in 21st-century skills (Carlson, 2022; Gardner, 1997; Kerr & Runquist, 2005; National Research Council et al., 2011), a study could be designed to investigate whether an employer sees experiences derived from college courses on an application as equal to or lesser than work-based experiences. Findings could highlight what aspects of college education are relevant to career preparation and workplace success. 479 REFERENCES Ashby, M. T., & Maher, M. A. (2019). Consensus, Division, and the Future of Graduate Education in the Chemical Sciences. Journal of Chemical Education, 96(2), 196–202. https://doi.org/10.1021/acs.jchemed.8b00642 Bell, S. (2010). Project-based Learning for the 21st Century: Skills for the Future. The Clearing House: A Journal of Educational Strategies, Issues, and Ideas, 83(2), 39–43. https://doi.org/10.1080/00098650903505415 Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining Twenty-first Century Skills. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and Teaching of 21st Century Skills (pp. 17–66). Springer. https://doi.org/DOI 10.1007/978-94-007-2324-5 Carlson, S. (2022). Building Tomorrow’s Work Force. In The Chronicle of Higher Education. The Chronicle of Higher Education. Chadwick, S., De La Hunty, M., & Baker, A. (2018). Developing Awareness of Professional Behaviors and Skills in the First-Year Chemistry Laboratory. Journal of Chemical Education, 95(6), 947–953. https://doi.org/10.1021/acs.jchemed.8b00025 Czajka, D., Reynders, G., Stanford, C., Cole, R., Lantz, J., & Ruder, S. (2021). A Novel Rubric Format for Providing Feedback on Process Skills to STEM Undergraduate Students. Journal of Science College Teaching, 50, 48–56. Gardner, P. (1997). Are Seniors Ready to Work? In The Senior Transition. Sage Publications. Helyer, R. (2015). Learning Through Reflection: The Critical Role of Reflection in Work-based Learning (WBL). Journal of Work-Applied Management, 7(1), 15–27. https://doi.org/10.1108/JWAM-10-2015-003 Jollands, M., Jolly, L., & Molyneaux, T. (2012). Project-based Learning as a Contributing Factor to Graduates’ Work Readiness. European Journal of Engineering Education, 37(2), 143– 154. https://doi.org/10.1080/03043797.2012.665848 Kerr, S., & Runquist, O. (2005). Are We Serious about Preparing Chemists for the 21st Century Workplace or Are We Just Teaching Chemistry? In In the Classroom www.JCE.DivCHED.org • (Vol. 82, Issue 2). www.JCE.DivCHED.org Koedinger, K. R., Carvalho, P. F., Liu, R., & McLaughlin, E. A. (2023). An Astonishing Regularity in Student Learning Rate. Proceedings of the National Academy of Sciences of the United States of America, 120(13). https://doi.org/10.1073/pnas.2221311120 Kondo, A. E., & Fair, J. D. (2017). Insight into the Chemistry Skills Gap: The Duality Between Expected and Desired Skills. Journal of Chemical Education, 94(3), 304–310. https://doi.org/10.1021/acs.jchemed.6b00566 480 Krajcik, J. S., & Shin, N. (2014). Project-based Learning. In The Cambridge Handbook of the Learning Sciences, Second Edition (pp. 275–297). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.018 Kuniyoshi, C., Martin, N., Schiavone, C., Schlatterer, J., & Stevens, T. (2021). 2019 ACS Graduate Student Survey Report. https://doi.org/10.1021/acs.edu.2019gssrpt Means, B., Neisler, J., & with Langer Research Associates. (2020). Suddenly Online: A National Survey of Undergraduates During the COVID-19 Pandemic. https://files.eric.ed.gov/fulltext/ED610781.pdf National Academies of Sciences Engineering and Medicine, Committee on Revitalizing Graduate STEM Education for the 21st Century, Board on Higher Education and Workforce, Policy and Global Affairs, Leshner, A. I., & Scherer, L. A. (2018). Graduate STEM Education for the 21st Century. National Academies Press. https://nap.nationalacademies.org/catalog/25038/graduate-stem-education-for-the-21st- century National Academies of Sciences Engineering and Medicine, Division of Behavioral and Social Sciences and Education, Board on Science Education, Cognitive Board on Behavioral, & Committee on How People Learn II: The Science and Practice of Learning. (2018). How People Learn II: Learners, Contexts, and Cultures. In How People Learn II: Learners, Contexts, and Cultures. National Academies Press. https://doi.org/10.17226/24783 National Association of Colleges and Employers. (2023). Job Outlook 2023. https://www.naceweb.org/store/2022/job-outlook-2023 National Research Council, Committee on the Assessment of 21st Century Skills, Center for Education, Board on Testing and Assessment, & Koenig, J. A. (2011). Assessing 21st Century Skills: Summary of a Workshop. In Assessing 21st Century Skills. National Academies Press. https://doi.org/10.17226/13215 Organisation for Economic Co-operation and Development. (2019). Future of Education and Skills 2030: Conceptual Learning Framework. https://www.oecd.org/education/2030- project/teaching-and-learning/learning/skills/Skills_for_2030.pdf Partnership for 21st Century Learning. (2019). Framework for 21st Century Learning Definitions. https://static.battelleforkids.org/documents/p21/P21_Framework_DefinitionsBFK.pdf President’s Council of Advisors on Science and Technology. (2012). Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and Mathematics. https://files.eric.ed.gov/fulltext/ED541511.pdf Reynders, G., Suh, E., Cole, R. S., & Sansom, R. L. (2019). Developing Student Process Skills in a General Chemistry Laboratory. Journal of Chemical Education, 96(10), 2109–2119. 481 https://doi.org/10.1021/acs.jchemed.9b00441 Rotherham, A. J., & Willingham, D. T. (2010). 21st-Century Skills: Not New, but a Worthy Challenge. American Educator, 17–20. Sandi-Urena, S., & Gatlin, T. (2013). Factors Contributing to the Development of Graduate Teaching Assistant Self-image. Journal of Chemical Education, 90(10), 1303–1309. https://doi.org/10.1021/ed200859e Sansom, R. L. (2020). Pressure from the Pandemic: Pedagogical Dissatisfaction Reveals Faculty Beliefs. Journal of Chemical Education, 97(9), 2378–2382. https://doi.org/10.1021/acs.jchemed.0c00657 Seymour, E., & Hunter, A. B. (2019). Talking about Leaving Revisited Persistence, Relocation, and Loss in Undergraduate STEM Education. Springer International Publishing : Imprint: Springer. Stowe, R. L., & Cooper, M. M. (2017). Practicing What We Preach: Assessing “Critical Thinking” in Organic Chemistry. Journal of Chemical Education, 94(12), 1852–1859. https://doi.org/10.1021/acs.jchemed.7b00335 Swaak, T. (2024). AI Will Shake Up Higher Ed. Are Colleges Ready? The Chronicle of Higher Education. https://www.chronicle.com/article/ai-will-shake-up-higher-ed-are-colleges-ready Telfor, B. (2017). 16 Essential Health and Science Skill Sets. https://natsci.msu.edu/sites/_natsci/assets/File/Current%20Students/College%20Career%20 Services/Option%201%20Skill%20Sets%20with%20Definitions%20for%20Website- CNS%20Articulation%20Exercise%20Website%20Posting%20BT_ST.pdf World Economic Forum, & in collaboration with The Boston Consulting Group. (2015). New Vision for Education: Unlocking the Potential of Technology. https://www3.weforum.org/docs/WEFUSA_NewVisionforEducation_Report2015.pdf Wurdinger, S. D. (2016). The Power of Project-based Learning: Helping Students Develop Important Life Skills. In The Power. Rowman & Littlefield. 482