1 PREPARE: Program Review and Evaluation for Professional Advancement in Nurse Residency Education Joshua M. Winowiecki College of Nursing, Michigan State University NUR 997: DNP Project III Dr. Jackeline Iseler April 18, 2024 2 Abstract ............................................................................................................................................. 5 PREPARE: Program Review and Evaluation for Professional Advancement in Nurse Residency Education .......................................................................................................................................... 6 Evaluation Goal ................................................................................................................ 7 Stakeholder Assessment ................................................................................................................... 8 Background & Description .............................................................................................................. 11 Program Accreditation .................................................................................................... 11 Need ................................................................................................................................ 12 Context ............................................................................................................................ 14 Target Population ........................................................................................................... 15 Objectives ........................................................................................................................ 16 Stage of Development ..................................................................................................... 17 Activities ......................................................................................................................... 18 Outputs ........................................................................................................................... 18 Outcomes ........................................................................................................................ 19 Evaluation Focus ............................................................................................................................ 20 Stakeholder Needs ......................................................................................................... 20 Evaluation Questions ...................................................................................................... 21 Evaluation Design ........................................................................................................... 21 Resource Consideration .................................................................................................. 23 Evaluation Standards ..................................................................................................... 23 Utility Standards ........................................................................................................ 23 3 Feasibility Standards ................................................................................................. 24 Propriety Standards .................................................................................................. 24 Accuracy Standards ................................................................................................... 24 Evaluation Accountability Standards ....................................................................... 24 Literature Review ............................................................................................................................ 24 Search Methods .............................................................................................................. 24 Literature Synthesis ........................................................................................................ 25 Data Collection ................................................................................................................................ 26 Indicators ........................................................................................................................ 26 Collection ........................................................................................................................ 26 Qualitative .................................................................................................................. 26 Quantitative ................................................................................................................ 27 Timeline .......................................................................................................................... 28 Analysis & Interpretation ............................................................................................................... 28 Analysis ........................................................................................................................... 28 Nurse Resident Program Completion Rate .............................................................. 28 Primary Evaluation Questions .................................................................................. 29 Secondary Evaluation Questions .............................................................................. 29 Interpretation ................................................................................................................. 31 Dissemination & Use ...................................................................................................................... 33 Dissemination ................................................................................................................. 33 Use .................................................................................................................................. 33 4 References ....................................................................................................................................... 35 Table 1: Stakeholder Assessment .................................................................................................. 40 Table 2: Project Description ........................................................................................................... 45 Table 3: Program Evaluation Standards ........................................................................................ 46 Figure 1: First-Year Reasons for Termination .............................................................................. 48 Figure 2: Thematic Analysis ........................................................................................................... 49 Figure 3: Top Reasons for Termination ......................................................................................... 50 Figure 4: Second-Year Reasons for Termination ........................................................................... 51 Figure 5: Reasons for Termination Beyond Two Years of Tenure ................................................. 52 Figure 6: Participant Competence .................................................................................................. 53 Figure 7: Transition Difficulties ..................................................................................................... 54 Figure 8: Preceptor Feedback ......................................................................................................... 55 Figure 9: Nurse Resident Feedback ............................................................................................... 56 Appendix A: Literature Synthesis ................................................................................................... 57 Appendix B: Stakeholder Assessment Matrix ............................................................................... 60 Appendix C: Logic Model ................................................................................................................ 61 5 Abstract A nurse residency program (NRP) at a large urban academic hospital was evaluated to assess the effectiveness of transitioning new graduate nurses (NGNs) into clinical practice, especially during the pandemic. The evaluation used a mixed-methods approach that combined quantitative data from NGN surveys with qualitative feedback. Results showed that the NRP has positively impacted NGNs’ confidence and professional skills, but critical areas for improvement exist in curriculum relevance and support mechanisms. Enhancing clinical simulations and revising curricular components could better align the NRP with the practical demands of clinical environments, leading to improved patient care and reduced turnover costs. Keywords: nurse residency program, new graduate nurses, program evaluation, transition to practice 6 PREPARE: Program Review and Evaluation for Professional Advancement in Nurse Residency Education Nurse residency programs (NRP) have been widely adopted to help new graduate nurses (NGN) transition from student to clinician. These programs aim to build confidence, solidify vital clinical skills, and build a sense of clinical inquiry by emphasizing role socialization (Alsalamah et al., 2022; Smith, 2021). The recent pandemic has led to graduate nurses having increased stress and anxiety in their transition to practice, directly related to their reduced clinical education time (Michel et al., 2021). These factors have led to NGNs who may be less than optimally prepared for practice, have increased stress and anxiety, and report low confidence in critical clinical skills and scenarios (Brown et al., 2022; Harper et al., 2021; Michel et al., 2021). A large, urban, academic hospital’s NRP will be evaluated using the Center for Disease Control and Prevention’s (CDC) Framework for Program Evaluation in Public Health (1999) for this project. This sizeable Midwestern hospital’s nurse residency program is a comprehensive program designed to support the transition of NGNs from the academic setting to clinical practice. The program aims to address the challenges NGNs face, particularly in light of the recent pandemic, which has reduced clinical education time and increased stress and anxiety among NGNs. The evaluation of this program is necessary to assess its effectiveness in meeting its objectives, such as improving self-reported competence among NGNs, reducing turnover rates, enhancing communication and teamwork skills, increasing knowledge of evidence-based nursing practices, and improving critical thinking and problem-solving abilities. By evaluating the program using the CDC’s Framework for Program Evaluation in Public Health (1999), the hospital can gain valuable insights into the program’s strengths and areas for improvement, leading to enhancements that will better support NGNs in their transition to professional practice. This evaluation is crucial to ensure the program’s continuous improvement and effectiveness in facilitating NGNs’ successful integration into nursing. 7 The CDC’s Framework for Program Evaluation in Public Health (1999) is a practical and nonprescriptive tool designed to guide public health professionals in conducting effective program evaluations (1999). The framework serves several purposes: a program overview, a structured framework for evaluation, clarifying evaluation steps, reviewing standards for practical evaluation, and addressing misconceptions about program evaluation (Centers for Disease Control and Prevention, 1999). The framework comprises key program evaluation steps and standards. These steps include engaging stakeholders, describing the program, focusing on the evaluation design, gathering credible evidence, justifying conclusions, and ensuring use and lessons learned. The standards encompass utility, feasibility, propriety, and accuracy. By following this framework, public health practitioners can better understand the context of each program and improve the conception and execution of program evaluations. The framework emphasizes the integration of evaluation with routine program operations and encourages the involvement of all program stakeholders. Real-world examples of its application include evaluating county-wide tuberculosis monitoring programs and community smoking cessation initiatives. Utilizing the CDC’s Framework for Program Evaluation in Public Health can drive effective planning program improvement and demonstrate the results of resource investments in public health initiatives (1999). Evaluation Goal This evaluation aims to assess the effectiveness of a hospital-based nurse residency program (NRP) by focusing on program impact, measured by participant self-reported competence, NGN termination rates and reasons for termination, participant completion rates and overall program satisfaction, and critical stakeholder perspectives. While the specific areas for improvement within the program are not yet identified, the evaluation will provide valuable insights to inform potential enhancements. The primary objectives of the NRP include successful program completion, participant satisfaction, and increased knowledge and skills. The evaluation identifies opportunities for improvement, mainly where participant-reported 8 confidence scores related to clinical skills and program satisfaction are currently low. Ultimately, the review aims to support continuous improvement and ensure the program’s effectiveness in facilitating new graduate nurses’ transition into professional practice. By December 2023 a complete a comprehensive evaluation of the hospital-based nurse residency program will be completed. The evaluation will assess NGN turnover rates, reasons for termination, stakeholder perceptions of the program, participant self-reported competence and participant feedback. It will also identify opportunities for improvement in areas where participant-reported confidence scores related to clinical skills and program satisfaction are currently low. Stakeholder Assessment The NRP involves stakeholders essential in its planning, implementation, and evaluation. These stakeholders include the program coordinator, nursing education administrator, unit-based clinical nurse specialists (CNSs), nursing administration, nurse managers, nurse educators, nurse residents, patients, and unit-based preceptors. The program coordinator oversees the program, ensuring its objectives are met. The nursing education administrator ensures alignment with educational standards. Unit-based clinical nurse specialists support nurse residents’ clinical education and development in their specialty areas. Nursing administration provides leadership and resources for the program’s success. Nurse managers oversee resident activities and professional growth. Unit-based nurse educators deliver unit-specific education and mentorship. The nurse residents are the primary stakeholders seeking to transition successfully into professional nursing practice. Patients benefit from the care provided by nurse residents. Unit-based preceptors are experienced nurses who mentor and guide nurse residents during clinical rotations. These stakeholders have different interests and perspectives. The program coordinator and nursing education administrator prioritize program effectiveness and educational quality. Clinical nurse specialists focus on clinical competency and specialty training. Nursing 9 administration emphasizes alignment with organizational goals and resource allocation. Nurse managers consider resident performance and unit dynamics. Unit-based nurse educators evaluate educational outcomes. Nurse residents aim for skill development and program completion, while patients expect safe and high-quality care. Unit-based preceptors emphasize nurse residents’ professional development and integration into the unit. Engaging these stakeholders in the evaluation process is crucial. Strategies include regular communication, seeking feedback, involving them in decision-making, and demonstrating the program’s value in achieving their goals. Through their active involvement, the evaluation can gather diverse perspectives, identify areas for improvement, and enhance the overall effectiveness of the nurse residency program. For example, engaging the unit-based preceptors could elicit that NGNs need help with specific clinical competencies or specialty training. Additionally, preceptors’ insights into program effectiveness and resident learning experiences can provide valuable perspectives on areas for improvement and skill development for nurse residents, as highlighted by the unit- based preceptors. Engaging with these preceptors throughout the evaluation can contribute to evidence-based recommendations for enhancing the nurse residency program’s impact and outcomes. The nurse residency program’s stakeholder assessment (see Table 1) identified the primary stakeholders (nurse residents) and categorized vital stakeholders based on their role (internal and external) and their magnitude of impact (influence) and commitment to the program (investment). The program coordinator, nursing education administrator, and unit- based clinical nurse specialists emerge as stakeholders with high influence and investment. These individuals are integral to the program’s planning, implementation, and evaluation. Their expertise and involvement directly impact the program’s success, making their perspectives and feedback highly valuable. Collaborating closely with them throughout the evaluation process can provide valuable insights into program strengths, areas for improvement, and potential 10 strategies for enhancing the program’s effectiveness (Halar et al., 2020; Simpson et al., 2020; Zamboni et al., 2020). Additionally, stakeholder involvement in the evaluation process significantly impacts the utilization of evaluation results (Okul & Nyonje, 2020). Nursing administration represents another important stakeholder group, characterized by high influence over program operations via decision-making and resource allocation but a relatively lower investment in program-specific functionality. While they may not have direct involvement in day-to-day program operations, their support and decision-making authority can significantly impact the program’s overall sustainability and resource allocation. Engaging nursing administration in the evaluation process can help ensure alignment with organizational goals and priorities and secure necessary resources and support for implementing evaluation recommendations (Chant & Westendorf, 2019; Shinners et al., 2021; Wilson et al., 2018). On the other hand, nurse managers, unit-based nurse educators, patients, and nurse residents are stakeholders with high investment but relatively lower influence. Nurse managers oversee the daily activities of the residents and are closely involved in their professional development. Their perspectives can provide valuable insights into the program’s impact on resident performance and growth. Likewise, as direct program beneficiaries, the nurse residents possess valuable firsthand experiences and perceptions that can contribute to the evaluation process. The impact on patients is the obvious extension of the program, as the overarching goal of developing the NGN is to allow them to perform at the limits of their knowledge and licensure, thus providing safe, efficient care to patients. Lastly, unit-based preceptors, characterized by low investment and influence, may have limited direct involvement in program evaluation activities. However, their insights as frontline educators and mentors for the nurse residents can provide valuable perspectives on program effectiveness and resident learning experiences. Engaging with these diverse stakeholders throughout the evaluation process can ensure a comprehensive understanding of the program’s strengths and areas for improvement, foster ownership and 11 buy-in, and ultimately contribute to developing evidence-based recommendations for enhancing the nurse residency program’s impact and outcomes. Background & Description Established in 1915, this large urban not-for-profit, academic hospital is one of the largest and most comprehensive healthcare systems in the United States (Henry Ford Health System 2018 Annual Report, 2018). With over 25,000 employees and $8.5 billion in annual revenue, it includes six hospitals, 29 medical centers, and a health insurance company (Henry Ford Health System 2018 Annual Report, 2018). Their mission is to improve people’s lives through health care and healing, and their values include compassion, innovation, integrity, excellence, respect, teamwork, and community (Henry Ford Health: Mission, Vision, Values, 2023). Moreover, they are known for pioneering work in cancer treatment, heart disease, and stroke care, and offer various education programs for healthcare professionals (Henry Ford Health Continuing Education - About Us, 2023; Henry Ford Health System 2018 Annual Report, 2018; Henry Ford Innovations, 2023). Additionally, they are committed to providing high-quality, affordable care to residents of Southeast Michigan (Henry Ford Health System 2018 Annual Report, 2018; Henry Ford Innovations, 2023). Program Accreditation The Practice Transition Accreditation Program® (PTAP) provides a framework for evaluating and improving nurse residency programs. PTAP is administered by the American Nurses Credentialing Center (ANCC) and aims to recognize excellence in transition-to-practice programs (American Nurses Credentialing Center, 2023). The PTAP standards are based on Benner’s novice-to-expert model and encompass five key domains: program leadership, program goals and outcome measures, organizational enculturation, development and design, and practice-based learning (ANCC, 2023). To achieve PTAP accreditation, programs must demonstrate adherence to eligibility criteria, submit documentation showing how the program meets all PTAP standards, and have 12 program participants complete an evaluation survey. Accredited programs undergo annual reviews to maintain their status. PTAP provides an evidence-based model to assess the effectiveness of transition-to-practice programs in achieving outcomes like nurse competence, satisfaction, and retention. The planned evaluation of the nurse residency program aligns well with the PTAP accreditation framework. Applying for PTAP accreditation would facilitate a comprehensive review using established criteria and outcome metrics. The PTAP standards overlap significantly with the proposed evaluation questions for this project, including assessing program leadership, curriculum, preceptor development, and outcome measurement. Preparing a PTAP self-study would require gathering evidence and demonstrating the residency program’s strengths across all domains. In addition, the required participant survey would provide valuable feedback on the learners’ perspectives and experiences. Pursuing PTAP accreditation could serve as an impetus for critically examining the residency program to identify areas for improvement. The self-assessment and accreditation process could inform enhancements to support nurse residents’ transition to practice. If accreditation is achieved, it would validate the program’s effectiveness in meeting PTAP criteria for successful transition programs. PTAP’s annual review requirements would also promote ongoing quality improvements. In summary, PTAP accreditation would facilitate the planned comprehensive evaluation using rigorous national standards and support continuous improvements in the nurse residency program. Need The recent pandemic has left the nursing workforce with low job satisfaction, leading to higher-than-usual job turnover and increased employer costs (Cadmus & Roberts, 2022; Peacock, 2023; Savitsky et al., 2021). Additionally, due to the increased turnover rates and vacancies left by nurses who either retired or left the profession during the pandemic, many NGNs are accepting positions in higher acuity areas that may not have historically taken newer 13 nurses (Cadmus & Roberts, 2022; Drexler et al., 2023; Peacock, 2023; Savitsky et al., 2021). Similar to those national trends, this hospital’s NGNs face several challenges in their transition to practice, including reduced clinical education time, increased online learning, and increased stress and anxiety (S. Brown, personal communication, March 1, 2023). At this Midwestern hospital, first-year registered nurse (RN) turnover remains high, with around 20% of all newly hired nurses leaving within the first year (C. Jackman, personal communication, September 12, 2023). The average cost to replace a bedside RN is $52,350 (NSI Nursing Solutions, 2023); this first-year turnover represents over $4 million in replacement expenses annually. Since 2017, over 865 nurses have completed the hospital’s nurse residency program (NRP), yet first-year turnover has remained around 14% cumulatively (C. Jackman, personal communication, September 12, 2023; (Vizient Inc., 2024)). While lower than the national rate of 27% first-year turnover, ongoing improvements are needed (NSI Nursing Solutions, 2018, 2019, 2020, 2021, 2022, 2023). Each percentage change in RN turnover costs this hospital approximately $380,000 per year (NSI Nursing Solutions, 2023). High RN turnover impacts the quality of care, patient satisfaction, and hospital finances. Evaluating the current residency program will identify opportunities to better prepare and retain nurse residents during this challenging time. One of the primary outcomes of many nurse residency programs is facilitating the transition to practice, leading to reduced turnover and improved retention (Cadmus & Roberts, 2022; Miller et al., 2023). Since the pandemic, other methods of reducing staff turnover have been suggested, but few precisely target NGN (Alsalamah et al., 2022; Cadmus & Roberts, 2022; Drexler et al., 2023; Harper et al., 2021; Miller et al., 2023; Smith, 2021). The organization is committed to achieving several key objectives critical to the business’s success. One of the top priorities is to reduce expenses, which will help the organization operate more efficiently and maintain profitability. Another important objective is to enhance employee happiness, which is 14 essential for creating a positive work environment and fostering a sense of loyalty and commitment among staff members. Additionally, the organization aims to minimize staff turnover, which is essential for ensuring continuity of care and maintaining high standards of patient care. Finally, the organization is focused on cultivating a skilled team of nurses, which is critical for delivering high-quality care and meeting the evolving needs of patients and their families. Implementing a nurse residency program with components like a prolonged preceptor experience, simulation exercises, debriefing sessions, and peer networking extends support beyond standard orientation and leads to higher retention rates and more successful transitions for new graduates (Asber, 2019; Van Patten & Bartone, 2019; Wildermuth et al., 2020). By focusing on these key objectives, the partner organization is well-positioned to achieve its goals and provide exceptional care to patients and their families. Context Several models of nurse residency programs are available, including commercially prepared programs like Vizient/American Association of Colleges of Nursing (Vizient/AACN), UnitedHealthcare/American Association of Critical-Care Nurses (UHC/AACN), and Versant or organization-specific programs developed internally (Asber, 2019). These programs vary from 12 weeks to 1 year and offer components like dedicated preceptors, simulation training, mentors, classroom education, and debriefing sessions (Asber, 2019). Research suggests that the nationally-recognized UHC/AACN and Versant programs achieve the highest 1-year retention rates of 90-100% compared to organization-specific programs, which have more variability in outcomes (Asber, 2019). The structured curriculum and experience implementing these external residencies across different organizations likely contribute to their success. While organization- specific programs can be tailored, they require more effort in creating content and initially garner lower retention. Therefore, implementing an established, rigorous external residency 15 program appears preferable when supporting new graduate nurses’ transition to practice and boosting retention. Since its inception in 2017, the nurse residency program at Henry Ford Hospital has experienced significant growth. It now includes four cohorts each year, with an average of 45 residents in each cohort. The program spans 12 months. As of mid-2023, 998 residents have enrolled in the program, with 865 completing it, representing an 81% completion rate. However, the program graduates have a cumulative 1-year turnover rate of 14%, with voluntary departures due to new job experiences, unsatisfactory performance, relocation, and the recent pandemic (see Figure 1 for top reasons for termination in 0–12-month tenure group). Involuntary separation can result from attendance issues, unsatisfactory performance, and disability. Target Population The new graduate nurse (NGN) population comprises individuals who have recently completed their nursing education and are entering the workforce as registered nurses in their first year of practice. Typically ranging in age from early 20s to late 30s and predominately female, these nurses possess formal education in entry-to-practice nursing, including an associate degree in nursing (ADN), a traditional Bachelor of Science in Nursing (BSN), and accelerated/second-degree BSN (Eckerson, 2018; Fowler et al., 2018; Miller et al., 2023; Van Patten & Bartone, 2019; Wildermuth et al., 2020). The coronavirus disease pandemic 2019 has had significant implications on their clinical education. Limited clinical placements were available due to increased demand for healthcare services and the need to minimize exposure. Consequently, virtual learning experiences were incorporated but could only partially replace hands-on experience. New graduate nurses also faced altered learning environments, with strict infection control measures and increased focus on telehealth. Additionally, the pandemic placed emotional burdens on these nurses as they navigated the healthcare crisis. Qualitative data collected during the program and post-program indicate that the most common themes that new nurses struggle with include “making a mistake,” “not knowing what 16 to do,” “not knowing how to respond to a code,” “not feeling confident,” “worried about how to advocate for my patient,” and “feel like I don’t know what I’m doing” (S. Brown, personal communication, March 1, 2023). While the recent pandemic did not create all the problems facing the nursing workforce – including new graduate nurses – it did lay bare many challenges. Specific to education and training, healthcare organizations are focusing efforts on reducing training costs but reducing turnover while retaining a burnout cadre of staff. Supporting NGNs in the transition to practice is an essential function of an NRP and is crucial in providing continued support for NGNs after their traditional orientation ends. Objectives The nurse residency program aims to achieve several key objectives that support the successful transition of new graduate nurses into professional practice. Firstly, the program seeks to provide a structured curriculum and adherence to program standards, including best practices and the published curriculum, ensuring that participants receive comprehensive and up-to-date education in various domains of nursing practice. Secondly, the program aims to enhance the self-reported confidence of new graduate nurses by providing them with opportunities for skill-building, knowledge acquisition, and hands-on experiences in a supportive learning environment. Thirdly, the program strives to decrease turnover rates among new graduate nurses by equipping them with the necessary tools, knowledge, and skills to navigate the challenges of professional practice, leading to a sense of job satisfaction and commitment to their nursing careers. Furthermore, the program strongly emphasizes enhancing communication and teamwork skills among participants, recognizing the crucial role these skills play in delivering safe and effective patient care. The program aims to improve patient outcomes and promote person-centered care by fostering effective interprofessional collaboration and coordination. Additionally, the program seeks to enhance participants’ knowledge of evidence-based nursing practices, equipping them with the tools to critically evaluate and apply current research 17 findings in their clinical decision-making processes. Lastly, the program aims to enhance participants’ proficiency in patient assessment and care planning, enabling them to provide high-quality, holistic care to individuals across diverse healthcare settings. Overall, these objectives collectively contribute to the overarching goal of the nurse residency program, which is to facilitate a successful transition into professional nursing practice for new graduates while ensuring they possess the knowledge, skills, confidence, and support necessary to thrive in their roles and provide optimal care to their patients. Stage of Development Established in 2017, the nurse residency program has entered its maintenance phase. The program has been an incredible opportunity for aspiring new nurses to receive hands-on training and gain valuable experience in the field. However, recent evaluations have shown poor participant program satisfaction scores and low self-reported confidence levels related to clinical skill. Therefore, the evaluation aims to uncover opportunities for improvement that will continue to provide valuable resources and support to current and future nurse residents. With the guidance and support of experienced nurses, the program has helped shape and mold the next generation of healthcare professionals. With its proven success and commitment to improvement, the nurse residency program will continue positively impacting the healthcare industry for years. Resources & Inputs The nurse residency program relies on various crucial resources and inputs to ensure its effectiveness and success. Adequate funding serves as the financial backbone of the program, providing the resources required for its operations, curriculum development, participant support, and evaluation efforts (S. Brown, personal communication, March 1, 2023). A dedicated program coordinator is pivotal in overseeing and coordinating various aspects of the program, ensuring smooth implementation, and addressing administrative needs. Facilitators with expertise in nursing education and mentorship guide the participants throughout the 18 program, facilitating meaningful learning experiences and fostering professional growth. Content experts contribute their specialized knowledge and expertise to enrich the program’s curriculum, ensuring its alignment with current best practices and evidence-based guidelines. Clinical facilities for training, including hospitals, clinics, and simulation centers, provide the necessary real-world environments for participants to gain hands-on experience and apply their skills. A well-designed curriculum and training materials also form the program’s backbone, providing structured content and resources for classroom instruction, skills development, and self-directed learning (Vizient AACN Nurse Residency Program, 2023). These resources and inputs collectively support the program’s objectives and provide the foundation for the subsequent activities and outcomes. Activities The nurse residency program implements various activities to provide participants with a comprehensive and immersive learning experience (see Appendix E and Table 2). Classroom instruction focuses on imparting nursing theory and best practices, fostering a deep understanding of core concepts. Skill-building workshops and simulations offer hands-on training, refining technical skills and enhancing clinical competence. Regular mentoring sessions provide individualized guidance and support, nurturing participants’ professional development and addressing any challenges they face. These activities work synergistically to create a well-rounded program that blends theoretical knowledge with practical application, enabling participants to transition smoothly into their nursing careers (Letourneau & Fater, 2015; Rush et al., 2019; Van Camp & Chappy, 2017). Outputs The nurse residency program generates several tangible outputs that reflect its successful implementation (see Appendices E and F). One notable output is the number of participants completing the program, indicating the program’s ability to engage and retain participants throughout its duration. Participant satisfaction ratings are essential, reflecting how much the 19 program meets the participants’ expectations and needs. Another significant output is the participants’ acquisition of skills and knowledge, demonstrating their growth and development during the program. These outputs not only signify the immediate achievements of the program but also lay the foundation for outcomes. Outcomes The nurse residency program aims to achieve short-term, intermediate, and long-term outcomes (see Appendix E and Table 2). Short-term outcomes include increased self-reported competence, confidence in participants’ clinical abilities, improved communication and teamwork skills, and enhanced knowledge of evidence-based nursing practices. These outcomes provide an immediate reflection of the program’s impact on participants’ skills and self- perception. Intermediate outcomes encompass enhanced critical thinking and problem-solving abilities, increased patient assessment and care planning proficiency, and improved medication administration skills. These outcomes demonstrate the progressive growth and advancement of participants’ clinical competence. The ultimate long-term outcomes encompass enhanced patient outcomes and quality of care, increased retention rates of new nurses at one, two, and three years, and continued professional development and career advancement. These outcomes signify the program’s ability to contribute to the overall enhancement of healthcare delivery, the retention of competent nursing professionals, and the promotion of lifelong learning and career success. Outcomes are self-reported by nurse residents via the Casey-Fink Graduate Nurse Experience Survey and are collected regularly throughout the program (Asber, 2019; Casey et al., 2021). This validated survey assesses new nurses’ perceptions of their skills, confidence, competence, transition, and job satisfaction across multiple time points to evaluate the residency’s impact on their development and transition to practice (Van Patten & Bartone, 2019; Wildermuth et al., 2020). 20 Stakeholder Needs Evaluation Focus In evaluating the nurse residency program, it is crucial to consider the needs of the various stakeholders involved. Understanding and addressing these needs will foster stakeholder engagement, ensure the relevance and usefulness of evaluation findings, and promote the implementation of recommended improvements (Halar et al., 2020; Okul & Nyonje, 2020; Simpson et al., 2020; Zamboni et al., 2020). The program coordinator, nursing education administrator, and unit-based clinical nurse specialists, who hold strong influence and investment in the program, require comprehensive and evidence-based evaluation findings (see Table 1). They seek insights into the program’s effectiveness, strengths, and areas for improvement to inform program planning, curriculum development, and resource allocation. Their needs revolve around understanding the program’s impact on new graduate nurses’ transition, professional development, and retention rates. Additionally, they may require information on the effectiveness of the program’s components, such as professional development seminars and mentorship initiatives, to guide future program enhancements. With high influence but lower investment, nursing administration requires evaluation findings that align with organizational goals and priorities. They seek assurance that the program is cost-effective, sustainable, and contributes to the overall strategic direction of the healthcare institution. Evaluation results highlighting the program’s impact on nurse retention, patient outcomes, and organizational outcomes will address their needs. Nurse managers and nurse residents, with high investment but lower influence, have needs related to the practical aspects of the program. Nurse managers require insights into the program’s impact on resident performance, skill development, and team dynamics. They seek evidence-based recommendations to support their role as mentors and preceptors. As direct beneficiaries, nurse residents desire evaluation findings that validate their experiences, address 21 their concerns, and ensure the program effectively supports their transition into professional practice. Finally, clinical preceptors, with lower investment and influence, may have limited specific needs in the evaluation process. However, they may appreciate evaluation findings that recognize their contributions and shed light on the effectiveness of their mentoring role. By considering and addressing the unique needs of these stakeholders throughout the evaluation process, the evaluation can provide valuable information to guide program improvements, enhance stakeholder satisfaction and engagement, and ultimately contribute to the success and impact of the nurse residency program. Evaluation Questions The evaluation process aims to answer several key questions to assess the nurse residency program’s effectiveness, identify improvement areas, and provide insights for stakeholders. The primary evaluation questions include: 1) What are the self-reported competence levels of new graduate nurses before and after completing the program? 2) How does the program contribute to nurse retention rates? Answering these questions will help determine if the program is achieving its intended outcomes and meeting the needs of new graduate nurses during this transition period. Additionally, secondary evaluation questions can provide more detailed insights. These questions include: 1) How do participants view the program? 2) What insights do vital stakeholders have about the program and its outcomes? 3) What is the self-reported confidence in clinical skills? Addressing these secondary evaluation questions will lead to a comprehensive assessment of the nurse residency program and generate valuable recommendations for improvement. The full range of evaluation questions will guide continuous quality improvements to best support new graduates. Evaluation Design The evaluation will focus on assessing the effectiveness of the nurse residency program in achieving its stated goals and objectives. It will examine the program’s impact on participant 22 outcomes, such as self-reported competence levels, program satisfaction, and retention rates. Additionally, the evaluation will explore critical stakeholder perspectives and participant feedback. The evaluation will cover all program cohorts from its inception in 2017 until the present. The evaluation will utilize a mixed-methods approach to gather comprehensive and robust data. Quantitative methods include surveys administered to program participants to assess their confidence and competence levels, satisfaction, and perceptions of the program’s effectiveness. Retention rates and program completion rates will also be analyzed. Qualitative methods will involve conducting focus groups or interviews with key stakeholders, such as the program coordinator, nursing education administrators, nurse managers, clinical preceptors, and nurse residents. These methods will provide in-depth insights into stakeholders’ experiences, perspectives, and suggestions for program improvement. Data collection will involve various strategies to obtain the necessary information. Surveys are already distributed to program participants before and after the program to measure confidence levels, self-reported competence, and satisfaction changes. Surveys are also administered after each session to gather feedback on specific program components. The results of these surveys will be available for analysis related to this program evaluation. Focus groups or interviews will be conducted with key stakeholders to explore their perceptions, challenges, and suggestions related to the program. Program documents, such as curriculum materials, attendance records, and performance evaluations, will be reviewed to gather additional data. Data collection may also involve reviewing program outcomes and tracking retention rates from program records and organizational databases. By employing a combination of quantitative and qualitative methods, the evaluation will capture numerical data and personal perspectives, providing a comprehensive understanding of the program’s effectiveness and areas for improvement. The data collection strategies will ensure that various stakeholders’ voices are heard, and the evaluation methods will generate reliable and valid data to inform decision-making and program enhancement. 23 Resource Consideration Adequate resources are vital for conducting a comprehensive, methodologically rigorous evaluation of the nurse residency program. Essential resources include personnel time and effort, funding, participant access, and infrastructure. The program coordinator, nurse educators, clinical nurse specialists, and unit managers must dedicate time to meetings, providing information, and facilitating data collection from nurse residents. Additional personnel time is required for focus groups and interviews. Funding is also necessary for access to curriculum and survey administration. Gaining full access to program participants across multiple cohorts for surveys, interviews, and record reviews is essential for robust data collection and will require coordination. Potential challenges include voluntary survey participation, high nursing staff workload limiting availability, and possible data accessibility issues. Strategies to mitigate limitations include utilizing already existing survey data, offering flexible times for participation, seeking leadership support for prioritizing evaluation activities, and allotting sufficient time for navigating data systems. While limitations exist, emphasizing the benefits of evaluation to all stakeholders can help secure necessary resources for a successful evaluation. Evaluation Standards This program evaluation will uphold the Joint Committee on Standards for Educational Evaluation Program Evaluation Standards (Yarbrough et al., 2011). Using these standards helps ensure that the evaluation is systematic, competent, ethical, and utility-focused (see Table 3). Utility Standards Attention will be given to meeting stakeholder needs and ensuring timely communication (U2, U7). Considering their context, the evaluation purposes will be negotiated with the hospital (U3). Findings will be relevant and actionable (U5). 24 Feasibility Standards Effective project management strategies will be used to ensure an efficient evaluation (F1). Procedures will align with hospital operations (F2) and available resources (F4). Propriety Standards Respect for nurse resident participants will be maintained (P3). Transparent communication with stakeholders will occur while protecting confidential data (P5). Any conflicts of interest will be disclosed (P6). Accuracy Standards Conclusions will be justified based on credible evidence (A1). Systematic data collection, analysis, and verification methods will be used (A3, A5, A6). Limitations will be disclosed (A8). Evaluation Accountability Standards Thorough documentation will allow an external review of the evaluation processes and outcomes (E1). The evaluation design and implementation will be critically examined for quality and effectiveness (E2). Adhering to these standards can enhance the methodological rigor, ethicality, and overall quality of the evaluation, resulting in more valuable recommendations (Yarbrough et al., 2011). The standards will guide each step, from planning to reporting, to increase the credibility and utility of findings. Search Methods Literature Review A thorough search of the literature was performed on May 25, 2023, using the Cumulative Index of Nursing and Allied Health Literature (CINAHL), the PubMed index of the US National Library of Medicine National Institutes of Health, and the Education Resources Information Center (ERIC) of the Institute of Education Sciences division of the US Department of Education. Initial search terms “nurse residenc*” AND effectiv* OR outcome* OR evaluat*. The results were limited to English and published within the last ten years. 25 Initial results returned 100 articles (CINAHL), seven articles (PubMed), and zero articles (ERIC). After duplicate removal, the aggregate results totaled 100 unique articles. After title and abstract screening, 54 individual articles remained for full-text abstraction. Of the remaining articles, 14 were excluded as they evaluated the wrong outcomes measured compared to this project, 16 were excluded due to low level or quality of evidence (Dang et al., 2022), seven were excluded due to setting, six due to wrong study design, three for wrong intervention, and one for lack of full-text availability. After the final screening, seven studies remained for full-text abstraction and review. Literature Synthesis There is strong and consistent evidence demonstrating the benefits of nurse residency programs (NRPs) for improving new graduate nurse transition, readiness, satisfaction, and retention compared to standard orientation models (Ackerson & Stiles, 2018; Asber, 2019; Eckerson, 2018; Miller et al., 2023) Multiple literature reviews and integrative analyses of existing studies have concluded that NRPs increase retention rates, confidence, competence, and professional integration for new nurses during the transition from academia to clinical practice (Ackerson & Stiles, 2018; Eckerson, 2018) Studies show that extended residency programs of 12 months or more have higher retention outcomes compared to shorter programs (Asber, 2019; Van Patten & Bartone, 2019; Wildermuth et al., 2020) In particular, commercial NRPs, like Vizient, tend to yield the highest retention rates ranging from 74-100%, surpassing national averages for new nurses of 50-61% (Asber, 2019). Qualitative reports provide insight into new nurse perceptions, highlighting the value of peer support, preceptored experiences, and a sense of belonging within an NRP to facilitate role transition (Fowler et al., 2018). Quasi-experimental studies demonstrate that nurse residents significantly improve clinical readiness, retention perceptions, and job satisfaction more than nurses undergoing standard orientation without a structured residency program (Miller et al., 2023). These quantitative findings reinforce the themes identified in qualitative studies. 26 Overall, there is substantial evidence that NRPs improve transition to practice and retention across diverse methodologies, including literature reviews, qualitative discoveries, and quantitative comparisons. However, continued research is warranted using more robust experimental designs, diverse settings, larger sample sizes, and long-term follow-up. There is an opportunity to build on this knowledge base through rigorous program evaluation at individual organizations to guide quality improvements and justify the return on investment of NRPs. Indicators Data Collection Several critical indicators were used to evaluate the outcomes of the nurse residency program. Self-reported competence levels and satisfaction ratings were gathered through pre- and post-program surveys administered to nurse residents using the validated Casey-Fink Graduate Nurse Experience tool (Casey et al., 2021). This survey allowed a comparison of residents’ perceptions at the beginning and end of the program. Retention rates at 1-, 2-, and 3- years post-program completion were calculated for each cohort by tracking whether participants remained employed using internal program records. Additionally, program completion rates were determined by examining program records to identify the percentage of enrolled nurse residents in each cohort who successfully finished all requirements. Collection Qualitative Qualitative data were collected through focus groups, virtual meetings, and interviews with crucial nurse residency program stakeholders. Two 60-minute in-person focus group sessions were conducted with clinical specialists, nurse managers, and unit educators; six to eight participants from each stakeholder group were recruited. Additionally, two 120-minute virtual drop-in meetings were held for preceptors and nurse residents using video conferencing. These informal meetings did not garner any participation. As a result, informal rounding on various inpatient nursing units targeted current nurse preceptors, past nurse residents, and 27 current nurse residents. These informal interviews included seven former nurse residents, eight nurse preceptors, and three individuals who were both former nurse residents and current preceptors. Additionally, 30-minute semi-structured interviews were performed with the nurse resident program coordinator, nursing professional development and education administrator, and nursing administrators. All sessions employed pre-determined questions related to the program evaluation, with follow-up probing to gather additional insights. Discussions and interviews were recorded and transcribed, with participant confidentiality maintained. The qualitative data collection through focus groups, virtual meetings, and interviews elicited valuable perspectives about the nurse residency program’s strengths, weaknesses, effectiveness, and opportunities for improvement from diverse stakeholder lenses. Quantitative Quantitative data was collected from multiple sources to evaluate nurse residency program outcomes. Pre- and post-program surveys were administered to residents using the Casey-Fink Graduate Nurse Experience Survey to gather self-reported data on confidence levels across various clinical domains and satisfaction with the overall program (Casey et al., 2021). Descriptive statistics were used to analyze the survey results. Program records were reviewed to calculate retention rates at 1, 2, and 3 years after completion for each cohort of residents who participated in the nurse residency program. Any attrition and reasons for separation were tracked. Additionally, program completion rates showing the percentage of enrolled nurse residents who successfully finished all program requirements were determined for each cohort based on program records. The quantitative data analysis using descriptive statistics provided insights into critical outcomes like self-reported competence levels, satisfaction, retention rates, and program completion. 28 Timeline The formal project proposal was submitted in September 2023 to obtain the required approval for the evaluation. Baseline data analysis of records from previous nurse residency cohorts began in September 2023 to gather background metrics. In October 2023, focus group sessions were held with clinical specialists, nurse managers, and unit educators to elicit their perspectives. Additionally, virtual drop-in meetings with preceptors took place in early November 2023, while meetings for nurse residents occurred in mid-November 2023. In-person rounding to collect information from nurse residents and preceptors occurred in early December 2023. Comprehensive data analysis was performed from December 2023 to January 2024 to integrate the qualitative and quantitative findings. Preliminary results were shared with key stakeholders in March 2024 for member-checking to validate interpretations. Finally, the program evaluation report was disseminated to all stakeholders in April 2024. This timeline allowed for a phased approach to systematic data collection, analysis, interpretation, and reporting to facilitate a rigorous program evaluation. Analysis & Interpretation Analysis Nurse Resident Program Completion Rate Program record evaluation of all cohorts enrolled from 2017 through 2023 provides critical insights into the progression and outcomes of new graduate nurses (NGNs) transitioning into clinical practice. Throughout this period, 865 NGNs completed the program, yielding an overall completion rate of 81%. Notably, 9% of the participants did not complete the program, 7% withdrew before the start date, and 3% were Department of Emergency Medicine (DEM) withdrawals (Vizient Inc., 2024). The latter group of DEM nurses was explicitly removed from the program between late 2022 and early 2023 due to changes in departmental training requirements and objectives. 29 Primary Evaluation Questions Retention & Turnover. Analysis of program records identified the top five reasons for nurse resident termination across all tenure groups (Figure 3). Within the first year, termination was driven by obtaining a position representing a new job experience, unsatisfactory performance, relocation, and family obligations (see Figure 1 for first-year termination data). Beyond the first year, additional factors like leaving for travel nursing positions and returning to school full-time dominated the reasons for termination (see Figures 4 and 5 for top turnover reasons in the 12-24 month and 24+ month tenure groups, respectively). Therefore, the program’s influence on nurse retention rates presented a complex picture. While the program appeared to positively impact 1-year retention rates, suggesting an effective bridge from academia to practice for new nurses, a deeper dive into the data revealed that longer-term retention saw diminishing returns, particularly beyond the first year. Self-Reported Competence. An examination of overall self-reported competence levels showed improvement in self-reported competence levels among new graduate nurses upon program completion (Figure 6). This improvement, however, while positive, did not reach the benchmark levels, suggesting that while the program effectively boosts confidence and competence, there is room for further enhancement. (Figure 6). Secondary Evaluation Questions 30 Focus Group and Interview Thematic Analysis. Focus groups and interviews with key stakeholders involved in the nurse residency program, including leadership, clinical nurse specialists, educators, and coordinators, yielded valuable insights. Several prominent themes emerged regarding areas where stakeholders perceived the program benefits nurse residents’, including role socialization and transition support (see Figure 2 for complete thematic analysis). However, needs were also identified as a perceived lack of clinical simulation, a lack of benefits related to the required evidence-based practice project, and a lack of relevance to practice for didactic content. Specific feedback related to program benefits includes: “my observations show that the structured support system in place is pivotal to helping with their (NGN) stress” and “the residency program fosters a sense of community and belonging among the new nurses, which is crucial for their identity development”. Conversely, discrete feedback related to program shortfalls includes: “adding more simulation experiences could potentially bridge the theory-practice gap, making the transition to real patient care smoother”, “there seems to be a disconnect between the evidence-based practice projects and the residents’ daily clinical experiences”, and “the didactic component would benefit from a closer alignment with practical skills, to better prepare residents for the demands of patient care”. 31 Nurse Resident and Clinical Preceptor Feedback Analysis. Feedback from nurse residents and clinical preceptors provides valuable insights into the strengths and areas for improvement within the nurse residency program. Analysis of clinical preceptor feedback, as depicted in Figure 8’s word cloud, underscores a strong emphasis on nurturing communication skills and support mechanisms for nurse residents. Preceptors critically assessed the practical application of the training, pointing out that while the program effectively builds interpersonal skills, it falls short of enhancing clinical competencies. They expressed concerns that much of the program’s content consists of “busy work,” which they perceived as not adding substantial value to the practical nursing skills needed in high-pressure environments. One preceptor said, “they (NGN) say it helps, but I don’t see any changes of benefits to their skills”. In contrast, feedback from nurse residents highlighted different priorities and experiences. The common themes extracted from their interviews and represented in Figure 9’s word cloud included stress, the need for effective communication channels, and the healthcare team’s socialization process. Despite these challenges, residents frequently acknowledged the critical role of support systems and the importance of maintaining work-life balance during their transition into clinical roles. This feedback suggests that while the residency program provides a foundation for professional growth, there is a significant need to enhance support structures that address new nurses’ emotional and practical challenges. These contrasting perspectives from preceptors and residents underline the necessity for a balanced approach in the residency curriculum that equally prioritizes the development of clinical skills and cultivating a supportive learning environment. Addressing these concerns could lead to a more holistic development of nurse residents, equipping them with the tools necessary to perform their duties effectively and thrive in the demanding nursing field. Interpretation In summary, integrating the qualitative and quantitative data provides insights into areas where the nurse residency program shows positive impacts, along with identifying improvement 32 opportunities to facilitate nurse residents’ transition to practice better. The qualitative feedback from stakeholders such as nurse residents, preceptors, educators, and leadership shed light on the perceived strengths of the program, like building communication skills, assisting with role socialization, and providing a support system. However, it also highlighted the need to increase the amount of clinical skills training, aligning content with crucial practice issues and revealing the need to reimagine the required evidence-based practice project. Examination of the quantitative survey and program records data uncovered mixed results. While the program improved confidence levels, concerns about causing harm persisted, and overall confidence remained below benchmark levels. This may indicate that although the program improved confidence, it did not adequately alleviate harm-related concerns. Despite increased confidence levels, the persistence of harm-related concerns suggests a gap in effectively translating knowledge into safe, confident practice. This disparity might be addressed by integrating more scenario-based training and simulations focusing on critical thinking and decision-making in high-pressure situations. The Casey-Fink survey data (Casey et al., 2021; Vizient Inc., 2024) analysis revealed mixed outcomes related to nurse residents’ transition difficulties (Figure 7). However, problems related to fear of harm increased throughout the program compared to the benchmark. This indicates that although the program improved confidence, it did not adequately alleviate harm-related concerns. The increasing fear of harm despite program progression points to possible areas for curriculum enhancement, particularly in strengthening the components that support clinical judgment and error prevention. Introducing more comprehensive assessments of clinical decision-making throughout the program could help identify learning gaps and provide targeted educational interventions. The program positively impacted 1-year retention but had less success with longer-term retention. Analysis of termination reasons provided valuable insights into factors driving turnover at different stages after program completion. Further exploration into the specific 33 reasons for turnover highlights several factors, including workload, work-life balance, and a perceived lack of ongoing professional development opportunities. Addressing these issues through structured mentorship programs extending beyond the first year and offering more continuous professional development opportunities could improve long-term retention rates. Engaging diverse stakeholders and utilizing multifaceted data allows for a comprehensive and holistic evaluation of the program’s effectiveness. The insights gained provide evidence- based guidance to inform targeted improvements to enhance outcomes for nurse residents as they transition into professional practice. Continued evaluation and adaptation focused on identified gaps will be instrumental in optimizing the program’s impacts moving forward. Dissemination Dissemination & Use The dissemination of the evaluation findings was comprehensive and inclusive, ensuring that all stakeholders and the broader nursing education community were well-informed. A detailed report that outlined the evaluation’s methodology, findings, and recommendations was shared with the hospital’s nursing leadership and all relevant stakeholders. Supplementary materials such as infographics and executive summaries were created to make the findings accessible to a broader audience. These materials facilitated a deeper understanding and engagement with the findings, encouraging dialogue and collaborative efforts toward future program enhancements. Additionally, the findings were presented at academic and professional conferences, contributing to the broader discussion on effective transition-to-practice programs for nurses. Use Integrating the evaluation findings has enhanced the nurse residency program at this Midwestern hospital. The combination of qualitative and quantitative data provided a robust foundation for identifying areas for improvement. An action plan was developed with key stakeholders, including the program coordinator and nursing education administrator, targeting 34 these areas. Efforts focused on enriching clinical simulation experiences and aligning the didactic content more closely with the practical demands of nursing. The evidence-based practice project component was reviewed for its relevance and impact on the residency experience. Additionally, the plan outlined strategies to improve preceptor education and engagement. The implementation of this action plan marked the beginning of an ongoing improvement cycle. This approach ensured that the residency program remained responsive to the evolving needs of new graduate nurses, facilitating their successful transition into the professional nursing environment. By embedding a continuous feedback loop, this iterative process addresses immediate gaps and positions the program for proactive adaptations, reflecting changes in clinical practices and emerging healthcare challenges. This dynamic adjustment mechanism is crucial for maintaining the program’s relevance and effectiveness, ensuring it meets the standards of excellence expected in contemporary nursing education. 35 References Ackerson, K., & Stiles, K. A. (2018). Value of nurse residency programs in retaining new graduate nurses and their potential effect on the nursing shortage. Journal of Continuing Education in Nursing, 49(6), 282–288. https://doi.org/10.3928/00220124-20180517-09 Alsalamah, Y. S., Al Hosis, K., Al Harbi, A., Itani, M. S., El Tassi, A., & Fawaz, M. (2022). Student to nurse transition and the nurse residency program: A qualitative study of new graduate perceptions. Journal of Professional Nursing, 42, 195–200. https://doi.org/10.1016/j.profnurs.2022.07.007 American Nurses Credentialing Center. (2023). Practice transition accreditation program: 2024 application manual. Asber, S. R. (2019). Retention outcomes of new graduate nurse residency programs: An integrative review. JONA: The Journal of Nursing Administration, 49(9), 430–435. https://doi.org/10.1097/NNA.0000000000000780 Brown, J. D., Hart, L., & Wludyka, P. (2022). Improving Practice Readiness Among Nurse Residents. Journal of Continuing Education in Nursing, 53(9), 411–416. https://doi.org/10.3928/00220124-20220805-07 Cadmus, E., & Roberts, M. L. (2022). First year outcomes: Program evaluation of a statewide nurse residency program. JONA: The Journal of Nursing Administration, 52(12), 672– 678. https://doi.org/10.1097/NNA.0000000000001230 Casey, K., Tsai, C. L., & Fink, R. M. (2021). A Psychometric Evaluation of the Casey-Fink Graduate Nurse Experience Survey. JONA: The Journal of Nursing Administration, 51(5), 242–248. https://doi.org/10.1097/NNA.0000000000001008 Centers for Disease Control and Prevention. (1999). Framework for Program Evaluation in Public Health. MMWR, 48(No. RR-11). 36 Chant, K. J., & Westendorf, D. S. (2019). Nurse residency programs: Key components for sustainability. Journal for Nurses in Professional Development, 35(4), 185–192. https://doi.org/10.1097/NND.0000000000000560 Dang, D., Dearholt, S., Bissett, K., Ascenzi, J., & Whalen, M. (2022). Johns Hopkins evidence- based practice for nurses and healthcare professionals: Model and guidelines (4th ed.). Sigma Theta Tau International. Drexler, D., Cornell, D., Cherrie, C., Consolo, C., & Doonan, R. L. (2023). Reducing Staff Turnover and Clinician Burnout With a Structured Support Group During the COVID-19 Pandemic. Nursing Administration Quarterly, 47(1), 31–40. https://doi.org/10.1097/NAQ.0000000000000566 Eckerson, C. M. (2018). The impact of nurse residency programs in the United States on improving retention and satisfaction of new nurse hires: An evidence-based literature review. Nurse Education Today, 71, 84–90. https://doi.org/10.1016/j.nedt.2018.09.003 Fowler, S. B., Lind, S. C., Johnson, P. R., & Lewis, J. (2018). Qualitative description of new graduate nurses’ experiences in a nurse residency program. Journal for Nurses in Professional Development, 34(6), 319–324. https://doi.org/10.1097/NND.0000000000000496 Halar, J., Didic, S., Amare, E., Labrecque, L., & Rowan, M. (2020). The use and benefits of evaluation framework modules at the Canadian Foundation for Healthcare Improvement: Engaged capacity building and collaborative program evaluation planning. Canadian Journal of Program Evaluation, 35(2), 258–267. https://doi.org/10.3138/CJPE.69230 Harper, M. G., Bodine, J., & Monachino, A. (2021). The effectiveness of simulation use in transition to practice nurse residency programs. Journal for Nurses in Professional Development, 37(6), 329–340. https://doi.org/10.1097/NND.0000000000000787 Henry Ford Health continuing education - About us. (2023). https://hfhs.cloud- cme.com/about/?p=25 37 Henry Ford Health: Mission, vision, values. (2023). https://www.henryford.com/about/culture/mission-vision-values Henry Ford Health System 2018 annual report. (2018). https://www.henryford.com/about/annual-report/2018-annual-report Henry Ford innovations. (2023). https://www.henryford.com/innovations Letourneau, R. M., & Fater, K. H. (2015). Nurse residency programs: An integrative review of the literature. Nursing Education Perspectives, 36(2), 96–101. https://doi.org/10.5480/13-1229 Michel, A., Ryan, N., Mattheus, D., Knopf, A., Abuelezam, N. N., Stamp, K., Branson, S., Hekel, B., & Fontenot, H. B. (2021). Undergraduate nursing students’ perceptions on nursing education during the 2020 COVID-19 pandemic: A national sample. Nursing Outlook, 69(5), 903–912. https://doi.org/10.1016/j.outlook.2021.05.004 Miller, C. M., Meyer, K., Riemann, L. A., Carter, B. T., & Brant, J. M. (2023). Transition into practice: Outcomes of a nurse residency program. The Journal of Continuing Education in Nursing, 54(1), 32–39. https://doi.org/10.3928/00220124-20221207-08 NSI Nursing Solutions. (2018). 2018 NSI national health care retention & RN staffing report. NSI Nursing Solutions. (2019). 2019 NSI national health care retention & RN staffing report. NSI Nursing Solutions. (2020). 2020 NSI national health care retention & RN staffing report. NSI Nursing Solutions. (2021). 2021 NSI national health care retention & RN staffing report. NSI Nursing Solutions. (2022). 2022 NSI national health care retention & RN staffing report. NSI Nursing Solutions. (2023). 2023 NSI national health care retention & RN staffing report. www.nsinursingsolutions.com. Okul, E. O., & Nyonje, R. O. (2020). Examining stakeholder involvement in the evaluation process for program improvement. International Journal of Research in Business and Social Science (2147- 4478), 9(5), 179–191. https://doi.org/10.20525/ijrbs.v9i5.835 38 Peacock, A. (2023). Compassion satisfaction, compassion fatigue, and vicarious trauma. Nursing Management, 54(1), 14–22. https://doi.org/10.1097/01.NUMA.0000905000.95966.96 Rush, K. L., Janke, R., Duchscher, J. E., Phillips, R., & Kaur, S. (2019). Best practices of formal new graduate transition programs: An integrative review. In International Journal of Nursing Studies (Vol. 94, pp. 139–158). Elsevier Ltd. https://doi.org/10.1016/j.ijnurstu.2019.02.010 Savitsky, B., Radomislensky, I., & Hendel, T. (2021). Nurses’ occupational satisfaction during Covid-19 pandemic. Applied Nursing Research, 59, 151416. https://doi.org/10.1016/j.apnr.2021.151416 Shinners, J., Africa, L., Mallory, C., & Durham, H. (2021). Versant’s nurse residency program: A retrospective review. Nursing Economic$, 39(5), 239–246. Simpson, D., Riddle, J. M., Hamel, D. L., & Balmer, D. F. (2020). Blueprinting Program Evaluation Evidence Through the Lens of Key Stakeholders. Journal of Graduate Medical Education, 12(5), 629–630. https://doi.org/10.4300/JGME-D-20-01015.1 Smith, A. L. (2021). Evidence-based practice training in nurse residency programs: Enhancing confidence for practice implementation. Teaching and Learning in Nursing, 16(4), 315– 320. https://doi.org/10.1016/j.teln.2021.05.005 Van Camp, J., & Chappy, S. (2017). The effectiveness of nurse residency programs on retention: A systematic review. AORN Journal, 106(2), 128–144. https://doi.org/10.1016/j.aorn.2017.06.003 Van Patten, R. R., & Bartone, A. S. (2019). The impact of mentorship, preceptors, and debriefing on the quality of program experiences. Nurse Education in Practice, 35, 63–68. https://doi.org/10.1016/j.nepr.2019.01.007 Vizient AACN nurse residency program. (2023). https://www.vizientinc.com/what-we- do/operations-and-quality/vizient-aacn-nurse-residency-program 39 Vizient Inc. (2024). Nurse Residency Program Data. Wildermuth, M. M., Weltin, A., & Simmons, A. (2020). Transition experiences of nurses as students and new graduate nurses in a collaborative nurse residency program. Journal of Professional Nursing, 36(1), 69–75. https://doi.org/10.1016/j.profnurs.2019.06.006 Wilson, T., Weathers, N., & Forneris, L. (2018). Evaluation of outcomes from an online nurse residency program. JONA: The Journal of Nursing Administration, 48(10), 495–501. https://doi.org/10.1097/NNA.0000000000000657 Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). SAGE Publications. Zamboni, K., Baker, U., Tyagi, M., Schellenberg, J., Hill, Z., & Hanson, C. (2020). How and under what circumstances do quality improvement collaboratives lead to better outcomes? A systematic review. In Implementation Science (Vol. 15, Issue 1). BioMed Central Ltd. https://doi.org/10.1186/s13012-020-0978-z Table 1: Stakeholder Assessment Henry Ford Hospital’s Nurse Residency Program Stakeholder Assessment, Description, and Categorization Stakeholder Role Responsibilities Category Program Coordinator Oversees the nurse residency program, coordinates program activities, and ensures program objectives are met. Responsible for successfully planning, implementing, and evaluating the nurse residency program. Internal stakeholders with high influence and high investment. Nursing Education Administrator Provides oversight and guidance for nursing education programs, including the nurse residency program. Ensures alignment of the nurse residency program with educational standards and best practices. Internal stakeholders with high influence and high investment. Role in Evaluation Plays a key role in providing insights into program strengths, areas for improvement, and potential strategies for program enhancement. Interests & Perspectives Interested in program effectiveness, participant satisfaction, and program completion. Views the program from an administrative perspective. Interested in the educational quality and outcomes of the nurse residency program. Views the program from an educational perspective. Provides expertise and guidance in evaluating the program’s educational aspects, curriculum adherence, and effectiveness. Engagement Strategies Engage the program coordinator through regular meetings and discussions, seek their input on evaluation goals and objectives, and involve them in designing and implementing the evaluation plan. Engage the nursing education administrator by involving them in evaluation planning, seeking their input on educational aspects, and leveraging their expertise in evaluating the Stakeholder Role Responsibilities Category Interests & Perspectives Role in Evaluation Support nurse residents' clinical education and development in their respective specialty areas and across the organization. Internal stakeholders with high influence and high investment. Interested in the clinical competency, education, and development of nurse residents in their specialty areas. Contribute insights into the program’s impact on clinical skills, specialty-specific training, and resident performance. Views the program from a clinical practice and educational perspective. 41 Engagement Strategies program’s educational outcomes. Engage unit- based clinical nurse specialists by involving them in the evaluation process, seeking their feedback on specialty- specific training, and leveraging their expertise in evaluating the program’s clinical outcomes. Supports the nurse residency program through resource allocation and decision- making. Internal stakeholders with high influence and relatively lower investment. Interested in the program’s alignment with organizational goals, resource allocation, and overall sustainability. Plays a key role in providing resources and support for implementing evaluation recommendations and aligning the program with Engage nursing administration by regularly communicating evaluation progress and findings, seeking their input on Unit-based Clinical Nurse Specialists Nursing Administration Serve as experts and resources in specialized areas of clinical nursing practice within the hospital units. Share responsibility for nurse resident professional development. Also function as facilitators and evidence-based practice experts for nurse residency program. Provides overall leadership and strategic direction for nursing services in the organization. Stakeholder Role Responsibilities Category Nurse Managers Oversee the daily activities of nurse residents and share responsibility for their professional development. Provide guidance, support, and supervision to nurse residents in their assigned units. Internal stakeholders with high investment and relatively lower influence. Role in Evaluation organizational priorities. Interests & Perspectives Views the program from a leadership and organizational perspective. Contribute insights into the program’s impact on resident performance, skill development, and unit dynamics. Interested in the program’s impact on resident performance, professional development, and unit dynamics. Views the program from a managerial perspective. Unit-based Nurse Educators Provide education and training to nurse residents within specific units and share responsibility for their professional development. Deliver unit- specific educational content, support skill development, and provide mentorship to nurse residents. Internal stakeholders with high investment and relatively lower influence. Interested in the educational outcomes of nurse residents within their respective units. Provide insights into nurse residents' educational outcomes and skill development within their units. Views the program from an educational 42 Engagement Strategies resource allocation, and demonstrating the program’s value in achieving organizational goals. Engage nurse managers through regular communication, seek their feedback on resident performance and skill development, and involve them in discussions about program enhancements. Engage unit- based nurse educators through regular communication, seek their input on educational resources and training, and involve them in Stakeholder Role Responsibilities Category Nurse Residents Newly graduated nurses participating in the residency program. Engage in the program to transition from academia to professional nursing practice. Primary stakeholders with high investment and relatively lower influence. Patients Individuals receiving care from nurse residents in the clinical setting. Benefit from the care provided by nurse residents under supervision. External stakeholders with relatively lower investment and influence. Interests & Perspectives and mentorship perspective. Role in Evaluation Provide feedback on their experience, learning outcomes, and challenges during the residency program. Potentially provide insights into the impact of nurse residents’ care on patient outcomes and experiences. Interested in gaining clinical skills, professional development, and successful completion of the residency program. Views the program from a participant’s perspective. Interested in receiving safe and high-quality care from nurse residents. Views the program from a patient’s perspective. 43 Engagement Strategies the evaluation process to gather their perspectives on resident progress. Engage nurse residents through regular surveys, focus groups, and individual interviews to gather their perspectives, assess their satisfaction, and understand their needs. Engage patients through surveys, feedback mechanisms, or patient advisory groups to assess their satisfaction, gather their perspectives on the care received, and identify areas Stakeholder Role Responsibilities Category Interests & Perspectives Role in Evaluation Unit-based Clinical Preceptors Experienced nurses assigned to supervise and guide nurse residents in the clinical setting. Provide nurse residents mentorship, guidance, and support during their clinical rotations. Internal stakeholders with high investment and influence. Contribute insights into nurse residents' clinical development, progress, and performance. Interested in the professional development, clinical competency, and successful integration of nurse residents into the unit. Views the program from a preceptor’s perspective. 44 Engagement Strategies for improvement. Engage unit- based preceptors through regular communication, seek their feedback on resident progress, and involve them in evaluating the program’s clinical outcomes. 45 Table 2: Project Description Nurse Residency Program Description Resources Activities Outputs Outcomes Funding Program coordinator Facilitators Didactic instruction on best practices and cultural considerations Skill-building workshops and simulations Evidence-based practice implementation project Regular facilitator debrief sessions Content experts Clinical facilities for training Curriculum and training materials Number of participants successfully completing the program Participant program satisfaction scores Skills/knowledge obtained during the program Short-term Mid-term Long-term Increased confidence levels in clinical abilities Enhanced critical thinking and problem- solving abilities Improved patient outcomes and quality of care Improved communication and teamwork skills Enhanced knowledge of evidence-based nursing practices Increased proficiency in patient assessment and care planning Increased retention rates of new nurses Continued professional development and career advancement Table 3: Program Evaluation Standards Standards: Utility, Feasibility, Propriety, Accuracy, Evaluation Accountability (adapted from Yarbrough et al., 2011) Standard Category & Designation Description 46 Utility U1 U2 U3 U4 U5 U6 U7 U8 Feasibility F1 F2 F3 F4 Propriety P1 P2 P3 P4 P5 P6 P7 Accuracy Evaluator credibility Attention to stakeholders Negotiating purposes Explicit values Relevant information Meaningful processes and products Timely and appropriate communicating and reporting Concern for consequences and influence Project management Practical procedures Contextual viability Resource use Responsive and inclusive orientation Formal agreements Human rights and respect Clarity and fairness Transparency and disclosure Conflicts of interest Fiscal responsibility 47 A1 A2 A3 A4 A5 A6 A7 A8 Justified conclusions and decisions Valid information Reliable information Explicit program and context descriptions Information management Sound designs and analyses Explicit evaluation reasoning Communication and reporting Evaluation accountability E1 E2 E3 Evaluation documentation Internal metaevolution External metaevolution Figure 1: First-Year Reasons for Termination Top Reasons for Termination, 0-12 Month Tenure Group, All Cohorts, 2017-2023 48 Figure 2: Thematic Analysis Clinical Nurse Specialists, Unit Educators, Nurse Managers/Assistant Managers (Focus Groups) & Nursing Administrators, NRP Coordinator (Interviews) 49 For a more detailed look at this thematic analysis, click here for a poster-sized version. Figure 3: Top Reasons for Termination Top 5 Reasons for Termination by Tenure Group, All Cohorts, 2017-2023 50 Figure 4: Second-Year Reasons for Termination Top Reasons for Termination, 12-24 Month Tenure Group, All Cohorts, 2017-2023 51 Figure 5: Reasons for Termination Beyond Two Years of Tenure Top Reasons for Termination, 24+ Month Tenure Group, All Cohorts, 2017-2023 52 Figure 6: Participant Competence Overall Participant Self-Reported Competence by Survey Period, All Cohorts, 2017-2023 53 Figure 7: Transition Difficulties Self-Reported Transition Difficulties Comparison to Benchmark, All Cohorts, 2017-2023 54 Figure 8: Preceptor Feedback World Cloud 55 Figure 9: Nurse Resident Feedback World Cloud 56 Limitations Weak methods (one database searched, limited details on methods, potential positive finding bias) Weak methods (small sample size, no control group, tool variability, potential self- report data bias) 57 LOE/Quality Level III-B Relevance Provides evidence that NRPs improve NGN retention and satisfaction Level III-B Provides evidence that commercially developed programs tend to have higher retention rates Provides benchmark retention rates Appendix A: Literature Synthesis Author/Citation Methods/Type Main Findings NRPs improve Ackerson, K., & NGN retention Stiles, K. A. and satisfaction (2018). Literature review Outcome(s) Retention rates Job satisfaction Asber, S. R. (2019). Literature review NGN retention rates Factors impacting retention NRPs help address nursing shortage by improving retention NRP participants had nigher retention rates Retention rates were higher in NRPs > 6 months and lower in NRPs < 6 months Commercially developed NRPs showed higher retention rates compared to organization developed programs Eckerson, C. M. (2018). Literature review NRPs have a positive impact on NGN retention and satisfaction Retention Job satisfaction Turnover costs Weak methods (small sample size, tool variability, no control, Reviews evidence showing positive impacts of NRPs on retention and satisfaction Level III-B Author/Citation Methods/Type Main Findings Outcome(s) 58 LOE/Quality Limitations potential positive finding bias) Relevance compared to standard orientation Qualitative Fowler, S. B., Lind, S. C., Johnson, P. R., & Lewis, J. (2018). Quasi- experimental Miller, C. M., Meyer, K., Riemann, L. A., Carter, B. T., & Brant, J. M. (2023). Retention is higher with NRPs compared to general orientation Financial benefits from reduced turnover NRPs provide support, improve teamwork and belonging NGN values peer connection and support in NRPs NRPs improve readiness for practice, retention, and satisfaction NRPs reduce turnover compared to standard orientation Support 12- month NRP model Qualitative themes Weak methods (small sample size, no comparison, not longitudinal) Highlights value of peer connections and support in NRPs Level III-B Readiness for practice Retention Job satisfaction Weak methods (non- randomized, smaller control group) Level II-B NRPs improve readiness to practice, retention, and satisfaction compared to standard orientation Provides a methodology for a comparison group Author/Citation Methods/Type Main Findings Van Patten, R. R., & Bartone, A. S. (2019). Qualitative Transition to practice improved with NRP Wildermuth, M. M., Weltin, A., & Simmons, A. (2020). Qualitative, phenomenolo- gical Extending support is beneficial beyond 12 month Precepted experiences are important to nurse residencies More precepted time improved readiness for practice 59 LOE/Quality Level III-B Outcome(s) Transition experiences with an extended residency Limitations Weak methods (single site, small sample size, potential selection bias) Relevance Suggests extended NRP support past 12-months may further improve transition experiences Precepted hours Readiness for practice Weak methods (single site, non- validated tool, small sample size, potential self-report data bias) Emphasizes the importance of precepted experiences for improving readiness to practice Level III-A 60 Appendix B: Stakeholder Assessment Matrix Investment High Low Program Coordinator Nursing Administration Influence High Nurse Education Administration Unit-Based Clinical Nurse Specialists Nurse Managers Unit-Based Preceptors Unit-Based Nurse Educators Low Nurse Residents Patients Appendix C: Logic Model 61