Improved Pediatric Immunization Administration Process Within a Primary Care Setting Rachel N. Price and Lindsay N. Runft College of Nursing, Michigan State University NUR 997: Doctor of Nursing Practice Project III Dr. Dontje 1 Abstract Background and review of the literature: Immunizations are essential health care measures that protect patients from serious and life threatening illnesses. Vaccine errors can cause inadequate immunity, patient injury, increased cost, inconvenience, and reduced trust in the healthcare system. Available literature suggests the utilization of educational interventions in conjunction with simulation to reduce vaccine error rates within the clinical setting. Purpose: The purpose of this Quality Improvement (QI) project was to develop an educational experience to reduce the vaccine administration error rate at an urban midwestern university pediatric clinic. Methods: The project was implemented at an urban midwestern university pediatric clinic. The Doctor of Nursing (DNP) students created an educational presentation and low-fidelity simulation. Implementation Plan/Procedure: A pre-recorded narrated educational series was distributed to all staff and providers involved in the administration process. After completion of the educational portion of the intervention, a limited number of administering staff were provided with a low-fidelity simulation experience. Pre/post knowledge questionnaires and confidence surveys were utilized to measure the impact of educational series and low-fidelity simulation. Implications/Conclusions: Numerous project limitations were encountered during implementation and evaluation. However, the project has potential to improve error rates within the pediatric clinic setting. Keywords: Vaccine, Immunization, Error, Education, Simulation 2 Significance of Immunization Administration Errors in Primary Care Development of routine vaccinations have considerably reduced diseases that once frequently harmed or killed many infants, children, adolescents, and adults across the world (Centers for Disease Control and Prevention [CDC], 2022). There are currently 17 vaccine-preventable diseases that the CDC recommends being vaccinated against (Kroger et al., 2023). Without the proper administration of these vaccines, people can still become dangerously ill or die. The CDC estimates that 4 million lives are saved by childhood vaccinations each year worldwide (CDC, 2023). Vaccinations are given in a variety of settings including primary care offices, pediatric offices, inpatient facilities, and more. Safe and proper administration of vaccines is essential in vaccine efficacy and patient safety (CDC, 2021). In order to prevent errors, healthcare professionals who administer these vaccines should be educated on their indications, timing, proper administration, and potential errors that may take place during the administration process. Errors Vaccine-related errors continue to pose significant patient safety concerns within the outpatient setting. The Institute for Safe Medication Practices National Vaccine Errors Reporting Program (ISMP National Vaccine Errors Reporting Program [ISMP VERP], 2022) reported 1,440 vaccine-related events between June 2020 and December 2021. During this time, 68% of the reported vaccine events were related to the coronavirus disease 2019 (COVID-19) vaccines, and for that reason were excluded from ISMP VERP analysis. The remaining reports showed that most vaccination errors occur in outpatient settings; 49% occurred in medical clinics, 20% in doctors offices, 11% in public health immunization clinics, and 9% in community pharmacies (ISMP VERP, 2022). 42% of the events involved registered nurses or nurse practitioners, and 34% involved medical assistants (ISMP VERP, 2022). Among the analyzed vaccine errors, wrong vaccine, expired vaccine or contamination/deterioration, wrong age, extra dose, and 3 wrong dose were the most commonly reported, which aligns with the previous data gathered by ISMP VERP in 2017. A systematic review done by Morse-Brady & Marie Hart (2020) found that incorrect vaccine and off-schedule administration were the most common error types which aligns with information from the ISMP VERP. According to a longitudinal cohort study conducted over 12 years at a large healthcare system by Reed et al. (2019), the majority of vaccine errors affected those between birth and two years of age. Between birth and two years of age, influenza vaccine was the most common type associated with error (Reed et al., 2019). Increased medication errors taking place within this age group are proposed to be related to the volume of vaccines given during childhood. Marufu et al. (2022) also note the complexity of medication therapy due to specific age and weight based dosing. Staff education Medical errors usually represent failures in the design of systems that are in place to prevent them. Systems where protocols and education programs are complex in nature and fail to address active and latent errors that may be present are likely to be less successful (Agency for Healthcare Research and Quality, 2019). Key factors in the prevention of active medical errors that take place include: utilizing a step-wise approach to prevent missing areas of implementation, minimizing workarounds, and removing variation within protocols or education (Agency for Healthcare Research and Quality, 2019). Prevention of latent errors encompasses ongoing monitoring and revision of design elements and how medical staff interact within the system (Agency for Healthcare Research and Quality, 2019). Staff education and training is an essential component of safe vaccine administration. Multiple studies have shown increased confidence among those that have participated in a vaccine training program (Lin et al., 2018; McKeirnan et al., 2018). Guidelines for vaccine administration by Kroger et al. (2023) details important steps within the administration process 4 including vaccine schedules, administration technique, contraindications, storage, and more. These guidelines should be implemented with any vaccine training program as supported by the CDC. In addition to these guidelines, the CDC (2018) also offers a training module on vaccine administration where they recommend vaccine administration be a part of new employee training and annual education requirements. Problem Description The vaccination process encompasses several steps within most primary care clinics. These steps include: prescribing, dispensing, preparation, adequate patient Michigan Care Improvement Registry (MCIR) review, immunization handling and storage, vaccine administration, scheduling of vaccine doses, monitoring of precautions and contraindications, management of vaccine side effects, reporting of suspected side effects, communication of vaccine benefits and risks, as well as reviewing standards for immunization practices for children, adolescents and adults (Poiraud et al., 2023). An error has the potential to occur at any step of the vaccine process. The National Coordinating Council for Medication Error Reporting and Prevention (2024) defines a vaccine administration error as any preventable event that may lead to improper medication use or cause patient harm. Vaccine related errors can have many ramifications. These ramifications can include but are not limited to: insufficient immunological protection, patient injury, financial cost, inconvenience, and diminished trust in the healthcare system. Health care professionals and clinics that offer and provide vaccines play an important role in considering the varying steps that are integral to the vaccine administration process. Vaccine administration errors at an urban midwestern university pediatric clinic prompted a root cause analysis to identify the main cause of vaccine error (see Appendix A). Interviews with staff members including nurses and medical assistants at the pediatric clinic revealed that lack of 5 time, independent double checks, and lack of vaccine education may be contributing to vaccine errors. Leadership staff expressed concerns regarding vaccine name confusion, a lack of education, vaccine schedule knowledge, and homogeneous MCIR review process among providers. Additionally, an increased number of new staff members, and parents utilizing their own immunization schedule may be affecting the administration process. Vaccine storage and preparation areas are also lacking signage and easily accessible vaccine reference guides. Critical evaluation of vaccine processes and environmental analyses at the pediatric clinic revealed a general need to implement a uniform staff education program, vaccine storage organization, and posted reference guides. Between June 2018 and April 2023, the healthcare organization as a whole, including the pediatric site in which intervention implementation took place, incurred 28 total vaccine errors and 2 near misses. In those 28 errors, there were 4 wrong doses, 16 wrong vaccines, 3 vaccines given off schedule, 3 vaccine preparation errors, 1 wrong administration technique, and 1 vaccine documentation omission. The 2 near misses included ordering a vaccine for a patient with a known contraindication and scheduling a child outside of his window for vaccine administration. The most notable error type was incorrect vaccine administration and is the focus of this quality improvement project. Literature Review A literature search was conducted within the CINAHL and PubMed databases. The following search terms were utilized: medication* AND admin* AND (error* OR mistak* OR accident*) AND safety. All articles from 2013-July 2023 and the English language were considered. In addition, CINAHL articles were further restricted to research and peer reviewed articles. PubMed articles were restricted to free full text articles, meta-analyses, randomized controlled trials, and systematic reviews. An additional general search of the internet revealed 6 additional articles that met the above criteria. For a detailed breakdown, see Appendix B for the PRISMA table. The literature was investigated to evaluate the effect of educational interventions on the reduction of medication errors in healthcare settings that prescribe and administer varying medications and vaccines. Fourteen research studies were synthesized. Similarities and differences were analyzed. The articles included seven systematic reviews, two of which were also meta-analyses; one additional meta-analysis; two quasi-experimental studies; one descriptive analysis; and three quality improvement projects. Of the fourteen articles included and evaluated, seven were level (I) evidence, one was level (II) evidence, two were level (III) evidence, and four were level (V). Other key concepts of synthesized studies included level of evidence, variables, instruments, strengths, limitations, and implications. These concepts are shown in the synthesis of the literature table, see Appendix C. Of the reviewed articles several common themes were identified when comparing interventions to reduce medication errors. The interventions investigated fell into one of the following categories; combination educational interventions (which included multiple educational techniques within one intervention), e-learning, pharmacist-led educational interventions, interventions that emphasized level of control, and simulation interventions. A breakdown of interventions included in each study is displayed in Appendix D. The majority of studies synthesized looked at mutli-variable interventions and the impact they had on reducing medication errors. Of the six articles that investigated combination educational interventions (Marufu et al., 2022; Keers et al., 2014; Lapkin et al., 2016; Plutinská & Plevová, 2019; Durham et al., 2020; Khalil & Lee, 2018), they contained several similarities as well as some slight differences. Marufu et al. (2022) and Lapkin et al. (2016) both proposed educational program interventions with a multifaceted approach that included a combination of educational material and risk management strategies. Both articles demonstrated a multifaceted intervention to be more successful than a single intervention at reducing medication errors. All 7 interventions investigated by Marufu et al. (2022) and Lapkin et al. (2016) showed a reduction in medication errors but failed to describe the exact educational intervention process or duration of the educational program. While the results from Marufu et al. (2022) were statistically significant, the results concluded from Lapkin et al. (2016) were not. Both articles were systematic reviews, therefore level (I) evidence. A level (I) systematic review by Keers et al. (2014), and a level (V) descriptive review by Plutinská & Plevová (2019) displayed a similar multifaceted education intervention with homogeneous results. Both studies concluded that multifaceted educational interventions that included components of modules of education, medication reconciliation technology, pharmacist involvement, protocols and guidelines, support systems for clinical decision-making, and review of electronic health records could be useful in reducing medication errors. Both Plutinská & Plevová (2019) and Keers et al. (2014) interventions were geared to be implemented within a hospital setting. Both articles require further investigation using rigorous and standardized study designs to confirm evidence concluded from these particular studies. Plutinská & Plevová (2019) and Keers et al. (2014) failed to describe the exact educational intervention process or duration of the educational programs. No single intervention or combination of interventions could be statistically proven to decrease medication administration errors at the conclusion of both studies. Two level (V) quality improvement projects (Durham et al., 2020; Khalil & Lee, 2018), displayed a multifaceted educational program intervention that both included aspects of medication safety training and medication/vaccine checklists/guidelines. While the educational program by Durham et al. (2020) emphasized simulated experiences, Khalil and Lee (2018) emphasized a safety program that included lectures, case studies, and small group discussions. Both articles were single-site studies that took place in a primary care setting. Processes and new insight may be transferable to other primary care settings, however, more research is needed to confirm these results. 8 Two of the studies found during the review of the literature displayed e-learning interventions to reduce medication errors. Both level (I) systematic-umbrella review by Khalil et al. (2020) and quality improvement project by Anderson et al. (2020) identify the importance of an e-learning intervention in reducing medication errors. Anderson et al. (2020) implemented videos to staff members on quality standards for medication administration, infection control, patient identification, documentation and preventing falls. These videos were utilized as an educational tool. Improvement in patient safety and quality outcomes were seen, however, results were not statistically significant. Similarly, Khalil et al. (2020) investigated e-learning programs that not only incorporated videos, but also included interactive online simulation, slideshow presentations, and interactive (Compact Disk—Read-Only Memory) CD-ROM programs. All studies reported a significant outcome in favor of the intervention in regards to medication administration safety and skills. Duration and exact educational processes were not described. Two level (I) systematic reviews by Manias et al. (2020) and Jaam et al. (2021) showed statistically significant reductions in medication related errors with a pharmacist education intervention. Manias et al. (2020) intervention included comprehensive conversations about recent prescribing errors. These conversations were pharmacist led, and took place over three 10-min sessions per week over the intervention period of 4-weeks. Jaam et al. (2021) educational programs led by pharmacists involved lectures, posters, practical teaching sessions, audit and feedback method, and flash cards of high-risk abbreviations. All studies had educational sessions as part of their program, alone or in combination with other methods. Both systematic reviews showed that educational interventions provided to healthcare providers by pharmacists are effective at reducing medication error rates. Level (I) systematic review by Koeck et al. (2021) was the only article to emphasize level of control within intervention implementation to decrease medication errors. Koeck et al. (2021) observed eight interventions at a single point in the medication use process (administrative or 9 dispensing). The remaining studies investigated interventions at multiple stages in the medication use process. No clear cut conclusions were drawn regarding a specific intervention. However, it could be concluded that when designing interventions to reduce pediatric medication related errors, the hierarchy of controls model should be considered. In addition to utilizing the hierarchy of controls model, a focus should be placed on the introduction of higher-level controls. These controls may be more likely to reduce medication related errors than the administrative controls that are often seen in practice. Lastly, level (III) quasi-experimental design studies (Pol-Casteñeda et al., 2022, Sanko & Mckay, 2017), and meta-analysis including randomized and nonrandomized controlled trials (Lee, 2019) proposed comparable interventions and findings. Pol-Casteñeda et al. (2022) and Sanko and Mckay (2017) found similar results with simulation interventions in reduction of medication errors. Pol-Casteñeda et al. (2022) developed a simulation with three patient scenarios involving intravenous medication for the nursing students to participate in, each included a 15 minute intervention where the “nurse” simulated medication administration verifying each right in the medication process. Post surveys demonstrate that simulation appears useful and students were satisfied with the experience. Sanko and Mckay (2017) found similar conclusions in two cohorts of nursing students enrolled in pharmacology. One cohort served as a control. The other cohort received four manikin-based scenarios focusing on skills that aligned with safety competencies including calculations, high-alert medication procedures, hand hygiene, PPE, medication information searching, checking appropriate lab values, and vital signs prior to administration. The control group had a greater amount of adverse events, incorrect medication administrations, incorrect route, failure to check two forms of identification, problems with equipment, problems with administration records, events caused by knowledge deficits, and feelings of personal work overload. A reduction in medication errors were found to be statistically significant in the educational simulation intervention (Sanko and Mckay, 2017). 10 Lee (2019) concluded that statistically significant reductions in medication errors were seen in medical devices and simulation education interventions and should be considered for practice implementation. Various (medical device/simulation education) interventions were described among the 30 studies included in the meta-analysis. Variable outcome measures were utilized across these studies, however comparisons can be made. Anderson et al. (2020) found following implementation, there was a 34% reduction in medication incidents, an increase in staff awareness and identification of medication errors. However, this study did not reveal statistically significant outcomes. Similarly, Khalil and Lee (2018) reported an increase in staff knowledge regarding medications and confidence in applying learned material to practice. However, Durham et al. (2020) found that additional education and interventions were needed following the initial implementation to ensure new habits did not regress. Although, they do note zero errors occurred during and following implementation. Limitations including fewer number of vaccine administrations due to clinic size (Durham et al., 2020) and non-statistically significant results (Anderson et al., 2020) should be taken into account. In addition, all three studies (Andersen et al., 2020; Durham et al., 2020; Khalil & Lee, 2018) implemented a different intervention containing an educational component. Overall, these studies may show promise in educational interventions. However, due to their low level of evidence, lack of significance, blended results, and variable study designs, additional higher levels of evidence should be evaluated. While not all fourteen studies revealed statistically significant results, all fourteen studies were useful in the evaluation of educational interventions in the reduction of medication administration errors. Overall, these studies point to improved medication administration outcomes when educational interventions are utilized, specifically when simulation is a portion of the learning process. No two studies have identical study designs and there was a large amount of heterogeneity present, which makes translation into practice difficult. However, based 11 on the above findings it is important to ensure staff is appropriately trained and educated. Simulation can be an effective tool to ensure knowledge is retained and utilized in the practice setting to reduce medication administration errors. Rationale The Plan-Do-Study-Act (PDSA) Cycles were utilized to evaluate and explain the problem of vaccine errors. The PDSA Model for Improvement allows for a fluent process of frequent re-evaluation and improvement throughout the course of project design and implementation with greater flexibility in project implementation, evaluation, and improvement (Institute for Healthcare Improvement, 2017). During the planning phase, our intervention was chosen based on clinic leadership preference and the outcomes of our root cause analysis. The root cause analysis was performed and a fishbone diagram (Appendix A) was utilized to determine the cause of vaccine error. Conversations with staff members involved in administering vaccines revealed system barriers and concerns. New staff members; lack of detailed vaccine education, experience, knowledge, and visual aids; variation between MCIR review processes between providers, and a busy work environment were at the forefront of these concerns and barriers. These elements were emphasized when planning an intervention to decrease vaccine errors for staff involved in vaccine administration. Based on our root cause analysis and literature search, it was evident that education with simulation should reduce vaccine administration errors in the pediatric clinic and improve patient outcomes. A SWOT analysis was also utilized to evaluate how internal and external factors may impact project success (See Appendix E). Organizational strengths included motivated and supportive leadership; designated quality improvement staff; and a positive work environment. While weaknesses included staff and provider buy-in; consistency of staff; variable provider practices; and a busy clinic environment. Project and organizational opportunities included no current established vaccine training; quality improvement goals; and increasing patient/family 12 confidence in the health system. Finally, identified threats to project success were identified as confusing vaccine brand names. The goal of this quality improvement (QI) project was to develop an educational experience to reduce the vaccine administration error rate at an urban midwestern university pediatric clinic. Vaccine administration and vaccine schedule education were expected to decrease vaccine errors due to a current lack of vaccine knowledge and standardized entry-level staff training. By filling this gap, we expected the rate of vaccine errors to decrease, as well as staff confidence and patient safety to improve. Methods Context Medication administration errors are common occurrences and in some instances can be prevented. Factors considered when planning the intervention of this QI project surround the culture and work environment of the setting that implementation will take place in. The involved pediatric clinic provides primary care services to the local urban community and surrounding areas. Multiple providers within the clinic see an average 65-80 patients per day combined with an addition of 30-50 nurse visits per day during influenza season. Staff members of the clinic include; certified medical assistants (MAs), licensed practical nurses (LPNs), registered nurses (RNs), advanced practice registered nurses (APRNs), Doctors of Human Medicine (MDs), and Doctors of Osteopathic Medicine (DOs). Interventions The intervention was developed in collaboration between two DNP students and clinic leadership to improve the immunization error rate within a pediatric clinic. The intervention included a narrated educational series provided to all staff and providers involved in the vaccine administration process. See Appendix F for presentation details. The series was made available to the clinic’s leadership team for future use. After completion of the educational portion of the 13 intervention, staff were provided with a low-fidelity simulation experience. The details and overview of this simulation can be reviewed in Appendix G. One week prior to simulation, all staff members received an informational email explaining the simulation and expectations of the experience. Instructions were also provided verbally just prior to simulation.The goal of the simulation experience was to increase vaccine administration confidence amongst staff by providing a framework for safe vaccine administration in the form of a checklist as well as providing a hands-on learning experience. Educational sessions and simulation have been shown to have a significant impact on administration errors. By implementing an educational and informative presentation followed by a hands-on low-fidelity simulation activity, vaccine errors within the pediatric clinics were expected to be reduced. Following training, medication administration checklists, vaccine schedules, and “quick-tip” vaccine education posters were displayed throughout the clinic for reinforcement. In addition, a labeling and organizational system were put in place within vaccine storage areas to ensure quick and reliable access to correct vaccines. Regular emails with the pediatric clinic manager and director of risk, safety, and credentialing were conducted to ensure distribution of narrated educational powerpoint to all applicable staff members. The DNP students made near monthly visits to the pediatric clinic to assess efficiency and possible barriers of intervention implementation. Measures The impact of the chosen interventions were measured in multiple ways. A pre- and post-confidence survey (Appendix H), utilizing a Likert scale, was administered to healthcare professionals who administer vaccines at the pediatric clinic. These staff members included: MAs, LPNs, and RNs. A survey format was chosen as a measurement tool based on evidence-based success found within multiple articles of the literature review. The framework and questions asked in the survey were developed by the DNP students to meet the needs of 14 this specific vaccine related project. The goal of this survey was to measure staff confidence in the clinic's vaccine administration process before and after interventions. To specifically measure staff knowledge of vaccines that are provided within the clinic, staff completed a knowledge questionnaire (Appendix I) before and after completion of the narrated educational powerpoint presentations. Pol-Casteñeda et al. (2022) and Khalil and Lee (2018) both utilized questionnaires to successfully evaluate intervention effectiveness. Therefore, a questionnaire format was selected to evaluate the educational portion of the intervention. This quantitative data was monitored with bar and pie charts to observe scores of pre- and post-confidence surveys as well as pre- and post- knowledge questionnaires after implementation of narrated educational powerpoint presentations and low-fidelity simulation events. Initially, two sample t-tests for comparison of pre/post knowledge questionnaires were supposed to be used to analyze if the mean difference is due to our intervention or not. Two sample t-tests for comparison were not able to be carried out due to unpaired pre- and post-knowledge questionnaires and low post-knowledge response rates. Comparison of pre/post confidence surveys will also be utilized for evaluation of progression of staff confidence. Staff members that participated in the low-fidelity simulation experience were evaluated with a checklist (Appendix J) during the simulation. Staff members received credit for each step of the checklist that they followed. The checklist was completed by a trained simulation observer. A group debriefing was also conducted after all staff members completed the low-fidelity simulation. This qualitative data was used to monitor for variation between experiences as well as assess efficiency and possible barriers of intervention implementation processes. Debrief discussions with staff members and leadership will be utilized to implement future improvement activities when needed and inform the intervention. Based on the investigated literature surrounding the measures of intervention success to reduce vaccine errors by increasing staff confidence and knowledge, pre and post surveys and 15 questionnaires are proven to be evidence based options with high reliability and validity. Goals of pre and post surveys and knowledge questionnaires were to see a statistically significant increase in confidence and knowledge surrounding vaccine administration that is due to the implemented intervention. Future identified data gathered from the director of risk, safety, and credentialing were used to measure vaccine error rates by type of error following implementation of intervention. Methods employed for assessing completeness and accuracy of this data were discussed with a statistician. Ongoing assessment of interventions took place during debrief sessions with staff members following simulation. Failures, barriers, successes, improvements, and sustainability were addressed and applied to future implementation of interventions. Ethical Considerations This project’s focus is QI. It was reviewed by IRB and found it does not concern human subjects research as defined by university policy. Staff members who participated in the described interventions were aware of the purpose of this project being to decrease immunization error rates within the clinic they are employed by. The QI was a routine part of clinic/staff education expectations. Qualitative and quantitative data that was collected from staff members of the pediatric clinic throughout the duration of this project is in the form of aggregate data that is de-identified. There was no utilization of any patient information from electronic or non-electronic sources within the clinic during the duration of this project. Potential for harm was minimized during simulation experience as there was no actual administration of vaccines or medications of any kind. Funding Financial aspects were considered prior to implementing the intervention associated with the QI project. In total the budget for this project was $14,626.60. See Appendix K for a financial breakdown of the project budget. In order to sustain this project, the cost of the intervention only 16 included: LPN and MA hourly rates, as well as materials required for printing, visual aid creation, and simulation props. If the project is continued, the hourly salaries of the intervention implementer and the trainer of trainers would need to be accounted for. The team involved in this QI project included two DNP students, the pediatric clinic manager, and the director of risk, safety, and credentialing. The outline and organization process of this QI project included communication, planning, and data extraction from the pediatric clinic manager and director of risk, safety, and credentialing. The intervention was in place from October 12, 2023 - March 1, 2024. During that time frame several meetings took place over zoom with the pediatric clinic manager and the director of risk, safety, and credentialing. Zoom meetings were utilized to discuss project/intervention updates and to obtain data. Data acquired during these meetings included the number of vaccines given and the number of vaccine errors within the pediatric clinic during specified timeframes. Approximately 4 hours were spent with the director of risk, safety, and credentialing and clinic manager in combination. Approximately 2 additional hours were spent with the pediatric clinic manager in person at the clinic site during intervention planning and intervention completion. The hourly salaries of the pediatric clinic manager and director of risk, safety, and credentialing were not accounted for in the budget, as actions to assist in project development and completion were within their job description. Completion of the narrated educational powerpoint presentation and low-fidelity simulation took the two LPNs and two MAs that participated in both aspects of the intervention approximately 1 hour and 45 minutes. The cost of compensation for staff that only participated in the narrated educational powerpoint alone was not calculated into the project budget, as it was a routine part of their annual competency. Pre/post knowledge questionnaires and pre/post confidence surveys were all completed anonymously. Therefore, eliminating our ability to find out the staff member’s role who completed those elements. Without this knowledge, it is 17 impossible to compensate staff members with the correct hourly wage associated with their job title for time spent completing those elements. Results Progression of the collaborative intervention between two DNP students and pediatric clinic leadership occurred. The intervention was modified as continual knowledge was investigated and unanticipated barriers were encountered during implementation. A consolidated evolutionary diagram of this quality improvement project is depicted in Appendix L. Initial intervention implementation steps consisted of distribution of a pre-knowledge questionnaire to gather baseline vaccine knowledge amongst staff. Pre/post-knowledge questionnaires and confidence surveys were initially designed to be printed on paper and completed by hand. They were adapted to an electronic version via Qualtrics for ease of gathering data, as well as to limit loss of data. In addition to the above, simulation participants also completed a pre-confidence survey (Appendix H) prior to presentation and simulation completion. However, due to miscommunication, the pre-confidence survey was distributed to more staff than were scheduled to participate in the simulation. Therefore, data evaluation between pre- and post- confidence surveys were skewed. Next, an in-person low-fidelity simulation (Appendix G) was conducted. Unintended problems that were confronted at the beginning of the in-person simulation event included: reports from participating staff that they were unable to hear or access the educational presentation that was to be completed prior to simulation; that they did not receive a pre-confidence survey; and that they did not receive the informational email explaining the simulation and its expectations. The informational email included a checklist to be followed by staff during the event, which was also the basis of our simulation based data. Verbal instructions just prior to the simulation were modified to explain the simulation and checklist. A physical copy of the checklist was also distributed for participating staff to review during the 18 explanation of the event. Some participating staff also brought the checklist into the simulation with them, which was not the original intent of the document. Completion of the simulation took place as described in Appendix G, with re-distribution of pre/post-confidence surveys being distributed after the event. While re-distribution of pre-confidence surveys following simulation may have altered the analysis of data, only one response was received following simulation. Therefore, the other eight responses were an authentic representation of pre-simulation confidence levels. The narrated powerpoint was uploaded to a new platform (MediaSpace) and emailed to the clinic manager to re-distribute to staff with instructions to contact DNP students if audio and accessibility issues persisted. There was also a limited number of post-knowledge questionnaires completed. While pre-knowledge questionnaires were completed promptly upon distribution and the narrated powerpoint presentation was distributed thereafter, post-knowledge questionnaires were intended to be completed after viewing the presentation. However, only two responses were received after a number of reminders to staff and providers by clinic leadership compared to fifteen pre-knowledge questionnaire responses. The post-knowledge questionnaire was closed to responses after approximately three months due to concerns with data interpretation. The main contextual element that interacted with the intervention effectiveness was communication. Lack of knowledge that staff were unable to hear the narrated powerpoint, did not receive pre-confidence surveys, and did not receive the informational email explaining the simulation and expectations could have consequences in relation to the accuracy of data. In terms of outcomes, a number of data points and measures were collected including quantitative and qualitative data. Pre-confidence survey data submissions included nine responses, which as described earlier, was greater than the number of individuals (4) who participated in the simulation event. Post-confidence survey response rate was four out of four participants. Pre-confidence survey data revealed that all participants either agreed or strongly 19 agreed to a number of questions, which are detailed in Appendix M. The survey revealed mixed confidence ranging from unsure to strongly agree in disease pathophysiology and potential side effects and adverse reactions. Post-confidence survey data revealed similar levels of confidence compared to the pre-confidence survey in many areas as evidenced in Appendix M. However, there was a decrease in confidence regarding the knowledge of vaccines; safe administration practices; disease pathophysiology; contraindications; and side effects/adverse events as compared to the pre-survey data. Next, 15 responses were received for the pre-knowledge questionnaire resulting in a mean score of 90.9%. A large number of participants answered the majority of questions correctly. However, 5 of the 14 total questions on the pre-knowledge questionnaire were investigated related to variance in answer choice among staff members (see Appendix N). The topics of those five questions were either related to vaccine schedule or dose. Unfortunately, only two post-knowledge questionnaires were received. Given this low response rate, we elected not to compare results between pre- and post-knowledge checks. Simulation was evaluated through the use of a vaccine administration checklist (see Appendix J). Out of four participants, scores ranged from 7/10 to 9/10 steps completed of the administration process. Only one participant appropriately verified that the vaccine matched the EMR order and was from the appropriate stock (federally funded vs private), however they did not verify the vaccine dose against the EMR order nor did they verify with the parent/patient the appropriate reason for vaccination. Additionally, one individual verified all vaccines were labeled after preparation that other participants missed. All other remaining criteria were met by all participants. 20 During simulation debrief, qualitative data was collected verbally through discussion with staff members who reported an overall positive experience. They felt the training would be a beneficial experience specifically for staff members new to immunization administration. Simulation debrief also revealed concerns with the current administration process. Concerns included: a lack of a safe structured medication “double check”; incorrect MCIR review and vaccine orders; and patient/parent’s changing their mind regarding the vaccines they previously agreed to just prior to administration when vaccine products have already been prepared. Lastly, vaccine error rates were evaluated at intervals throughout our implementation timeline and compared to pre-intervention data. Prior to the intervention, 43,413 vaccines were administered between January 1, 2018, and October 12, 2023. This resulted in 13 administration errors, one of which was a near miss and four were unknown error types, at the pediatric clinic. The first project-evaluation interval started on October 13, 2023 and concluded on December 13, 2023. During that time, the pediatric clinic administered 2,236 vaccines. Of those 2,236 vaccines, there were no errors reported. Lastly, vaccine analysis took place from December 14, 2023 to March 1, 2024. During this time period, the pediatric clinic administered 1,885 vaccine doses and again there were no administration errors. Summary Discussion Vaccine errors cause a multitude of problems including inadequate disease protection, injury, increased use of resources, and reduced confidence in the healthcare system. Errors, clinical feedback, and clinic processes occurring in an urban midwestern university pediatric clinic were evaluated and revealed the need for staff education, vaccine storage organization, and easily accessible vaccine reference material. This quality improvement project was developed with these needs in mind and aimed to reduce errors within this pediatric clinic 21 through the use of clinic-wide virtual education, low-fidelity simulation, and vaccine storage area improvements. Providing education virtually allowed this project to be implemented in an accessible and approachable manner, which was a clear strength of the project. In addition, providing a hands on component allowed for in-person interaction and a safe space for practice, another project strength. Multiple data points were collected in a variety of formats to allow for greater opportunity in project analysis which proves to be beneficial in concluding outcomes and future implications. Interpretation Data collection included pre- and post-surveys, pre- and post-knowledge questionnaires, administration checklists, verbal debriefs, and vaccine error data. Pre- and post-surveys revealed a transformation in confidence between pre- and post- responses. Pre-surveys primarily revealed overall confidence in each element of vaccine administration. Following simulation, confidence decreased in almost half of the administration categories. This was not an expected finding when compared with the literature of Khalil and Lee (2018) and Sanko and Mckay (2017). The change in confidence between the pre- and post-simulation timeframe experienced in this project may be due to a number of factors. For one, pre-surveys were initially sent out for completion at the beginning of project implementation before the educational presentation was distributed. The educational presentation contained a number of elements mostly including education on individual vaccines, but also proper administration technique, vaccine schedule, and essential medication rights. Simulation focused on proper necessary steps within the administration process. While the original intent of the confidence survey was to measure the change in confidence before and after project implementation, with the anticipation of increased confidence following a complete educational experience, the decrease in confidence must be 22 evaluated contextually. Since confidence data was distributed for collection prior to both the educational presentation and the simulation, it is difficult to interpret which portion of the project caused a decrease in staff confidence. Both the presentation and simulation have the potential to be an enlightening experience, which may cause staff to question their true understanding and confidence regarding vaccines and their process of administration. This emphasizes the difference in outcomes between this project and available studies (Khalil & Lee, 2018; Sanko & Mckay, 2017) where there was no notable decrease in confidence. Although, this data is difficult to draw conclusions from due to incorrect pre-survey administration, as mentioned previously; and a small sample size. Next, the pre-knowledge questionnaire revealed a high level of pre-intervention knowledge beyond what was anticipated based on our initial root cause analysis. Our root cause analysis revealed staff and leadership concerns regarding baseline vaccine knowledge and education. In addition, the majority of errors throughout the healthcare system were caused by incorrect vaccine administration. Due to this analysis, baseline knowledge was anticipated to be lower than was actually evident. Comparisons between pre- and post-knowledge questionnaires were unable to be analyzed due to low post-knowledge questionnaire response rates impacting our ability to draw conclusions and compare these results to the available literature. Despite the fact that the scores of these knowledge questionnaires were unable to be compared statistically, insite was still gathered from the 15 completed pre-knowledge questionnaires through the analysis of individual question responses. A portion of questions answered did not yield consistent results among responders, which provided benefit when considering future improvement plans for the project. Reviewing the content of these questions revealed areas of the vaccine administration process, including vaccine schedules and doses, that may need to be emphasized with future education. Simulation evaluation was conducted through checklists and verbal debriefs. Checklists displayed mostly consistent results among participants, which indicated a high level of 23 compliance with essential pieces of the administration process. However, there are areas that were blatantly deficient as detailed in the results section and are important to recognize for future implications. These deficits may have been due to communication errors that were evident leading up to the simulation event, as previously discussed. While simulation replicated an activity that participants perform on a daily basis, given pre-intervention concerns with vaccine education and training, it was questionable how staff would perform during this exercise. Overall, however, participants performed well during this event. A number of other studies in the available literature also utilized checklists or some form of a validation tool during simulation (Durham et al., 2020; Pol-Casteñeda et al., 2022). Both Pol-Casteñeda et al. (2022) and Durham et al. (2020) had an increase in staff adherence to checklist expectations either during simulation or over time. While this project was not designed to evaluate adherence to vaccine administration steps pre- and post- intervention, it is encouraging that there is a high level of adherence to these steps, which is consistent with available studies (Durham et al., 2020; Pol-Casteñeda et al., 2022). Verbal debriefing following simulation yielded expected results through a unanimous consensus. Staff responses were consistent with the available literature. Anderson et al. (2020), Pol-Casteñeda et al. (2022), and Sanko and Mckay (2017) all received participant feedback indicating that simulation was valuable and participants were overall satisfied with the event, which aligns with this project’s outcome. Finally, no vaccine errors occurred. While we anticipated errors would decrease following project implementation due to reported deficiencies in staff and provider knowledge and education prior to project implementation, it is difficult to determine the true impact of the intervention. Error rates were initially extremely low for the preceding 5 years in the healthcare system overall. Given our relatively short term follow-up of 5 months it is not possible to confidently draw the conclusion that this intervention caused a reduction in error rates. Nor are we able to evaluate any subsequent errors to determine future project aims. 24 A majority of the available literature that reported error rates or overall intervention effects (Anderson et al., 2020; Jaam et al., 2021; Keers at al., 2014; Khalil et al., 2020; Koeck et al., 2017; Lee, 2019; Manias et al., 2020; Marufu et al., 2022; Plutinská & Plevová, 2019) reported that either education or simulation or both were effective at reducing vaccine errors or improving outcomes. It is expected that this quality improvement project will follow the results of these studies given the information found during the root cause analysis and the established data of the chosen interventions. However, long term data evaluation is needed to determine true intervention efficacy. Limitations This pilot quality improvement project conducted at a single pediatric clinic revealed a number of limitations throughout implementation as well as data collection and evaluation making it difficult to generalize. More information and project evolution is necessary for generalizability. To start, implementation was completed via the computer with presentation distribution occurring through e-mail by clinic leadership. This allowed each individual staff member and provider to complete the presentation on their own time. Given this was a self-directed activity, participants may not have been fully engaged with the presentation or simply stated they completed the presentation without doing so. The simulation’s impact may have been limited due to reported delay in distribution of the vaccine administration checklist and simulation instructions. With this, staff had limited time to review expectations and prepare for the simulation, which impacted the fluidity of the event. In regards to the event itself, this simulation was a low-fidelity simulation and lacked a number of components found in real practice. This simulation did not provide physical syringes for practice labeling; a system for differentiation between federally funded and private vaccine stock; and an electronic medical record (EMR) system for documentation. Without these elements, participants are missing some key components of the administration process of which are valuable for real practice. While we had participants verbalize any steps they were unable to 25 complete, the lack of physical action has the potential to affect the simulation's impact on outcomes. The use of e-mail for distributing the presentation and simulation information may have caused limitations. For example, some staff members and providers may not routinely monitor their emails for surveys or instructions. This may have limited the number of individuals that participated in vaccine administration training and responded to necessary surveys and questionnaires. Simulation participants may have missed information that was distributed through email causing difficulties during the pre-simulation brief. Regarding data evaluation, there were multiple limitations. Pre/post knowledge questionnaires (see Appendix I) created to measure staff vaccine knowledge base before and after the narrated powerpoint were designed without capability to pair a particular staff member’s pre-test with that same staff members post-test. This would have interfered with the ability to run a paired t test, which would have allowed a more specific and appropriate evaluation between specific individuals. Theoretically to adjust for this limitation we would have an unpaired two sample t-test to assess the significance of the narrated powerpoint as part of the intervention evaluation. Although due to low post-knowledge response rates, a t-test was not computed. The small number of staff able to participate in the low-fidelity simulation may have affected the internal and external validity of the results collected via the checklist (see Appendix J) and pre/post confidence surveys (see Appendix H). In addition, error in survey distribution and response rates of the pre/post confidence surveys will limit the validity and evaluation of these surveys. Debrief discussions with participating staff were helpful to evaluate staff experience, however, due to the limited number of participants these results may not be generalizable. Vaccine error data was gathered using the pediatric clinic’s error reporting database. Therefore our error data was reliant on a number of factors. For one, an error would need to be recorded in order to be reportable data. If an error occurred (or a near miss occurred), but the 26 situation was never entered into the error database there is no way to track this error. The error, if reported, would need to be appropriately assigned to the correct reporting category in order for it to appear on the vaccine error report. If it was assigned to an inaccurate category, it may have been missed when pulling historical error data. Given these possibilities, true error data is difficult to obtain given mistakes may occur, but may not be appropriately reported to the database. In this case, error rates may actually be higher than what current data suggests, therefore affecting the outcomes and conclusions of the project. Finally, length of follow-up was also a limitation of error rate evaluation. This project timeline allowed for less than 5 months of data collection and monitoring. With the extremely low error rate across the university health system over the prior 5 years, less than 5 months of data presents a problem when drawing conclusions. An extended monitoring time following intervention is necessary for a true depiction of project outcomes. Despite the limitations that presented themselves throughout the course of the pilot QI project, steps were taken prior to implementation to minimize and adjust for anticipated obstacles. To begin, the narrated educational powerpoint was distributed by email for staff to complete on their own time. This was initially done for staff convenience in hopes for higher completion adherence. As previously stated, it presented its own set of barriers. In regard to simulation, limitations were evident in the amount of hands on elements that could be incorporated in this low-fidelity event. To minimize this limitation, participants were encouraged to verbalize any steps that were not actionable to ensure recognition of the essential step in hopes of preserving the impact of the event. Adjustments were also made in real time to manage limitations during the simulation. These adjustments included providing a more in-depth simulation pre-brief when participants reported instructions were not received prior to simulation. This intervention maximized the ability of project leaders to provide a thorough introduction in hopes of maintaining the utility of the event. Future Plans or Implications 27 Looking ahead, future plans include expanding the intervention to outlying clinics within the health system. To do so, additional training and educational sessions will need to be developed. A “train the trainer” program surrounding the implementation and conduction of a low-fidelity simulation and educational experience is necessary for a successful program. Utilization of the Centers for Disease Control and Prevention’s Training of Trainers (ToT) model was used to create a manual for clinic staff and future DNP students (see Appendix O) (CDC, 2019). If continued implementation of this intervention is desired, this manual can be used to sustain the efforts and findings of the current project. Key components of the CDC’s ToT model were adapted into a manual unique to the pediatric clinic served during the course of this project (CDC, 2019). Elements of an all encompassing adapted ToT program include: instructions to complete pre-work (education that provides participants with background information on the desired intervention), a manual (step by step written instructions on completion of the intervention), instructions to complete intervention practice with feedback (in person session to practice the intervention with an opportunity for participants to ask questions), and instructions to complete planned follow-up and support sessions (contact information provided) (CDC, 2019). As time was a limitation present within this QI project, implementation of a complete ToT program was not carried out. With a descriptive step by step manual on how to recreate and sustain the current intervention in place (see Appendix O), the hope is that future DNP students or healthcare system staff will be able to replicate the intervention and train other individuals with ease. Creation of the manual for this specific intervention and population should foster an environment of sustainability for this project. Additionally, modifications to the educational powerpoint presentation should be considered. While there is convenience to distributing a narrated powerpoint that can be completed independently by participants, ensuring completion of the educational presentation is difficult. Consideration of a live education session may be warranted to confirm that all 28 participants receive the proper educational experience. In addition, vaccines change over time and updates may be required periodically to provide the most relevant and up-to-date information. In hopes to produce statistically significant results, a few key aspects should be emphasized in project continuation. Along with correcting the previously discussed limitations of this study, time and sample size are at the forefront of producing generalizable results. Expansion of this project to be completed by outlying clinics will allow for a larger sample size to be observed. Extended length of time will allow for several cycles of the project to be carried out. Therefore, increasing reliability, validity, precision, accuracy, and generalizability of future results. Lastly, If implementation of this QI project is continued in the future, the financial aspects of supplies, expansion, and participating staff compensation must be considered. Supply costs that will need to be compensated for include: ink, paper, lamination, and additional materials used for the creation of visual aids displayed in the participating clinics. Additional supply costs include: ink, paper, and tape used in the creation of props during the low-fidelity simulation experience. Expansion costs encompass compensation for hired or designated staff members that will train other trainers, as well as for staff that will implement interventions at participating clinics. Hourly wages will need to be determined for the specific roles of trainer of trainers and intervention implementers. Some costs may be avoided if a staff member is already designated to a quality improvement position and trainer of trainers/intervention implementer falls within their job title. Compensation for staff participating in the intervention is another aspect to consider in regards to project finances. If the intervention intends to remain as part of an annual competency for participating staff, staff compensation may not need to be considered. If the intervention becomes an additional requirement, the hourly wages of the varying medical professionals participating in the intervention will need to be calculated. Conclusions 29 Vaccinations are a pivotal component of routine preventative care provided to most children during development. Improper knowledge base and administration of vaccines pose a significant safety risk to the pediatric population. This quality improvement project encompasses leading evidence-based practices into a streamlined educational powerpoint and a low-fidelity vaccine administration simulation, to decrease vaccination error rates and improve patient safety. Plans of a project outline and turnover were created for up-coming DNP students to monitor and adapt the project as seen fit to meet the needs of the pediatric clinics served. Development of a “train the trainer” program allows for this education/simulation intervention to be sustained for long term use and expanded to multiple clinics of similar settings. Ongoing analysis and refinement of the project intervention are required to sustain safe vaccination administration practices while desired behaviors are integrated. Previously discussed short-falls and barriers to this pilot study will need to be addressed and altered as the project progresses. The PDSA cycles that were utilized in the creation of this project will allow for a fluent process of frequent re-evaluation and improvement throughout continued intervention implementation. 30 References Agency for Healthcare Research and Quality. (2019). Systems Approach. https://psnet.ahrq.gov/primer/systems-approach Andersen, P., Downer, T., Spencer, A., & Willcocks, K. (2020). Using observational simulation teaching methods in professional development to address patient safety. Collegian, 27(2), 207–212. https://doi.org/10.1016/j.colegn.2019.07.005 Centers for Disease Control and Prevention. (2021). Vaccine administration: Preventing vaccine administration errors. Retrieved June 5, 2023, at https://www.cdc.gov/vaccines/hcp /admin/downloads/vaccine-administration-preventing-errors.pdf Centers for Disease Control and Prevention. (2023). Fast Facts on Global Immunization. Retrieved June 4, 2023, at https://www.cdc.gov/globalhealth/immunization/data /fast-facts.html#:~:text=Immunization%20Prevents%20Death%20Worldwide,save%20ne arly%2019%20million%20lives Centers for Disease Control and Prevention. (2024). You Call The Shots - Web-based Training Courses. Retrieved June 10, 2023, at https://www.cdc.gov/vaccines /ed/youcalltheshots.html Centers for Disease Control and Prevention. (2018). Vaccine Administration. Retrieved June 11, 2023, at https://www.cdc.gov/vaccines/hcp/admin/admin-protocols.html Centers for Disease Control and Prevention. (2022, September 12). Reasons for adults to be vaccinated. Retrieved June 11, 2023, at https://www.cdc.gov/vaccines/ adults/reasons-to-vaccinate.html Centers for Disease Control and Prevention. (2019, May 29). Training and professional development. Retrieved January 25, 2024, at https://www.cdc.gov/healthyschools/ trainingtools.htm Durham, M., Didovic, I., & Gingell, M. (2020). Pediatric vaccine administration: Sustaining an improved process in a primary care setting. Patient Safety, 2(2), 36–47. 31 https://doi.org/10.33940/med/2020.6.5 Institute for Healthcare Improvement. (2017). QI Essentials Toolkit: PDSA Worksheet. Retrieved April 7, 2024, at https://www.ihi.org/resources/tools/quality-improvement-essentials-toolkit#downloads ISMP National Vaccine Errors Reporting Program 2017 analysis (part I): Vaccine errors continue with little change. Institute For Safe Medication Practices. (2018, June 14). Retrieved June 17, 2023, at https://www.ismp.org/resources/ismp-national- vaccine-errors-reporting-program-2017-an%20alysis-part-i-vaccine-errors ISMP National Vaccine Errors Reporting Program: 2020-2021 analysis focuses on age-related, non-covid-19 vaccine errors. Institute For Safe Medication Practices. (2022, September 22). Retrieved June 17, 2023, at https://www.ismp.org/resources/ismp-national-vaccine-errors-reporting-program-2020-20 21-analysis-focuses-age-related-non#:~:text=Our%20analysis%20of%20the%20remaini ng,Wrong%20age%20(13%25) Jaam, M., Naseralallah, L. M., Hussain, T. A., & Pawluk, S. A. (2021). Pharmacist-led educational interventions provided to healthcare providers to reduce medication errors: A systematic review and meta-analysis. PLos ONE, 16(6). https://doi.org/10.1371/journal.pone.0253588 Keers, R. N., Williams, S. D., Cooke, J., Walsh, T., & Ashcroft, D. M. (2014). Impact of interventions designed to reduce medication administration errors in hospitals: A systematic. Drug Safety, 37(5), 317–332. https://10.1007/s40264-014-0152-0 Khalil, H., & Lee, S. (2018). The implementation of a successful medication safety program in a primary care. Journal of Evaluation in Clinical Practice, 24(2), 403–407. https://doi.org/10.1111/jep.12870 Khalil, H., Kynoch, K., & Hines, S. (2020). Interventions to ensure medication safety in acute 32 care: An umbrella review. International Journal of Evidence-Based Healthcare, 18(2), 188–211. https://10.1097/XEB.0000000000000232 Koeck, J. A., Young, N. J., Kontny, U., Orlikowsky, T., Bassler, D., & Eisert, A. (2021). Interventions to reduce medication dispensing, administration, and monitoring errors in pediatric professional healthcare settings: A systematic review. Frontiers in Pediatrics, 9. https://doi.org/10.3389/fped.2021.633064 Kroger, L., Bahta, L., Long, S., & Sanchez, P. (2023). General Best Practice Guidelines for Immunization. Centers for Disease Control and Prevention. Retrieved June 10, 2023, from https://www.cdc.gov/vaccines/hcp/acip-recs/general-recs/downloads/ general-recs.pdf Lapkin, S., Levett-Jones, T., Chenoweth, L., & Johnson, M. (2016). The effectiveness of interventions designed to reduce medication administration errors: A synthesis of findings from systematic reviews. Journal of Nursing Management, 24(7), 845–858. https://doi.org/10.1111/jonm.12390 Lee, H. (2019). A meta-analysis of the effects of intervention on the prevention of medication administration errors in nurses. Medico-Legal Update, 19(1), 659. https://10.5958/0974-1283.2019.00117.8 Lin, J. L., Bacci, J. L., Reynolds, M. J., Li, Y., Firebaugh, R. G., & Odegard, P. S. (2018). Comparison of two training methods in community pharmacy: Project VACCINATE. Journal of The American Pharmacist Association, 58, S94-S100. https://doi.org/10.1016/j.japh.2018.04.003 Manias, E., Kusljic, S., & Wu, A. (2020). Interventions to reduce medication errors in adult medical and surgical settings: A systematic review. Therapeutic Advances in Drug Safety, 11, 204209862096830. https://doi.org/10.1177/2042098620968309 Marufu, M. C., Bower, R., Hendron, E., & Manning, J. C. (2022). Nursing interventions to reduce medication errors in paediatrics and neonates: Systematic review and meta-analysis. 33 Journal of Pediatric Nursing, 62, e139-e147. https://doi.org/10.1016/j.pedn.2021.08.024 McKeirnan, K. C., Frazier, K. R., Nguyen, M., & Maclean, L. G. (2018). Training pharmacy technicians to administer immunizations. Journal of the American Pharmacist Association, 58, 174-178. https://10.1016/j.japh.2018.01.003 Morse-Brady, J., & Marie Hart, A. (2020). Prevalence and types of vaccination errors from 2009 to 2018: A systematic review of the medical literature. Vaccine, 38(7), 1623–1629. https://doi.org/10.1016/j.vaccine.2019.11.078 National Coordinating Council for Medication Error Reporting and Prevention. (2024). About Medication Errors. https://www.nccmerp.org/about-medication-errors Plutinská, Z. & Plevová, I. (2019). Measures to prevent medication errors in intensive care units. Central European Journal of Nursing and Midwifery, 10(2), 1059-1067. https://10.15452/CEJNM.2019.10.0014 Poiraud, C., Réthoré, L., Bourdon, O., Lorrot, M., & Prot-Labarthe, S. (2023). Understanding and preventing vaccination errors. Infectious Diseases Now, 53(2), 104641. https://doi.org/10.1016/j.idnow.2023.01.001 Pol-Casteñeda, S., Carrero-Planells, A., & Moreno-Mulet, C. (2022). Use of simulation to improve nursing students’ medication administration competence: a mixed-method study. BMC Nursing, 21-117. https://doi.org/10.1186/s12912-022-00897-z Reed, L., Tarini, B. A., & Andreae, M. C. (2019). Vaccine administration error rates at a large academic medical center and its affiliated clinics – familiarity matters. Vaccine, 37(36), 5390–5396. https://doi.org/10.1016/j.vaccine.2019.07.027 Sanko, J. S. & Mckay, M. (2017). Impact of simulation-enhanced pharmacology education in prelicensure nursing education. Nurse Educator, 42(55), S32-S37. https://doi.org/10.1097/NNE.0000000000000409 Appendix A Root Cause Analysis: Fishbone Diagram 34 35 Appendix B PRISMA Table Citation Level of evidence/ design/ purpose (Pol-Caste ñeda et al., 2022) Level III Quasi Experimental Purpose: Evaluate nursing students’ skill in the safe administration of medication, using simulation. Secondarily, to obtain student opinions of the activity. 36 Sample Intervention Appendix C Literature Table Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations - 179 nursing students in the second year of their program, between 2018 and 2019, that were currently taking pharmacology (convenience sampling) No exclusion criteria - Simulation based activity; designed using the INACSL Standards of Best Practice: Simulation Design. - Pre-questionnaires, a teacher evaluation during the simulation activity, and a final open-ended opinion survey were utilized - Three different scenarios were created to reflect a patient within the hospital setting receiving intravenous medication(s). - 2 weeks before the simulation took place, students were provided the case files with instructions as a pre-briefing activity. - 24 groups of 6-8 students; each student played the part of nurse/patient/caregi ver/observer/etc. - Descriptive analysis was performed on the study population, results of the pre-questionnaire and simulation based activity data. - SPSS v22.0 software was utilized for analysis - Content analysis was made using the answer to the open-ended final survey. Answers were codified by three independent researchers. Manual analysis was performed. - Variables were measured through direct observation of a single instructor during the - Pre-questionnaire was completed by 73 (41%) students - The simulation evaluation was performed by all 179 (12% male; 88% female) students - Open-ended final survey was completed by 42 (23.5%) students Pre-questionnaire results: 1. The right patient (64.4%) 2. The right medication (60.3%) 3. The right dose (60.3%) 4. The right route (54.8%) 5. The right time (24.7%) 6. The right documentation (54.8%) Results during simulation activity: 1. The right patient (83.3%) Strengths: - Simulation evaluation was completed using 1 teacher and a structured evaluation form/checklist was useful for evaluation consistency - Detailed description of intervention and evaluations Limitations: - Low number of pre-questionnaires were completed resulting in potential bias and invaluable results - Study included majority female gender - Some students are repeating this course potentially altering findings 2. The right medication - Study completed in Spain (95.8%) 3. The right dose (100%) 4. The right route (95.8%) Implications: Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments simulation. - During the 15 minute intervention, the “nurse” simulated medication administration verifying each right in the medication process. - Debriefing occurred following each simulation. Interventions included in reviewed studies: - Automated infusion devices - Computerized physician order entry - Changes in work schedules - Intravenous systems - Modules of education - Study analysis included study exclusion utilizing the PRISMA recommendations. Primary and secondary studies evaluating the effect of reducing medication errors were chosen. - Study details were summarized in tables to include the intervention, methods, and study conclusions. - Qualitative synthesis of the available literature developed (Plutinská & Plevová, 2019) Level V Descriptive Review Purpose: Summarize studies on intervention effectiveness to reduce adverse events of the medication error type and to identify recommendatio ns for preventing medication Final analysis studies included: - 3 systematic reviews - 1 PDSA design - 1 direct observational study - 2 retrospective studies - 3 prospective studies - 1 quantitative 37 Findings Strengths/Limitations/Impli cations 5. The right time (70.8%) 6. The right documentation (45.8%) Open-ended final survey: - Simulation appears useful and students were satisfied with the experience. - Smaller groups should be used for simulation - All interventions have the potential to reduce certain medication related errors - No single approach was recommended over another related to study limitations. - Resources and type of errors should also be taken into account with implementation amongst the varying interventions investigated. - Simulation may be useful to include in addition to standard education, however study results should be taken with caution given a number of potential biases. Strengths: - Number of study inclusion with systematic reviews - Thorough description of study interventions and outcomes Limitations: - No thorough description of study evaluation methodology was available - ICU only study - Review completed outside of the US Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations errors in ICUs survey 38 Inclusion criteria: - Published 2008-2018 - Focus on prevention strategies and measures to reduce risks associated with medication administration in ICU - Full text availability Exclusion criteria: -Theoretical reviews Sources: SCOPUS and EBSCO searched - 18 studies Inclusion criteria: - Peer reviewed published studies (Marufu et al., 2022) Level I Systematic Review and Meta-analysis Purpose: To identify nursing Implications: - Education and a number of other interventions may be useful in administration error reduction. - Medication reconciliation - Pharmacist intervention/involve ment - Protocols and guidelines - Support systems for clinical decision-making - Electronic health records - Bar-coded medication administration - Medication error minimization scheme Interventions included: - Education programmes (most common; included in 13 studies) - Two separate authors independently identified studies for inclusion - Differences in study opinion were resolved by discussion resulting in - The Meta-analysis showed a 64% reduction in medication administration errors post intervention Strengths: - Level of evidence - - Pooled OR = 0.36 95% Confidence - Well detailed analysis of study choices 39 Strengths/Limitations/Impli cations - Majority of studies included an educational component Limitations: - High heterogeneity Implications: - Medication safety education is an important intervention in reducing administration errors. Citation Level of evidence/ design/ purpose interventions to reduce medication administration errors and to perform a meta-analysis Sample Intervention Measurement: Variables and Instruments Findings Interval (CI) = 0.21–0.63 P = 0.0003) - - All interventions showed a reduction in medication errors. - Medication information services - Clinical pharmacist involvement - Double checking - Reduce interruptions during drug calculation/preparati on - Implementation of smart pumps - Intervention aimed at reducing administration errors in in-patient settings - English translation or published article Exclusion criteria: - Case studies - Epidemiological studies - Reviews - Editorials - Opinion papers consensus or involving a third author - A pre-piloted standardized form was utilized for independent data extraction - Risk of bias was assessed with The Quality Assessment Tool for Before and After (Pre-Post) studies with No Control Group (BAQA) - Majority of studies presented results in error rates or percentages - Pre and post intervention total drug administration error numbers, odds ratios, and 95% confidence intervals were utilized to find the likelihood of medication error reduction post intervention. - Meta-analysis was performed in Rev. Man5 Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 40 using the random effect method for a pooled size effect of implementing any error reduction intervention. - Studies that could not be calculated with the meta-analysis, a qualitative synthesis was provided. (Andersen et al., 2020) Level V Quality Improvement (QI) Purpose: To improve patient care to meet the national safety and quality service standards - Healthcare workers attending mandatory professional development (n=429), including: - Nurses - Nurse Assistants - Midwives - Physiotherapist - Occupational Therapists - Anesthetic Technical staff - Observational simulation Methods of measurements included: Results surrounding medications: Strengths: - A series of videos were created and used as educational tools for required professional development in response to audit findings - Educational video series depicted a patient's hospital stay The video series focused on: - Quality standards - Quality audit data - Surveys (anonymous) - Interviews -Simulation experience scale (Cronbach’s ɑ= 0.78) - 5 point Likert scale - 34% reduction in medication incidents Surveys + Interviews: - Increased awareness of medication procedures was seen post intervention - Increased early identification and medication error reporting seen post intervention - Descriptive statistics (means, SD, t-test) Post intervention practice changes: - Standard statistical tests used (descriptive, chi square, Pearson’s - Increased number of staff members asking for assistance - Detailed explanation intervention - Multiple data measures utilized Limitations: - Lower level evidence - Results not statistically significant. - Results specific to the hospital where the study was conducted - Study conducted in Australia Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 41 for medication administration - Infection control - Patient identification - Documentation - Preventing falls Correlation, Cross tabulation, MultiVariant Analysis) - Increased inquiry about medications Implications: - Using simulation videos in mandatory professional development can positively impact safety and health outcomes. - Further research is needed related to the use of simulation and the impact observation has on learning and patient care Administering provider satisfaction: - 101 simulation satisfaction survey responses received - 83% of participants were satisfied with the learning experience. Of that 83%, 60% agreed and 23% strongly agreed that the simulation was a valuable educational experience. - Clinical indicators for falls, high-alert medications, and infection during the post-training period revealed significant decrease compared to pre-simulation (Khalil et al., 2020) Level I Umbrella review of Systematic Reviews Purpose: To synthesize evidence from - 23 systematic reviews Interventions evaluated: Databases searched: MEDLINE, CINAHL, Web of Science, EMBASE, and - Prescriber education - Medication administration education - Data was evaluated and analyzed by two independent reviewers Educational Interventions (utilized to prevent medication administration error): - A standardized data extraction tool was utilized - Findings were - 2 of the 23 reviews included data on medication administration educational interventions and their outcomes Strengths: - High level of evidence Limitations: - High heterogeneity - Several of the reviews used Citation Level of evidence/ design/ purpose all systematic reviews investigating the effectiveness of medication safety interventions for preventing medication errors, medication related harms and death in acute care patients. Sample Intervention Measurement: Variables and Instruments Findings presented in tables to illustrate individual intervention outcomes. - Strengths of the evidence for each intervention was indicated for each article utilizing a traffic stop light color coding system. - Educational interventions included: traditional classroom training, simulation; E-learning; Slide show presentations; Interactive CD-ROM program; Posters and pamphlets - All studies reported a significant positive effect of interventions on medication administration safety and skills. 42 Strengths/Limitations/Impli cations narrative summaries to report their findings, impacting the pooled statistical findings. This limited the ability to accurately synthesize the findings. - Multiple studies used “bundled/multifactorial “ interventions to describe strategies used without naming all the parts of those interventions. - Pooled analysis of the results favored the 2 interventions Implications: - No strong recommendation - Some safety interventions (medication reconciliation, barcoding systems, reprinted order sheets, specialist pharmacy roles) should be considered - The two interventions showed a large effect size (Hedges’ g = 1.06) however, the heterogeneity between the studies was very high (I2 93%). - E-learning was evaluated by several included studies and found to be effective within a range of effect sizes. Protocols/guidelines/checklists/ checking systems for preventing medications errors: - 4 studies included these interventions - Medication reconciliation or review - Electronic prescribing - CPOE/CDSS interventions to reduce or prevent interruptions during medication administration - Bundled interventions that included multiple of the above interventions The Cochrane Library Inclusion criteria: - Quantitative systematic reviews - English language - Provide a clear and comprehensive search strategy and critical appraisal using standardized tools - Evaluated interventions designed to prevent medication prevent medication errors in acute care settings - Patients were adults or children in an acute care setting Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 43 - Meta-analysis was not completed due to study design and heterogeneity - Mixed results in terms of decreased errors - Participants that were healthcare workers (registered nurses, enrolled or licensed vocational nurses, midwives, pharmacists, medical doctors) involved in prescribing, dispensing or administering medications. - Outcomes reported medication errors, medication-relate d harms and medication-relate d deaths Exclusion criteria: -Systematic reviews including studies of pharmacy assistants or nursing assistants Citation Level of evidence/ design/ purpose (Lee, 2019) Level II (includes RCT and NRCT) Meta-Analysis Purpose: To determine study environment, variables, intervention effects, and size of effects for the study of intervention methods to prevent medication errors through meta-analysis by presenting comprehensive , reliable, and consistent results. Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 44 - 13 studies (RCT (5) and non-randomised controlled trials (8)) - Five electronic databases (CINAHL, PubMed, EMBASE, Ovid, and Cochrane Library) were searched to retrieve Inclusion Criteria: - Nurses that administer medications in hospitals - Interventions placed to prevent medication errors - Outcomes include error rates and knowledge score Interventions that involved medical devices included: - Two reviewers independently selected data - Bar code assisted medication administration - Dispensing systems that automatically dispense medications - Computerized prescribing Interventions that involved education included: - Simulation based learning - Designated medication administration nurses - Pharmacist guided education - Final RCT studies were evaluated using the RoB (The Cochrane’s Risk of Bias); non-randomized controlled trials evaluated with RoBANS (Risk of Bias Assessment tool for Non-randomized Studies) - Characteristics of the selected studies were analyzed and coded. - Comprehensive Meta- Analysis (version 3.0) was utilized to calculate effect sizes and homogeneity tests. - Random effect model was used in consideration of heterogeneity between studies - Odd ratio and standardized mean difference were used to - Randomized controlled trial or - Education using utilizing electronic Conclusions drawn from medical device interventions: Medical devices were found to directly reduce medication administration errors made amongst nurses. Strengths: - Homogeneity was obtained through splitting up outcome variables between intervention types - High level of evidence - OR=0.64 - - 95% CI: 0.45 to 0.93 p=.020 Conclusions drawn from simulation educational interventions: Limitations: - No detailed descriptions of interventions analyzed - Heterogeneity among included studies Simulation educational activities were the only educational activity that was effective in improving nurse medication knowledge. - Non-randomized controlled trials were included limiting the studies ability to determine the effects of the intervention - - - SMD=1.06 95% CI: 0.07 to 2.05 p=.036 Implications - Simulation and medical devices are useful in the medication administration process Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 45 non-randomized controlled trial Exclusion criteria - Non-experimental studies such as survey and qualitative research - Studies that cannot calculate the effect of the intervention - Studies performed in outpatient setting - Abstracts and case studies without its original text - 16 systematic reviews - 10 electronic databases were searched; search consisted of a three-step approach (Lapkin et al., 2016) Level I Review of Systematic Reviews Purpose: Examine the effectiveness of interventions devices calculate the effect size - Instructor led educational experiences Interventions investigated within the systematic reviews included the following: - Interventions involving medication - Assessment of Multiple Systematic Reviews (AMSTAR) protocol was utilized by two independent reviewers to examine the included systematic reviews. - Study evaluation revealed variable quality scores. The median AMSTAR score was 8 with a range between 6 to 11. Education and training: - Mixed results - AMSTAR was utilized - Some evidence supports Strengths: - High level of evidence - Multiple interventions evaluated Limitations: Citation Level of evidence/ design/ purpose designed to improve patient safety by reducing medication administration errors using data from systematic reviews Sample Intervention Measurement: Variables and Instruments Findings for the quality of the systematic reviews and they were broken into three score levels: - - - 8 - 11 = high quality 5 - 7 = medium quality 0 - 4 = low quality simulation training Checklists, protocols, guidelines - Checklists can be effective in reducing errors Conclusions: - Sparse evidence indicating a single intervention can prevent medication administration errors. - Multi-faceted combined interventions are more successful administration interruption management - Educational training sessions - Double checking - Technological support systems for medication administration - Checklist - Protocols - Guidelines Inclusion Criteria: - Systematic review - Evaluated ways to reduce or prevent medication errors in acute, subacute, and residential aged care settings -Nurse participants - Outcomes included the incidence and number of medication errors and adverse drug events or widely used indicators Exclusion criteria not discussed 46 Strengths/Limitations/Impli cations - Heterogeneity inhibited meta-analyses. - Large number of studies excluded due to not meeting AMSTAR criteria. - Available research based on self-reported medication incident data - Many reviews did not disclose the severity of harm associated with identified medication errors. Implications: - Simulation training with multifaceted approaches should be used for training (Manias et al., 2020) Level I Systematic Review - 34 articles Databases searched: The 12 intervention types: - Pharmacist-led - Rayyan used for independent screening of articles Prescriber education results: - Prescriber error rates reduced in 14 out of 26 studies Strengths: - High level of evidence - Several education Citation Level of evidence/ design/ purpose Purpose: Compare effectiveness of interventions in reducing medication errors occurring with prescribing, giving, and supplying medications in adult medical and surgical settings in hospital Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 47 MEDLINE, CINAHL, EMBASE, PsycINFO, Cochrane, Cochrane Database of Systematic Reviews and the Cochrane Central Register of Controlled Trials Inclusion Criteria - Study aimed at reducing medication errors in adult acute care medical or surgical settings -English language Exclusion Criteria: - Near misses - Case studies, commentaries, editorials, reviews, epidemiological studies, conference abstracts medication reconciliation - Computerized medication reconciliation - Medication reconciliation by trained mentors - Computerized physician order entry (CPOE) with or without a clinical decision support system - Pharmacist partnership - Prescriber education - Patient education - Trained medication experts - Medication dispensing - Use of an automated drug - Two authors reviewed articles independently; Third author assessed discrepancies - Increase in prescribing errors from baseline for both control (p<0.001) and e-learning group (p=0.025) - Discussion was used to resolve disagreements - Pharmacist education group decreased prescribing errors (p<0.001) interventions analyzed that revealed significant results Limitations: - High level of heterogeneity - Variable calculations of error rates Trained medication experts: - Out of 4 studies, 1 study showed significant improvement - Variable data collection methods - The study evaluation with improvement evaluated dedicated trained pharmacy assistants in providing education and showed improvement in error rates (p<0.0001) Implications: - More research is needed with a greater focus on the clinical significance of the interventions. Interventions comprising interdisciplinary approaches also needed. - RCTs assessed with CONSORT guidelines, non-randomized controlled trials assessed with TREND guidelines, quality improvement guidelines assessed by SQUIRE guidelines - Data synthesis was qualitative by grouping results - Meta-analysis calculated use RevMan - Risk ratio calculated for categorical outcomes - Some errors had standard mean difference calculated Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 48 distribution system with or without electronic medication administration record - Interdisciplinary collaboration and electronic administration system - Medication related problems as the outcome - Effect of intervention measured outside of hospital - Specialty wards (intensive care, emergency care, perioperative care, neurologic and cancer care) - Outpatient, subacute settings (rehab, geriatric units) 49 - First cycle assessed with verbal interviews with CMAs to review comfort of new process and feedback - 1 month following implementation, behavior compliance was 29% and 57% for 2 CMAs; additional training was then implemented Strengths: - Continued PDSA assessment - Vaccine administration competency validation - Compliance was variable (86%) from May through August 2019 - Peer audits - Audit and monitoring utilizing a bar chart for compliance evaluation - Inconsistent use of the flag system and vaccine information sheets (VIS) Limitations: - Small number of participants - Single-site study - Small number of vaccine administration Implications: - May be transferable to primary care. (Durham et al., 2020) Level V Quality Improvement (QI) project - Federally qualified health center (FQHC) - Vaccine administration checklist created - Certified medical assistants (CMAs), nurse manager, advanced practice registered nurses (APRNs) Purpose: Simplify and standardize the vaccine administration process, improve staff knowledge, safe administration behaviors to prevent errors - Process mapping revised to align with CDC guidelines; training and education provided - Revised process map displayed in vaccine preparation area and patient rooms - Vaccine labels placed on refrigerator for easy identification - Regular vaccine/diluent rotation schedule implemented - Comprehensive training including simulation, vaccine education, vaccine administration, refrigerator orientation, teach-back, paper handouts was provided to administering staff - Weekly audits of Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 50 process behaviors with immediate feedback provided over initial 2 months - A second cycle of education was implemented due to a decrease in safe behaviors. - Medication safety training that focused on issues identified in incident reports: 1 day of lecture, case studies, small group discussion - Creation of medication safety committee including a multi-disciplinary team - Implementation of new medication guidelines, which were introduced on the day of training (Khalil & Lee, 2018) Level V Quality Improvement (QI) - Conducted at a non-profit healthcare organization - Study participants included clinicians - No other description or number of participants were included in the study Purpose: To describe the steps involved for the implementation of a medication safety program in primary care in rural Australia; To report on its evaluation and provide recommendatio ns for future initiatives. - Anonymous questionnaire completed before and after training targeting medication safety knowledge, confidence in practice, behavior in implementation, and training satisfaction utilizing a likert scale - Questionnaires completed before training, immediately after training, and 6 months after training - 29 completed surveys before intervention, 18 post training, and 9 at 6 months post training Strengths: - Multiple steps of intervention were detailed - t-test revealed statistically significant change in medication knowledge and confidence - Description of study measure and analysis provided - Increase in clinician confidence in applying training into daily practice (p=0.02) Limitations: - Low level of evidence - Clinician knowledge improved after 6 months of training - Study conducted in Australia - Poor detail of study sample Implications: - Implementation was successful at this large organization - No further commentary was provided regarding its Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings - 2 cohorts of accelerated nursing students (120 students) enrolled in pharmacology (Sanko & Mckay, 2017) Level III Quasi-experim ental design Purpose: Not explicitly stated. Aim appears to be to reduce gaps in training and build medication safety practices through the use of simulation - 1 cohort (60 students) was utilized as the control (standard teaching) - 1 cohort (60 students) was utilized as the intervention group (simulation) - 4 manikin-based scenarios divided into two 2-hour sessions - Sessions focused on administration skills that aligned with QSEN safety competency knowledge, skills, attitudes including calculations, high-alert medication procedures, hand hygiene, PPE, medication information Data collection included: - Self reported medication administration confidence and competence - Observation of medication administration - Self-reported adverse events - Post Intervention participant evaluations Measures - Self-reported medication administration competence and confidence scale - Medication administration observation tool - Post-intervention evaluation - Observation of medication administration revealed a statistically significant increase in infusing medications over the correct time (p=0.021) and performing hand hygiene (p=0.017). - Self-reported confidence and competence analysis was statistically significant (P<.001) over time - Improvements in competence was notable in both groups (P<.001) over time - Intervention group only had improved confidence (P<.001) over time - Control group showed decrease in confidence over time (P<.001) - Mean comparisons at time 2 was significant (P=0.034) - No difference in confidence between groups at time 1 (control, M = 54.68; 51 Strengths/Limitations/Impli cations implications for other sites or future research Strengths: - Well described study design - Thorough description of study analysis Limitations: - Small sample size - Must of the medication administration items were not statistically significant outcomes - Tracking long term effect of simulation was not feasible Implications: - Simulation can be useful in increasing medication administration confidence, competence, and actions including the reduction of adverse events. Citation Level of evidence/ design/ purpose Sample Intervention searching, checking appropriate lab values, and vital signs prior to administration. - Pairs of students completed simulation together Measurement: Variables and Instruments - S-AERS Analysis - Scale analysis - Descriptive statistics - Student t tests - Spearman’s p - Stop-action simulation utilized - X^2 - Debriefing and didactic teaching was utilized - Fischer’s 52 Findings Strengths/Limitations/Impli cations intervention, M = 55.65; P = .718)) or time 2 (control, M = 52.58; intervention, M = 67.18; P = .096) - Control group had a greater amount of adverse events, incorrect medication administrations, incorrect route, appropriate patient identification, problems with equipment, problems with administration records, events caused by knowledge deficits, feelings of personal work overload - Participants found simulation helpful (Jaam et al., 2021) Level I Systematic Review and Meta-analysis Purpose: Describe and compare various pharmacist-led educational interventions delivered to - 12 studies (115,058 participants) - Study locations: Egypt, Australia, USA, Pakistan, Spain, Netherlands, Saudi Arabia, Vietnam Inclusion Criteria: - Didactic lectures were included in all interventions to some extent - In addition to didactic lectures, some studies included posters, practical teaching sessions, audit and feedback method with weekly - Two researchers independently searched using the same strategy, reviewed titles/abstracts, and discrepancies were resolved through full-text screening. - A data extraction sheet was utilized and completed by two independent researchers and consensus was met - All studies were eligible for meta-analysis Strengths: - Level of evidence - 10 out of 12 studies revealed a significant decrease in medication errors - Study consistent with other systematic reviews and meta-analyses - Pooled OR across all studies was 0.38 (95% CI 0.22 to 0.65) - Frequent sessions more effective than one-time sessions - Detailed description of sample collection and methodology Limitations: - High heterogeneity Citation Level of evidence/ design/ purpose healthcare providers and to evaluate their impact qualitatively and quantitatively on medication error rates Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 53 - Interventions were more effective when handouts, posters, or flashcards utilized - Risk of publication bias - Mainly inpatient setting - Included studies had a variable definition of “medication error” Implications: - Pharmacist-led interventions to healthcare providers are effective interventions to reduce medication errors. newsletter, flashcards of high-risk abbreviations - Interventions lasted from 2 weeks to 26 months (3 studies did not report intervention time length) - Crowe Critical Appraisal Tool (CCAT) used for quality assessment; conducted by two independent researchers and average quality score was reported - Meta-analysis conducted utilizing Mantel-Haenszel odds ratio (OR) with 95% confidence interval. - Random-effect model used due to study variances - P-values utilized for test effect; value of less than 0.05 was considered significant - Published in English - Pharmacist-led educational interventions provided to healthcare providers - Reported medication error rates or number before and after the intervention Exclusion Criteria: - Non-interventional descriptive studies - Systematic reviews or meta-analyses - Investigating pharmacy reconciliation and their effect on medication discrepancies - Led by students Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 54 (Koeck et al., 2021) Level I Systematic review Purpose: Identify interventions designed to reduce and/or prevent drug dispensing, administration, and monitoring errors and determine their effect or pharmacy technicians - Editorials, opinions, abstract-only studies - 20 studies (1 study with dispensing error, 7 studies drug administration error, 12 studies targeting multiple steps of the medication use process) - Five study types included: randomized controlled trial, controlled clinical trial, controlled before-after study, interrupted time-series study, uncontrolled before-after study Inclusion criteria: - Intervention to reduce drug - 44 different interventions (majority were administrative controls) - Single intervention studies vs bundle intervention studies - Interventions included: Administrative controls (education or practical training, guidelines or protocols, rearrangement of staff or equipment, expert consultation, warning signs); Engineering controls (Electronic workflow/CPOE, Enhanced medication delivery - 1 researcher assessed titles and abstracts using a piloted form - A second reviewer independently examined a random 10% of the first researchers results using the same form - Interrater agreement was calculated using Cohen K. - Impact of interventions assessed through error rate - Individual interventions were classified in a hierarchical approach to risk control - 14 studies (34 interventions) revealed a statistically significant reduction in medication error rates - 3 studies revealed a non-statistically significant difference in error rates - 3 studies had mixed results - Non-statistically significant preference (p=0.28) for bundle interventions - Studies with substitution or engineering interventions were 1.4 times likely to reduce error rates compared to administrative controls alone; however this was not statistically significant (p=0.23) - Classification of interventions were performed independently - Studies focused on educational interventions revealed an absolute risk Strengths: - Level of evidence - Detailed description of sample collection and methodology - Definitions and outcome parameters described Limitations: - High heterogeneity - Not all interventions had p-values reported - Some included studies had a significant risk of bias - Inability to rule out publication bias - Primarily hospital based studies Implications: 55 Strengths/Limitations/Impli cations - When designing interventions, high-level hierarchical interventions should be considered, however it is important to evaluate local conditions prior to implementation. Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings by two authors. Discrepancies were resolved with a third author. - Fisher exact test (p<0.05) or Mann-Whitney U-test (two-tailed, p<0.05) used for group comparisons reduction of: 17.9%; 15.8%; 7.1 and 41% - 6 out of 7 studies that implemented higher level controls resulted in significant error reduction rate (86%) - 8 of 13 studies focused on administrative controls only resulted in significant error reductions dispensing, administration, and/or monitoring errors in pediatric setting - Randomized controlled trials, controlled clinical trial, controlled before-after study, interrupted time-series study, uncontrolled before-after study Exclusion criteria: - Medication errors when administered by patient or family equipment, Hands-free communication equipment, bar coded medication administration, computerized alert); Substitution (standardized dilution, pharmacist production unit, smart pumps) - 8 studies evaluated a single intervention in the medication use process - 12 studies investigated interventions at multiple stages in the medication use process - Many administrative controls were used with higher-level interventions (Keers et al., 2014) Level I Systematic - 6 Randomized controlled trials - Medication use technology (n=4) - Data extracted independently by two - 5 studies including automated drug dispensing (RR 0.72, 95% Strengths: - Search strategy is Citation Level of evidence/ design/ purpose Review Purpose: Review and critically appraise interventions designed to reduce medication administration errors in the hospital settings Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 56 and 7 controlled trials - Study locations included: 6 USA, 2 New Zealand, 1 France, 1 UK, 1 Canada, 1 Australia, 1 Vietnam Inclusion Criteria: - Studies published between 1985-November 2013 (any language) - Study reporting data needed to include medication adverse events and rate of medication administration errors - Hospital setting - RCTs and non-randomized controlled trials - Nurse education and training (n=3) - Changing practice in anesthesia (n=2) - Ward system changes (n=4) authors who then met a consensus on study details - Outcome rates between study groups compared using risk ratio (RR) with 95% confidence interval (CI) calculated using OpenEpi software - Two authors independently assessed risk of bias according to the EPOC Group criteria. Disagreements were rectified through a third author. CI 0.53-1.00), computerized physician order entry (RR 0.51, 95% CI 0.53-0.95), barcoded assisted medication administration with electronic administration records (RR 0.71, 95% CI 0.53-0.95), nursing education/training using simulation (RR 0.17, 95% CI 0.08-0.38), clinical pharmacist-led training (RR 0.76, 95% CI 0.67-0.87) reduced medication administration errors - Increased or equal rates were found in the remaining studies Nurse Education/Training - CD-ROM program did not change error outcomes - Didactic versus simulation-based learning showed statistically significant reduction in errors (RR 0.17, RR 95% CI 0.09-0.30) with simulation - A study utilizing lectures and practice-based training also showed significant reduction in errors (RR 0.76, 95% CI 0.67-0.87) thoroughly described - Detailed overview of inclusion/exclusion criteria and study definitions - Well described study comparisons and findings - Low bias risk for education studies - Level of evidence Limitations: - Study validity - High heterogeneity of included studies - Method of outcome reporting for individual studies Implications: - Further investigation is necessary with more rigorous studies. However, significant improvements were seen following nurse education/training, medication use technology interventions, but these results should be used cautiously due to less than optimal study designs or 57 Strengths/Limitations/Impli cations suitable data collection and all were susceptible to bias. Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings - Outcome rate comparisons between intervention and comparator group - Outcome rates reported or able to be calculated Exclusion Criteria: - Theses and conference proceedings - Review articles - Studies focused on simulation - Conference abstracts - Theses and conference proceedings - Home/nursing homes/primary care/outpatient clinic research - Before and after Citation Level of evidence/ design/ purpose Sample Intervention Measurement: Variables and Instruments Findings Strengths/Limitations/Impli cations 58 studies that did not utilize a separate comparator group Appendix D Literature Review Intervention Types 59 Table 1 Literature Review Intervention Types Study Citation Educatio n Simulatio n Medicatio n administr ation/disp ensing Operati onal Medicatio n reconciliati on Medicatio n preparatio n Bundle d Checklists /protocols/ guidelines Clinical experts/ pharma cists Safety Committe e Resources (posters/dis plays/etc.) Engineering/ technical controls/CP OE (Pol-Caste ñeda et al., 2022) (Plutinská & Plevová, 2019) (Marufu et al., 2021) (Andersen et al., 2020) (Khalil et al., 2020) (Lee, 2019) x x x x x x x x x x x x x x x x x x x x x x x 60 Table 1 Literature Review Intervention Types Study Citation Educatio n Simulatio n Medicatio n administr ation/disp ensing Operati onal Medicatio n reconciliati on Medicatio n preparatio n Bundle d Checklists /protocols/ guidelines Clinical experts/ pharma cists Safety Committe e Resources (posters/dis plays/etc.) Engineering/ technical controls/CP OE (Lapkin et al., 2016) (Manias et al., 2020) (Durham et al., 2020) (Khalil & Lee, 2018) (Sanko & Mckay, 2017) (Jaam et al., 2021) (Koeck et al., 2021) (Keers et al., 2014) x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 61 Appendix E SWOT Analysis SWOT ANALYSIS 62 Appendix F Narrated Powerpoint Vaccine Education Template VACCINES EDUCATION FOR ALL VACCINES CARRIED IN CLINIC INCLUDED: ● Vaccine schedule ● Safe administration practices ● Explanation of disease (pathophysiology) to which it protects from ● Contraindications ● Special administration instructions if applicable ● Potential side effects and adverse reactions INDIVIDUAL VACCINES IN PEDIATRIC HEALTH CLINIC INCLUDED: DTaP Inactivated poliovirus Hepatitis A Varicella Haemophilus influenzae type b (Hib) Measles, Mumps, Rubella Hepatitis B Meningococcal A,C,W,Y (MenACWY) Pneumococcal (PCV13) Human papillomavirus Rotavirus Meningococcal B COMBINATION VACCINES AVAILABLE IN PEDIATRIC HEALTH CLINIC INCLUDED: VAXELIS: DTaP, inactivated poliovirus, Haemophilus influenzae type b, and hepatitis B PENTACEL: DTaP, inactivated poliovirus, and Haemophilus influenzae type b QUADRACEL: DTaP and inactivated poliovirus PROQUAD: Measles, Mumps, Rubella, and Varicella MEDICATION ADMINISTRATION SAFETY IM administration sites: -Deltoid (2 finger breadths below the acromion process) -Vastus Lateralis (middle third of lateral thigh between trochanter and knee -Ventrogluteal (with thumb facing anteriorly, place palm over greater trochanter, index finger is placed on ASIS, middle finger positioned to iliac crest; forms V for injection site) SQ administration site: -Thigh less than 12 months -Upper outer tricep area 12 months and up Needle length/size 63 Multiple injections: -Infants and younger children: Thigh, at least 1 inch apart -Older children and adults: Deltoid can be used, 1 inch apart Medication Rights: 1. Right patient a. Ask patient/parent name + DOB, compare to EMR/MCIR 2. Right drug a. Verify vaccine against EMR order/MCIR 3. Right dose a. Verify vaccine dose against EMR order 4. Right time a. Verify with vaccine schedule 5. Right route a. Verify correct route 6. Right reason a. Verify patient/parent understanding and consent to vaccines being administered 7. Right documentation a. Document vaccine name, lot number, and expiration date in EMR 64 Appendix G Vaccine Administration Simulation Overview VACCINE ADMINISTRATION SIMULATION OVERVIEW Role Play ● 1 staff member plays the parent/guardian ● 1 staff member is the vaccine administer Simulation Scenario ● 2 different scenarios ● Each scenario utilizes a different sample immunization record, patient, and vaccine requirement Participant Expectations ● Administering staff member is required to: ○ Verbalize each step of the medication administration process while executing ○ Provide education to parent/guardian/patient ○ Answer any questions the parent/guardian patient may have Evaluation ● Organizers observe the simulation ● Verify all necessary safety checks have been completed via checklist ● Take note of any education that was provided Debrief ● Following simulation, the participants and organizers discuss the event and provide feedback 65 Appendix H Pre/Post-Confidence Survey PRE/POST-CONFIDENCE SURVEY Confidence/Understanding Questions: All questions evaluated on a scale of 1-5 1. I feel I have a good understanding of the steps involved in vaccine administration. 1 - Strongly agree 1. I feel confident in vaccine administration. 2. I feel I have a good understanding of: a. Vaccine schedules 2 - Agree 3 - Unsure 4 - Disagree 3. I feel confident in my knowledge base regarding the vaccines I administer. 5 - Strongly disagree 4. I feel I have a good understanding of: a. Safe administration practices/Special administration instructions if applicable 5. I feel I have a good understanding of: a. Explanation of disease (pathophysiology) to which it protects from 6. I feel I have a good understanding of: a. Contraindications to administration of vaccines provided in clinic 7. I feel I have a good understanding of: a. Potential side effects and adverse reactions Appendix I Pre/Post-Knowledge Questionnaire 66 KNOWLEDGE QUESTIONNAIRE PRE AND POST INTERVENTION QUIZ: Directions - complete the questions to the best of your ability. 1. A 2-month-old infant needs to start the DTap series. What is the recommended schedule? a. 2 months, 3 months, 4 months, 10 months, 16 months b. 2 months, 3 months, 4 months, 12 months, 4-6 years c. 2 months, 4 months, 6 months, 15-18 months, 4-6 years d. 2 months, 6 months, 12 months, 4-6 years, 11-12 years 2. Is the statement below true or false? a. Hepatitis B vaccine should be administered by subcutaneous route. i. ii. TRUE FALSE 3. According to ACIP, by what age should all doses of rotavirus vaccine be administered? (select the correct answer) a. 15 weeks b. 5 months c. 6 months d. 8 months 4. A 4 month old infant developed hives and breathing problems shortly after his first dose of IPV. Should he receive a second dose today? a. YES b. NO 5. How many doses of vaccine are needed to complete the hepatitis B vaccine series in infants and children? a. 2 doses b. 3 doses c. 4 doses d. 6 doses 6. When administering MenACWY to an older child, the preferred site is the anterolateral aspect of the thigh? a. TRUE b. FALSE 7. Is the statement below true or false? a. MMR vaccine is routinely recommended for a 5 year old child who received a dose of MMR at 15 months of age. i. ii. TRUE FALSE 8. You are screening clients to determine who needs to receive vaccines today, including HPV vaccine. Which person should receive the HPV vaccine today? a. A 7 year old with asplenia b. An 11 year old child who is being seen for a sports physical c. A 48 year old male who is sexually active with several partners 67 9. How do you correctly identify a 5 year old patient who came into the clinic to receive vaccinations today? a. Ask patient/parent name and DOB - compare against EMR/MCIR b. Assume patient identity? c. Verify by asking the child's name? d. Verify by asking the child DOB? 10. Seasonal influenza vaccine should be considered for a 5-month-old boy with congenital heart disease? a. TRUE b. FALSE 11. Correct first dose in mL for a 11 year old receiving GARDASIL 9 (PF)? a. 0.5 mL INTRAMUSCULAR SYRINGE b. 1mL INTRAMUSCULAR SYRINGE c. 3 mL INTRAMUSCULAR SYRINGE 12. PROQUAD is a combination of which of the following vaccines? a. Measles, Mumps, Rubella, and Varicella b. DTaP and inactivated poliovirus c. DTaP, inactivated poliovirus, Haemophilus influenzae type b, and hepatitis B 13. A 2 month old infant is given his first dose of the Hib vaccine today using ACTHIB. According to the recorded schedule, when should the infant receive the remaining doses in the primary series? a. At 3 months and 4 months of age b. At 4 months and 6 months of age c. At 6 months and 12 months of age d. At 8 months and at 14 months of age 14. Patient C is about to receive VAXNEUVANCE (PF) 0.5 ML vaccine, what route should this vaccine be administered by? (select the correct answer) a. Intramuscular b. Subcutaneous c. Transdermal Note. Adapted from “You Call the Shots - Web-based Training Courses” by Centers for Disease Control and Prevention, 2024, https://www.cdc.gov/vaccines /ed/youcalltheshots.html. In the public domain. Appendix J Checklist For Vaccine Administration CHECKLIST FOR VACCINE ADMINISTRATION: 68 Note. Adapted from “Pediatric vaccine administration: Sustaining an improved process in a primary care setting,” by M. Durham, I. Didovic, & M. Gingell, 2020, Patient Safety, 2(20), p. 42 (https://doi.org/10.33940/med/2020.6.5). Copyright 2020 by Patient Safety. 69 Appendix K Project Cost Personal Pay Total Rachel Lindsay LPN #1 LPN #2 MA #1 MA #2 $40.00/hour X 180 hours $7,200.00 $40.00/hour X 180 hours $7,200.00 $24.00/hour X 1.75 hours $24.00/hour X 1.75 hours $16.00/hour X 1.75 hours $16.00/hour X 1.75 hours Other expenses Simulation cases paper & ink $7.00 Vaccine prop boxes paper & ink $14.00 Visual aids paper, ink, lamination, & hanging materials $30.60 Vaccine labeling paper & ink $5.00 Vaccine labeling organizational materials $30.00 Project Total = $14,626.6 $42.00 $42.00 $28.00 $28.00 $7.00 $14.00 $30.60 $5.00 $30.00 Appendix L Evolution of Intervention 70 Appendix M Pre/Post-Confidence Survey Results PRE/POST-CONFIDENCE SURVEY RESULTS 71 Appendix N Pre-Knowledge Questionnaire Results PRE-KNOWLEDGE QUESTIONNAIRE ANALYSIS: 72 73 Appendix O Train the Trainer Manual TRAIN THE TRAINER MANUAL INTRODUCTION TO THE INTERVENTION + STEP BY STEP GUIDE TO IMPLEMENTATION INTRODUCTION TO THE INTERVENTION: Intervention components: ● The intervention includes a narrated educational series provided to all staff involved in the vaccine administration process including; certified medical assistants (MAs), licensed practical nurses (LPNs), registered nurses (RNs), advanced practice registered nurses (APRNs), Doctors of Medicine (MDs), and Doctors of Osteopathic Medicine (DOs). The details and overview of this educational series can be reviewed in Appendix F. ○ Link to current narrated education powerpoint in STEP 13. ● After completion of the educational portion of the intervention, administering staff are provided with a low-fidelity simulation experience. The details and overview of this simulation can be reviewed in Appendix G. ● 1 week prior to simulation, all staff members will receive an informational handout/email explaining the simulation and expectations of the experience. Instructions will also be provided verbally just prior to simulation. ○ Informational handout should be adjusted as seen fit for current state of intervention. The impact of the chosen interventions is measured in three ways: ● A pre and post survey (Appendix H) is administered to healthcare professionals who administer vaccines at the pediatric Clinic. These staff members include: certified medical assistants (MAs), licensed practical nurses (LPNs), and registered nurses (RNs). The survey is administered prior to distribution of educational powerpoint and the post survey will be completed after the simulation event. ● To specifically measure staff knowledge of vaccines that are provided within the clinic, staff will complete a knowledge questionnaire (Appendix I) before and after completion of the narrated educational powerpoint presentations. This quantitative data is monitored with bar charts to observe scores of pre and post confidence surveys as well as pre and post knowledge tests after implementation of narrated educational powerpoint presentations and low-fidelity simulation events. ● Staff members that participate in the low-fidelity simulation experience are evaluated with a checklist (Appendix J) during the simulation. Staff members will receive credit for each step of the check-list that they follow. ○ The checklist is completed by a trained simulation observer. ● A group debriefing is also conducted after all staff members complete the low-fidelity simulation. This qualitative data is used to monitor for variation between experiences as well as assess efficiency and possible barriers of intervention implementation processes. ● Debrief discussions with staff members and leadership are utilized to implement future improvement activities when needed and inform the intervention. (Questions to ask - see STEP 9) STEP 1: Distribution of pre-knowledge questionnaires to all participants and distribution of pre-knowledge questionnaires as well as pre-confidence surveys to simulation participants (approximately 1-2 weeks before powerpoint distribution). 74 STEP 2: Once pre-knowledge questionnaires have been completed - distribution of narrated educational powerpoint will take place (give 3-5 weeks for staff to complete education powerpoint) STEP 3: Once all required staff members have viewed education powerpoint distribute post-knowledge questionnaire (give 1-2 weeks to complete) STEP 4: Once all post-knowledge questionnaires are completed distribute pre-confidence surveys (give 1-2 weeks to complete prior to simulation event) STEP 5: Once all pre-confidence surveys are completed, 1 week prior to simulation, distribute an informational handout by email explaining the simulation and expectations of the experience (see appendix). Provide instructions verbally just prior to simulation in person. STEP 6: Agree upon a day with staff for implementation of low-fidelity simulation. STEP 7: Arrive at desired/agreed upon time on simulation day and in person verbally explain events to participating staff and confirm review of informational handout with check-list. STEP 8: Implement low-fidelity simulation following steps on informational handout with checkoff sheet. STEP 9: Complete debrief with staff. Ask the following questions: ● What went well? ● What didn’t go well? ● What did you learn from today’s simulation? ● Do you feel that the simulation was useful or would be useful for initial training? STEP 10: Distribute post confidence survey and Instruct staff to complete within 1 week. 75 STEP 11: Following educational powerpoint and simulation experience ensure medication administration checklists, vaccine schedules, and “quick-tip” vaccine education reinforcement posters are up to date and update as needed. STEP 12: Check-in with staff and leadership on a regular basis (ex: every month) STEP 13: LINK TO CURRENT EDUCATIONAL POWERPOINT: https://mediaspace.msu.edu/media/t/1_rvz2fe5q